dweiss commented on PR #811: URL: https://github.com/apache/commons-compress/pull/811#issuecomment-5451710412
> (I have a feeling the single byte and Boolean Arrays of the old code hurt, and the new one dient even use crazy optimisations. I agree - it doesn't look too terrifying. Sure - it's complex and uses some assumptions (cache and cpu parallelism friendliness) but it's not revolutionary. It is still a significant speed improvement. What was a bit shocking to me is this - this time on a mac (completely different architecture): ``` $ curl https://dumps.wikimedia.org/other/cirrus_search_index/20260823/index_name%3Denwiki_general/enwiki_general-20260823-00000.json.bz2 > /tmp/enwiki.bz2 $ mvn package dependency:copy-dependencies $ time java -cp target/dependency/commons-io-2.22.0.jar:target/commons-compress-1.29.0-SNAPSHOT.jar ./temp/BZip2SumTest.java /tmp/enwiki.bz2 /tmp/enwiki.bz2: 3,364,783,751 bytes, sum = 300,473,491,475 (34.71 s, 96.9 MB/s) java -cp ./temp/BZip2SumTest.java /tmp/enwiki.bz2 36.12s user 0.21s system 103% cpu 34.989 total $ time bunzip2 -c /tmp/enwiki.bz2 | wc -c 3364783751 bunzip2 -c /tmp/enwiki.bz2 44.51s user 0.43s system 99% cpu 44.946 total wc -c 0.03s user 0.64s system 1% cpu 44.944 total ``` Wild. It is much faster than the native tool. > Did you disclose your agent, model, harness and skills? I didn't want this to look like an ad for a particular vendor... you can sort of see it in the intermediate commits - I used claude and fable/ high. I don't have any particular skills set up but I'm sure it collects and reuses some knowledge I've used in the past sesssions... hard to tell what it does know about one and what it doesn't these days. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
