dweiss opened a new pull request, #811: URL: https://github.com/apache/commons-compress/pull/811
TL;DR; I am a human but I used LLMs to rewrite bzip2 code in commons-compress and it's 2x as fast. I thought you'll find this interesting - it isn't meant to be merged in as-is but is sure as hell interesting (at least to me!). So, I've been working with (huge) bz2 files from Java level ([1]) - their decompression takes... a good while (why wikimedia doesn't use zstd for this is another question I'll leave aside). Compression/decompression code is a great self-contained example of hairy stuff and I thought - it'd be interesting to burn some watts and give LLM a task to improve the current implementation. In short - I gave it a simple prompt of : "Would it be possible to speed up apache-commons bzip2 implementation (especially the decompression) using new Java language features? Consult with implementations in other languages, if needed." And worked iteratively from there. It was very much unsupervised in what it chose to optimize and how. Eventually though, this is what I see (paths a bit redacted, disk cache-primed): ``` $ time java -cp commons-io-2.22.0.jar:commons-compress-1.29.0-SNAPSHOT.jar BZip2SumTest.java enwiki_general-20260823-00000.json.bz2 enwiki_general-20260823-00000.json.bz2: 3,364,783,751 bytes, sum = 300,473,491,475 (47.00 s, 71.6 MB/s) real 0m47.469s user 0m49.976s sys 0m0.236s $ time java -cp commons-io-2.22.0.jar:commons-compress-1.28.0.jar BZip2SumTest.java enwiki_general-20260823-00000.json.bz2 enwiki_general-20260823-00000.json.bz2: 3,364,783,751 bytes, sum = 300,473,491,475 (91.01 s, 37.0 MB/s) real 1m31.506s user 1m32.751s sys 0m0.211s $ time bunzip2 -c enwiki_general-20260823-00000.json.bz2 | wc -c 3364783751 real 1m10.402s user 1m9.625s sys 0m3.266s ``` Which are... crazy numbers but they're consistent with what my eyes see... I don't know if it optimized for my particular CPU/ cache layout or not but I found it interesting that (a) it's correct and works all across the board, (b) it is much, much faster than the code currently in apache-commons. I wonder what people think and if these results reproduce for others (I haven't tried this yet across any other known compression data sets like the silesia corpus, etc.). I also include the "description" of changes from the LLM from those interested [2]. [1] https://dumps.wikimedia.org/other/cirrus_search_index/20260823/index_name%3Denwiki_content/ [2] LLM summary of changes. <img width="1956" height="1410" alt="image" src="https://github.com/user-attachments/assets/d95870d6-bac6-4c97-9953-80d1acf76fca"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
