[
https://issues.apache.org/jira/browse/FLINK-39502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18076300#comment-18076300
]
Hao Li commented on FLINK-39502:
--------------------------------
Below is what's found by AI, I need to further validate it. But flag it first:
h3. Root cause: lz4-java upgrade silently swapped the fast decompressor to a
pure-Java implementation
The regression bisects to a single commit in the f597bef942..360aa9709b range:
* commit: e9afe8bfc52
* PR: https://github.com/apache/flink/pull/27535
* JIRA: FLINK-38764 — "Update lz4-java to 1.10.3"
h4. Why it regresses these two benchmarks (and not
\{{uncompressedFilePartition}})
Both regressed benchmarks exercise the LZ4 shuffle-compression path:
* \{{compressedFilePartition}} sets \{{SHUFFLE_COMPRESSION_CODEC=LZ4}}
explicitly.
* \{{remoteSortPartition}} inherits the Flink default codec, which is also
\{{LZ4}}.
* \{{uncompressedFilePartition}} uses the same blocking-partition I/O path
with \{{NONE}} and is not in the regression list — isolating the cause to
compression.
return the *Java safe* fast-decompressor even when the native library is
available — likely as a defensive fix for CVE-2025-66566, which was an
out-of-bounds read in the JNI fast
decompressor. The factory's \{{toString()}} still reports
\{{"LZ4Factory:JNI"}} and nothing logs a fallback, so the swap is silent.
h4. Reproduction
|| jar || compressor || decompressor ||
| \{{org.lz4:lz4-java:1.8.0}} | \{{LZ4JNICompressor}} |
\{{LZ4JNIFastDecompressor}} |
| \{{at.yawk.lz4:lz4-java:1.10.3}} | \{{LZ4JNICompressor}} |
\{{LZ4JavaSafeFastDecompressor}} |
Tight-loop microbench on a 32 KiB buffer (numbers are cache-inflated; the
ratio is the point):
|| op || 1.8.0 || 1.10.3 || delta ||
| decompress | 29,597 MB/s | 4,498 MB/s | *~6.6x slower* |
| compress | 2,595 MB/s | 2,804 MB/s | unchanged |
Probe used (any JDK):
\{code:java}
import net.jpountz.lz4.LZ4Factory;
public class Probe {
public static void main(String[] args) {
LZ4Factory f = LZ4Factory.fastestInstance();
System.out.println(f);
System.out.println(f.fastCompressor().getClass().getName());
System.out.println(f.fastDecompressor().getClass().getName());
}
}
\{code}
h4. Suggested remediation
Reverting is not appropriate — the CVEs are real. Options:
# Pin \{{Lz4BlockDecompressor}} to \{{safeDecompressor()}} explicitly so the
cost is conscious and reflected in code/docs. Same runtime perf as today, but
the silent-fallback footgun goes
away.
# Switch the default \{{taskmanager.network.compression.codec}} to \{{ZSTD}}
for blocking shuffle (Flink already ships a \{{ZstdBlockCompressor}}) — its
native path is unaffected.
# Move to a different LZ4 binding whose JNI fast decompressor is not affected
by CVE-2025-66566 (e.g., \{{aircompressor}}'s pure-Java LZ4, which is
competitive with \{{at.yawk.lz4}}'s
safe path).
For tracking: the PR template on FLINK-38764 marked "runtime per-record code
paths (performance sensitive): no", which turned out to be inaccurate for
blocking shuffle — worth flagging
so future security backports of this dep get a benchmark gate.
> Performance regression in remoteSortPartition and compressedFilePartition
> -------------------------------------------------------------------------
>
> Key: FLINK-39502
> URL: https://issues.apache.org/jira/browse/FLINK-39502
> Project: Flink
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: David Anderson
> Priority: Blocker
> Fix For: 2.3.0
>
>
> flink-benchmarks shows a significant performance drop in remoteSortParition
> and compressedFilePartition occurred somewhere between commits f597bef942 and
> 360aa9709b. Unfortunately, these commits are several weeks apart; apparently
> the benchmarks weren't running between Feb 19 and Mar 14.
> See
> [http://flink-speed.xyz/timeline/#/?exe=12&ben=remoteSortPartition&extr=on&quarts=on&equid=off&env=3&revs=200]
> and
> [http://flink-speed.xyz/timeline/#/?exe=12&ben=compressedFilePartition&extr=on&quarts=on&equid=off&env=3&revs=200]
> for more info. The problem occurs for both Java 11 and Java 17.
> !image-2026-04-20-10-39-19-453.png!!image-2026-04-20-10-39-36-416.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
