driessamyn-sparta commented on PR #27700:
URL: https://github.com/apache/flink/pull/27700#issuecomment-3996249993
> Thnx for the patch. I do have some concerns, because what happens if the
internal implementation of AWS SDK v1 relies on Jackson somewhere? Relocating
Jackson would then break then the implementation.
Thanks, @MartijnVisser
My understanding is that given we shade the AWS SDK, relocation should work.
The shade config uses <include>*:*</include>, which means the AWS SDK v1
classes are bundled into the fat jar too. When the shade plugin applies the
`com.fasterxml.jackson` -> `org.apache.flink.fs.s3hadoop.shaded.jackson`
relocation, it rewrites all bytecode references in every included class,
including the AWS SDK classes.
So the SDK's compiled Jackson calls are transparently redirected to the
relocated package, which exists in the jar.
This is hard to verify using a unit test as the test would need to operate
on the packaged jar.
However, we can inspect the fat jar to verify this manually after creating
the fat jar:
```bash
JAR=$(find flink-filesystems/flink-s3-fs-hadoop/target \
-name "flink-s3-fs-hadoop-*.jar" \
-not -name "original-*" \
-not -name "*sources*" | head -1)
echo "Inspecting: $JAR"
echo ""
echo "Check 1: Relocated Jackson classes are present ==="
jar tf "$JAR" | grep
"org/apache/flink/fs/s3hadoop/shaded/jackson/databind/ObjectMapper.class"
echo ""
echo "Check 2: Original com/fasterxml/jackson classes are NOT present ==="
ORIGINAL=$(jar tf "$JAR" | grep "^com/fasterxml/jackson" || true)
if [ -z "$ORIGINAL" ]; then
echo "PASS: no com/fasterxml/jackson classes found in shaded jar"
else
echo "FAIL: unrelocated Jackson classes found:"
echo "$ORIGINAL" | head -5
fi
echo ""
echo "Check 3: AWS SDK bytecode references relocated Jackson ==="
MARKER="org/apache/flink/fs/s3hadoop/shaded/jackson/databind/ObjectMapper"
if unzip -p "$JAR" "com/amazonaws/util/json/Jackson.class" 2>/dev/null |
grep -qa "$MARKER"; then
echo "PASS: com.amazonaws.util.json.Jackson references relocated Jackson
($MARKER)"
else
echo "FAIL: relocated Jackson reference not found in AWS SDK class"
fi
```
```
Inspecting:
flink-filesystems/flink-s3-fs-hadoop/target/flink-s3-fs-hadoop-2.3-SNAPSHOT.jar
Check 1: Relocated Jackson classes are present ===
org/apache/flink/fs/s3hadoop/shaded/jackson/databind/ObjectMapper.class
Check 2: Original com/fasterxml/jackson classes are NOT present ===
PASS: no com/fasterxml/jackson classes found in shaded jar
Check 3: AWS SDK bytecode references relocated Jackson ===
PASS: com.amazonaws.util.json.Jackson references relocated Jackson
(org/apache/flink/fs/s3hadoop/shaded/jackson/databind/ObjectMapper)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
