kerwin-zk opened a new pull request, #8479: URL: https://github.com/apache/paimon/pull/8479
### Purpose Fix the todo left in `paimon-spark/pom.xml` by https://github.com/apache/paimon/pull/6082: ```xml <!-- todo(fix me): we should use the jackson libraries shaded by Paimon. --> ``` `JsonUtils` in paimon-spark-common was the only Scala code still using the unshaded jackson resolved from Spark's runtime (the spotless import-replacement rule that rewrites jackson imports to the shaded ones only applies to Java sources). So the jackson version it ran on was decided entirely by the Spark version (2.15.x for Spark 3.5, 2.18.x for Spark 4.x), which may differ from the version Paimon expects. This PR: - Rewrites `PaimonSourceOffset` json serde (the only user of `JsonUtils`) with `JsonSerdeUtil.OBJECT_MAPPER_INSTANCE`, i.e. the jackson shaded by Paimon. `ScalaObjectMapper`/`DefaultScalaModule` cannot be kept because `paimon-shade-jackson-2` does not bundle `jackson-module-scala`; since the offset only has 3 primitive fields, the serde is written manually. The JSON field names and layout stay exactly the same, so streaming checkpoints remain compatible in both directions. - Removes `JsonUtils`. - Replaces the todo comment: the `com.fasterxml.jackson.core:*` exclusion on `paimon-format` itself is correct (paimon-format already shades jackson into its jar) and is kept. ### Tests Covered by existing `PaimonSourceTest`, which asserts offset json round-trips through `query.lastProgress.sources(0).endOffset`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
