Hi JB, Thanks, these are really helpful points. To your questions:
1. I like your order better than mine. We could introduce the Iceberg-owned shim types while everything still runs on Jackson 2, as a pure refactor with no new dependency and no runtime-jar or LICENSE/NOTICE changes. The Jackson-typed APIs would be deprecated at that point. Adding Jackson 3, the license updates and the switch could then come after. That's cleaner! 2. Rather than writing new tests, we could keep both backends around temporarily behind a test-only switch (a system property, not a user-facing flag) and run the existing suites on each. Comparing serialized output between the two would also surface any differences in Jackson 3's defaults. 3. REST layer: you're right that shims alone won't be enough, since callers need to register serializers on their own mapper. Your adapter-artifact idea looks like the right shape to me: an agnostic core, and optional iceberg-jackson2 / iceberg-jackson3 artifacts that each provide a ready-made Jackson module. Each caller would just register the module it needs. Does that match what you had in mind? 4. Phasing: I agree with what you suggested, but I also think we need to strike a balance between the number of PRs and their size. It should also be possible to parallelize the work to some extent. Does this sound reasonable? Thanks, Alex On Mon, Nov 9, 2026 at 12:09 PM Jean-Baptiste Onofré <[email protected]> wrote: > > Hi Alex, > > I'm +1 generally speaking. Also, I think it's a good idea to hide Jackson > behind Iceberg types, it's a good cleanup (not directly related to Jackson 3). > > Reading your points, I have a few comments: > 1. About the ordering and "mixing" Jackson 2 and 3, I would do the shims > first (your items 3 and 4) while they are still backed by Jackson 2. We could > deprecate the Jackson typed API in 1.13 (I'm thinking especially > HTTPRequest.mapper() and HTTPClient.Builder.withObjectMapper()). The switch > to Jackson 3 then becomes an internal change, and we ship both Jackson 2 and > 3 for a short time. Bundling both is not really a technical problem: they use > different packages, and the Spark/Flink runtime jars already relocate > com.fasterxml, so we just need a tools.jackson relocation. It just makes the > jar bigger and more "exposed" (CVE, etc). > 2. Rather than writing new tests, I would parameterize the existing parser > tests (TestSchemaParser, REST request/response parser tests, ...) to run > against both Jackson 2 and Jackson 3 during the transition, They already > cover the JSON contracts. Also, comparing the serialized output of Jackson 2 > & 3 would also catch the default feature changes in Jackson 3. > 3. I would be very very careful on your point 4, about the REST layer. REST > catalog servers register the serializers on the ObjectMapper. Polaris, for > instance, does that with Quarkus mapper, and Quarkus is moving to Jackson 3. > These servers can't avoid Jackson completely: they need a Jackson module to > plug into their mapper. Also, RESTSerializers exposes about 20 public nested > serializer/deserializer classes that extend Jackson types, not only > registerAll(). So I don't think shims alone are enough here. The "separate > artifacts" idea from your first email (and Steve legacy module) fits this > part well. Core would stay binding agnostic, and small adapter modules > (something like iceberg-rest-jackson2 and iceberg-rest-jackson3) would each > expose a Jackson module. Servers pick the one they need. > 4. I agree with the splitting. I would suggest: > - introduce the shims and migrate the parsers package by package (several > PRs, I would be happy to help there) > - the REST layer > - dual binding tests and Jackson 3 switch > - removal of the deprecated Jackson 2 > > Happy to help there! > > Regards > JB > > On Fri, Nov 6, 2026 at 7:40 PM Alex Dutra via dev <[email protected]> > wrote: >> >> Hi Steve, hi all, >> >> I played with Steve's suggestion, and I think it's the right >> direction. Here are the main ideas: >> >> 1) Ship both Jackson 2 and Jackson 3 during a transition period. The >> runtime jars would bundle both, until Jackson 2 is removed. >> >> 2) Introduce compatibility tests to verify that the switch carries no >> regressions. >> >> 3) Remove Jackson 2 types from the Json serde layer. This requires >> introducing 2-3 Iceberg-owned shims on top of Jackson types, as Steve >> suggested. The change is mostly mechanical, but touches a lot of >> files. >> >> 4) Remove Jackson 2 types from the REST layer. We'd introduce more >> shims to replace exposure points like RESTSerializers.registerAll() >> and HTTPClient.Builder.withObjectMapper(). >> >> Would the community agree on this approach? I think it's a sane one, >> and has the benefit of cleaning up the public API, which is a good >> thing to do anyway. >> >> If so, we just need to figure out how to proceed. The full change >> would touch more than 100 files, especially item 3 above. It's >> certainly possible to split it into a few PRs. >> >> What are your thoughts? >> >> Thanks, >> Alex >> >> On Tue, Jul 28, 2026 at 12:22 PM Steve Loughran <[email protected]> wrote: >> > >> > >> > I'd go for >> > >> > leave jackson 2 as is, deprecate, with a goal to cut completely at some >> > point >> > accompany an api which hides the json binding, >> > Move to jackson 3 internally, never expose this, either remove the jackson >> > 2 apis or move to a iceberg-core-jackson2-legacy module for expllicit use >> > and make clear it's only there for one more release >> > >> > Maybe moving the jackson2 stuff to a legacy module from the outset makes >> > clear its doomed, but if there's one lesson from OSS, it's "unless you are >> > ruthless and make clear it will go, people don't believe you". >> > At then same time: make changes incompatible at the binary level and >> > people don't upgrade to your latest release, so don't pick up the latest >> > releases, with direct and indirect CVE fixes. >> > >> > On that topic, there's a new jackson 2 release out: no release notes. That >> > implies "security fix but they don't want to announce the CVE just yet" >> > >> > >> > On Mon, 27 Jul 2026 at 14:52, Alexandre Dutra <[email protected]> wrote: >> >> >> >> Hi Steve, hi all, >> >> >> >> We discussed this topic briefly in the last sync, in the context of >> >> iceberg-core specifically. Here is a quick summary: >> >> >> >> - The exposure of the Jackson 2 API in iceberg-core often appears >> >> accidental. >> >> >> >> - If exposure is unavoidable, introducing dual exposure via method >> >> overloads has been proposed. >> >> >> >> - Another option under consideration is providing two separate >> >> artifacts, one for Jackson 2 and one for Jackson 3. >> >> >> >> - Ultimately, we need more time to evaluate the optimal path forward. >> >> >> >> To help guide this decision, I did a deeper investigation into where >> >> these Jackson types actually leak out. The exposure broadly falls into >> >> four categories: >> >> >> >> 1) Parser classes (SchemaParser, MetadataUpdateParser, etc.) >> >> 2) The JsonUtil utility class. >> >> 3) The HTTPRequest.mapper() method. >> >> 4) RESTSerializers.registerAll(ObjectMapper) and >> >> HTTPClient.Builder.withObjectMapper(ObjectMapper). >> >> >> >> Based on these findings, I suggest the following approach for each area: >> >> >> >> - For the parser classes (1), we should ideally restrict visibility to >> >> package-private if feasible (implies revapi exceptions). >> >> >> >> - For JsonUtil (2), I recommend either transitioning it to >> >> package-private or deprecating it for removal. >> >> >> >> - For HTTPRequest.mapper() (3), the exposure was an unfortunate design >> >> choice: we should deprecate it for removal and re-implement the logic >> >> within a package-private helper class. >> >> >> >> Item (4) represents the biggest challenge since consumers legitimately >> >> rely on them to configure their object mappers. Removing them entirely >> >> might be too disruptive; a better approach would be to accept the >> >> tight coupling and introduce parallel Jackson 3 overloads instead. Or, >> >> as Steve mentions, add a layer of indirection to decouple the API >> >> completely. >> >> >> >> In conclusion, if we put the extension points (4) aside for a moment, >> >> it looks like we have a clear opportunity to perform some API cleanup >> >> before we fully commit to the Jackson 3 migration, by reducing the >> >> Jackson 2 exposure to its minimum. >> >> >> >> What do you all think? >> >> >> >> Thanks, >> >> Alex >> >> >> >> On Mon, Jul 20, 2026 at 2:57 PM Steve Loughran <[email protected]> >> >> wrote: >> >> > >> >> > >> >> > the public APIs are the pain point, and adding jackson 3 overloads is >> >> > possibly going to replicate the same problem. >> >> > >> >> > I'd propose >> >> > - identify those things where json parsing/generation takes place and >> >> > are published >> >> > - see how they could extended such that jackson 3 supported could be >> >> > done without adding jackson 3 to their signatures >> >> > >> >> > I'm thinking of how the various parsers have toJson() methods which >> >> > take jackson generators >> >> > >> >> > public static void toJson(TableIdentifier identifier, JsonGenerator >> >> > generator) >> >> > >> >> > if another layer of indirection is added (JsonBinding?) that supplied >> >> > the generator then an overridden entry point could be supplied >> >> > >> >> > public static void toJson(TableIdentifier identifier, JsonBinding >> >> > binding) { >> >> > if (binding instance of Jackson2Binding jackson2binding) { >> >> > return toJson(identifier, jackson2binding.generator()); >> >> > } else { >> >> > return ((jackson3binding)bindiong)....) >> >> > >> >> > These could be added for the apis today, with the jackson2 stuff tagged >> >> > as deprecated, and new apis only coming though the new overload >> >> > >> >> > I'm glossing over the many implementation details here, but the key >> >> > thing is: exposing someone else's library as part your public API >> >> > inevitably becomes something you end up regretting. >> >> > >> >> > >> >> > On Thu, 16 Jul 2026 at 15:53, Alexandre Dutra <[email protected]> wrote: >> >> >> >> >> >> Hi all, >> >> >> >> >> >> As more frameworks (like Spring Boot and Quarkus) migrate to Jackson >> >> >> 3, I wanted to open a conversation about what a Jackson 3 migration >> >> >> would look like for Iceberg and what trade-offs we should think >> >> >> through together. >> >> >> >> >> >> Iceberg currently depends on Jackson 2 in several modules, most >> >> >> significantly iceberg-core. Other modules with direct Jackson >> >> >> dependencies include iceberg-aws, iceberg-nessie, iceberg-snowflake, >> >> >> and iceberg-kafka-connect. Most modules could be migrated separately. >> >> >> >> >> >> The migration is largely mechanical but pervasive: artifact >> >> >> coordinates and Java package names change across the board, several >> >> >> core classes and methods are renamed, ObjectMapper construction moves >> >> >> to a builder pattern, and exceptions become unchecked. JDK 17 is >> >> >> required, which Iceberg already enforces. >> >> >> >> >> >> The blast radius for iceberg-core would be roughly: >> >> >> >> >> >> - Spark and Flink runtime JARs: low impact. Both already shade Jackson >> >> >> 2; adding a relocation rule for Jackson 3 is a one-line fix per >> >> >> runtime variant. >> >> >> >> >> >> - Kafka Connect: also low impact, although the Kafka Connect >> >> >> distribution uses unshaded JARs. >> >> >> >> >> >> - Downstream users of iceberg-core: this is the most significant >> >> >> concern. Jackson types leak into the public API of many XYZParser >> >> >> classes, JsonUtil, and the REST HTTP layer. >> >> >> >> >> >> The honest answer is that icebegr-core's public API is permanently >> >> >> Jackson-coupled by design. We should imho acknowledge this fact; the >> >> >> best migration approach is to add Jackson 3 overloads wherever >> >> >> applicable, and make Jackson 2 and 3 coexist for some time. >> >> >> >> >> >> Looking forward to hearing your thoughts on this topic. >> >> >> >> >> >> Thanks, >> >> >> Alex
