Hi Alex, Yes, it's a great approach.
The nice thing with the order I propose is that the first phase is a pure refactoring that we can review easily and ship on its own (with deprecation flag). It doesn't need any new dependency or LICENSE/NOTICE change. The test only switch works for me: it avoids to change every test class. Maybe worth to run as a dedicated Gradle task that covers only the core parser and REST tests, so CI time doesn't increase much. About the REST layer, yes, that's exactly what I had in mind. The way I see it, the parsers only use the Iceberg shim types. Each adapter artifact implements the shims on top of its Jackson version and provide a Jackson module with the seralizers (basically what RESTSerializers does today). The adapters stay thin, and core doesn't depend on them. It means the lifecycle of iceberg-jackson2 is not tied to the Jackson 3 switch in core. REST servers still on Jackson 2 can keep using it after the switch. We can decide later when to remove it, and mark it clearly as "legacy" as Steve suggested. To parallelize the work, I would start with one PR that adds the shim types and migrates one parser package as a reference. The shims become public API, so we should agree on them before we touch on a bunch of files. Regards JB On Mon, Nov 9, 2026 at 7:07 PM Alex Dutra via dev <[email protected]> wrote: > Hi JB, > > Thanks, these are really helpful points. To your questions: > > 1. I like your order better than mine. We could introduce the > Iceberg-owned shim types while everything still runs on Jackson 2, as > a pure refactor with no new dependency and no runtime-jar or > LICENSE/NOTICE changes. The Jackson-typed APIs would be deprecated at > that point. Adding Jackson 3, the license updates and the switch could > then come after. That's cleaner! > > 2. Rather than writing new tests, we could keep both backends around > temporarily behind a test-only switch (a system property, not a > user-facing flag) and run the existing suites on each. Comparing > serialized output between the two would also surface any differences > in Jackson 3's defaults. > > 3. REST layer: you're right that shims alone won't be enough, since > callers need to register serializers on their own mapper. Your > adapter-artifact idea looks like the right shape to me: an agnostic > core, and optional iceberg-jackson2 / iceberg-jackson3 artifacts that > each provide a ready-made Jackson module. Each caller would just > register the module it needs. Does that match what you had in mind? > > 4. Phasing: I agree with what you suggested, but I also think we need > to strike a balance between the number of PRs and their size. It > should also be possible to parallelize the work to some extent. > > Does this sound reasonable? > > Thanks, > Alex > > On Mon, Nov 9, 2026 at 12:09 PM Jean-Baptiste Onofré <[email protected]> > wrote: > > > > Hi Alex, > > > > I'm +1 generally speaking. Also, I think it's a good idea to hide > Jackson behind Iceberg types, it's a good cleanup (not directly related to > Jackson 3). > > > > Reading your points, I have a few comments: > > 1. About the ordering and "mixing" Jackson 2 and 3, I would do the shims > first (your items 3 and 4) while they are still backed by Jackson 2. We > could deprecate the Jackson typed API in 1.13 (I'm thinking especially > HTTPRequest.mapper() and HTTPClient.Builder.withObjectMapper()). The switch > to Jackson 3 then becomes an internal change, and we ship both Jackson 2 > and 3 for a short time. Bundling both is not really a technical problem: > they use different packages, and the Spark/Flink runtime jars already > relocate com.fasterxml, so we just need a tools.jackson relocation. It just > makes the jar bigger and more "exposed" (CVE, etc). > > 2. Rather than writing new tests, I would parameterize the existing > parser tests (TestSchemaParser, REST request/response parser tests, ...) to > run against both Jackson 2 and Jackson 3 during the transition, They > already cover the JSON contracts. Also, comparing the serialized output of > Jackson 2 & 3 would also catch the default feature changes in Jackson 3. > > 3. I would be very very careful on your point 4, about the REST layer. > REST catalog servers register the serializers on the ObjectMapper. Polaris, > for instance, does that with Quarkus mapper, and Quarkus is moving to > Jackson 3. These servers can't avoid Jackson completely: they need a > Jackson module to plug into their mapper. Also, RESTSerializers exposes > about 20 public nested serializer/deserializer classes that extend Jackson > types, not only registerAll(). So I don't think shims alone are enough > here. The "separate artifacts" idea from your first email (and Steve legacy > module) fits this part well. Core would stay binding agnostic, and small > adapter modules (something like iceberg-rest-jackson2 and > iceberg-rest-jackson3) would each expose a Jackson module. Servers pick the > one they need. > > 4. I agree with the splitting. I would suggest: > > - introduce the shims and migrate the parsers package by package > (several PRs, I would be happy to help there) > > - the REST layer > > - dual binding tests and Jackson 3 switch > > - removal of the deprecated Jackson 2 > > > > Happy to help there! > > > > Regards > > JB > > > > On Fri, Nov 6, 2026 at 7:40 PM Alex Dutra via dev < > [email protected]> wrote: > >> > >> Hi Steve, hi all, > >> > >> I played with Steve's suggestion, and I think it's the right > >> direction. Here are the main ideas: > >> > >> 1) Ship both Jackson 2 and Jackson 3 during a transition period. The > >> runtime jars would bundle both, until Jackson 2 is removed. > >> > >> 2) Introduce compatibility tests to verify that the switch carries no > >> regressions. > >> > >> 3) Remove Jackson 2 types from the Json serde layer. This requires > >> introducing 2-3 Iceberg-owned shims on top of Jackson types, as Steve > >> suggested. The change is mostly mechanical, but touches a lot of > >> files. > >> > >> 4) Remove Jackson 2 types from the REST layer. We'd introduce more > >> shims to replace exposure points like RESTSerializers.registerAll() > >> and HTTPClient.Builder.withObjectMapper(). > >> > >> Would the community agree on this approach? I think it's a sane one, > >> and has the benefit of cleaning up the public API, which is a good > >> thing to do anyway. > >> > >> If so, we just need to figure out how to proceed. The full change > >> would touch more than 100 files, especially item 3 above. It's > >> certainly possible to split it into a few PRs. > >> > >> What are your thoughts? > >> > >> Thanks, > >> Alex > >> > >> On Tue, Jul 28, 2026 at 12:22 PM Steve Loughran <[email protected]> > wrote: > >> > > >> > > >> > I'd go for > >> > > >> > leave jackson 2 as is, deprecate, with a goal to cut completely at > some point > >> > accompany an api which hides the json binding, > >> > Move to jackson 3 internally, never expose this, either remove the > jackson 2 apis or move to a iceberg-core-jackson2-legacy module for > expllicit use and make clear it's only there for one more release > >> > > >> > Maybe moving the jackson2 stuff to a legacy module from the outset > makes clear its doomed, but if there's one lesson from OSS, it's "unless > you are ruthless and make clear it will go, people don't believe you". > >> > At then same time: make changes incompatible at the binary level and > people don't upgrade to your latest release, so don't pick up the latest > releases, with direct and indirect CVE fixes. > >> > > >> > On that topic, there's a new jackson 2 release out: no release notes. > That implies "security fix but they don't want to announce the CVE just yet" > >> > > >> > > >> > On Mon, 27 Jul 2026 at 14:52, Alexandre Dutra <[email protected]> > wrote: > >> >> > >> >> Hi Steve, hi all, > >> >> > >> >> We discussed this topic briefly in the last sync, in the context of > >> >> iceberg-core specifically. Here is a quick summary: > >> >> > >> >> - The exposure of the Jackson 2 API in iceberg-core often appears > accidental. > >> >> > >> >> - If exposure is unavoidable, introducing dual exposure via method > >> >> overloads has been proposed. > >> >> > >> >> - Another option under consideration is providing two separate > >> >> artifacts, one for Jackson 2 and one for Jackson 3. > >> >> > >> >> - Ultimately, we need more time to evaluate the optimal path forward. > >> >> > >> >> To help guide this decision, I did a deeper investigation into where > >> >> these Jackson types actually leak out. The exposure broadly falls > into > >> >> four categories: > >> >> > >> >> 1) Parser classes (SchemaParser, MetadataUpdateParser, etc.) > >> >> 2) The JsonUtil utility class. > >> >> 3) The HTTPRequest.mapper() method. > >> >> 4) RESTSerializers.registerAll(ObjectMapper) and > >> >> HTTPClient.Builder.withObjectMapper(ObjectMapper). > >> >> > >> >> Based on these findings, I suggest the following approach for each > area: > >> >> > >> >> - For the parser classes (1), we should ideally restrict visibility > to > >> >> package-private if feasible (implies revapi exceptions). > >> >> > >> >> - For JsonUtil (2), I recommend either transitioning it to > >> >> package-private or deprecating it for removal. > >> >> > >> >> - For HTTPRequest.mapper() (3), the exposure was an unfortunate > design > >> >> choice: we should deprecate it for removal and re-implement the logic > >> >> within a package-private helper class. > >> >> > >> >> Item (4) represents the biggest challenge since consumers > legitimately > >> >> rely on them to configure their object mappers. Removing them > entirely > >> >> might be too disruptive; a better approach would be to accept the > >> >> tight coupling and introduce parallel Jackson 3 overloads instead. > Or, > >> >> as Steve mentions, add a layer of indirection to decouple the API > >> >> completely. > >> >> > >> >> In conclusion, if we put the extension points (4) aside for a moment, > >> >> it looks like we have a clear opportunity to perform some API cleanup > >> >> before we fully commit to the Jackson 3 migration, by reducing the > >> >> Jackson 2 exposure to its minimum. > >> >> > >> >> What do you all think? > >> >> > >> >> Thanks, > >> >> Alex > >> >> > >> >> On Mon, Jul 20, 2026 at 2:57 PM Steve Loughran <[email protected]> > wrote: > >> >> > > >> >> > > >> >> > the public APIs are the pain point, and adding jackson 3 overloads > is possibly going to replicate the same problem. > >> >> > > >> >> > I'd propose > >> >> > - identify those things where json parsing/generation takes place > and are published > >> >> > - see how they could extended such that jackson 3 supported could > be done without adding jackson 3 to their signatures > >> >> > > >> >> > I'm thinking of how the various parsers have toJson() methods > which take jackson generators > >> >> > > >> >> > public static void toJson(TableIdentifier identifier, > JsonGenerator generator) > >> >> > > >> >> > if another layer of indirection is added (JsonBinding?) that > supplied the generator then an overridden entry point could be supplied > >> >> > > >> >> > public static void toJson(TableIdentifier identifier, > JsonBinding binding) { > >> >> > if (binding instance of Jackson2Binding jackson2binding) { > >> >> > return toJson(identifier, jackson2binding.generator()); > >> >> > } else { > >> >> > return ((jackson3binding)bindiong)....) > >> >> > > >> >> > These could be added for the apis today, with the jackson2 stuff > tagged as deprecated, and new apis only coming though the new overload > >> >> > > >> >> > I'm glossing over the many implementation details here, but the > key thing is: exposing someone else's library as part your public API > inevitably becomes something you end up regretting. > >> >> > > >> >> > > >> >> > On Thu, 16 Jul 2026 at 15:53, Alexandre Dutra <[email protected]> > wrote: > >> >> >> > >> >> >> Hi all, > >> >> >> > >> >> >> As more frameworks (like Spring Boot and Quarkus) migrate to > Jackson > >> >> >> 3, I wanted to open a conversation about what a Jackson 3 > migration > >> >> >> would look like for Iceberg and what trade-offs we should think > >> >> >> through together. > >> >> >> > >> >> >> Iceberg currently depends on Jackson 2 in several modules, most > >> >> >> significantly iceberg-core. Other modules with direct Jackson > >> >> >> dependencies include iceberg-aws, iceberg-nessie, > iceberg-snowflake, > >> >> >> and iceberg-kafka-connect. Most modules could be migrated > separately. > >> >> >> > >> >> >> The migration is largely mechanical but pervasive: artifact > >> >> >> coordinates and Java package names change across the board, > several > >> >> >> core classes and methods are renamed, ObjectMapper construction > moves > >> >> >> to a builder pattern, and exceptions become unchecked. JDK 17 is > >> >> >> required, which Iceberg already enforces. > >> >> >> > >> >> >> The blast radius for iceberg-core would be roughly: > >> >> >> > >> >> >> - Spark and Flink runtime JARs: low impact. Both already shade > Jackson > >> >> >> 2; adding a relocation rule for Jackson 3 is a one-line fix per > >> >> >> runtime variant. > >> >> >> > >> >> >> - Kafka Connect: also low impact, although the Kafka Connect > >> >> >> distribution uses unshaded JARs. > >> >> >> > >> >> >> - Downstream users of iceberg-core: this is the most significant > >> >> >> concern. Jackson types leak into the public API of many XYZParser > >> >> >> classes, JsonUtil, and the REST HTTP layer. > >> >> >> > >> >> >> The honest answer is that icebegr-core's public API is permanently > >> >> >> Jackson-coupled by design. We should imho acknowledge this fact; > the > >> >> >> best migration approach is to add Jackson 3 overloads wherever > >> >> >> applicable, and make Jackson 2 and 3 coexist for some time. > >> >> >> > >> >> >> Looking forward to hearing your thoughts on this topic. > >> >> >> > >> >> >> Thanks, > >> >> >> Alex >
