Hi again, A correction and an update on option B.
I said around 19 classes hold a CatalogContext. On master it's 10 that hold one in a serialisable field: the four catalog loaders, CatalogEnvironment, FormatTableImpl, ResolvingFileIO, RESTTokenFileIO, UriReaderFactory and CDCSource. There is also one serialisable anonymous class that captures one (below). I've prototyped both options so they're easier to compare. A [1] moves the three methods to HadoopUtils. It's about 110 lines across 22 files, with a test that clones CatalogContext and a CatalogLoader while Hadoop classes are blocked. The test fails on master and passes with the change. This branch removes the methods outright. The deprecation step would be a separate, earlier PR. B [2] leaves CatalogContext as it is. A small carrier class holds the context's options, Hadoop conf and FileIO loaders, and the serialisable classes store that instead of the context, which is rebuilt after deserialisation. It's about 160 lines across 13 files, with a test that clones a catalog loader, a table and a UriReaderFactory while Hadoop classes are blocked. The existing catalog and table tests pass on both branches. B adds a public class where A removes methods. The bigger cost is that B is easy to break again. Every serialisable class that holds or captures a CatalogContext has to use the carrier instead. The compiler won't catch a mistake, and neither will tests, unless they happen to serialise that class without Hadoop. My first version of B missed one: BlobViewLookup returns a serialisable BlobViewResolver as an anonymous class, and it captures the context. The fix is a separate commit [3], but the next class written this way would bring the problem back, with nothing in the build to catch it. With A that can't happen, because CatalogContext itself serialises without Hadoop. Andreas [1] https://github.com/bube/paimon/compare/030b38f994...catalog-context-hadoop-free [2] https://github.com/bube/paimon/compare/3ffb4e4ce8...catalog-context-serial-holders [3] https://github.com/bube/paimon/commit/6389566d0dedc51ead7f25c3c8fd808ca02927eb On Wed, 28 Oct 2026 at 12:30, Andreas Bube <[email protected]> wrote: > > Hi all, > > Paimon jobs on Flink still need Hadoop on the classpath, even when the > file system doesn't use it, for example Flink 2.3's new native S3 file > system (FLIP-555 [1]). > > The first thing that breaks is CatalogContext. When Flink serializes > Paimon's sink to submit the job, Java loads every type that appears in > CatalogContext's method signatures. hadoopConf() returns a Hadoop > Configuration, so the job fails with NoClassDefFoundError even though > the context holds no Hadoop object. #8193 [2] made it possible to > create a context without Hadoop, but it still can't be serialized. > #6654 [3] > tracks this. > > You only see it on Flink, since Spark and Hive always ship Hadoop, but > the fix would be in paimon-common. > > #6653 [4] tried to fix it by moving the Hadoop methods into a > subclass, and the review asked that existing integrations not break. > As far as I can tell, any fix has to remove these three methods from > CatalogContext, since serialization fails as long as one of them is there: > > create(Options, Configuration) > create(Options, Configuration, FileIOLoader, FileIOLoader) > hadoopConf() > > paimon-trino, Gravitino, Doris, Impala and XTable call them, so I'd > like to agree on how to break them before opening a PR. > > My preference is to move them to HadoopUtils over two releases. In one > release we add HadoopUtils.createCatalogContext(...) and > HadoopUtils.hadoopConf(CatalogContext) and deprecate the old methods. > In the next we > remove the old methods. Callers change one line each. I have a > prototype of about 110 lines, with a test that serializes > CatalogContext and CatalogLoader while Hadoop classes are blocked. The > test fails on > master and passes with the change. Nothing changes when Hadoop is on > the classpath. > > The alternative is to leave CatalogContext alone and stop serializing > it. The classes that hold one would serialize its options, Hadoop conf > and FileIO loaders instead, and rebuild the context on read. Nothing > breaks for callers, but around 19 classes hold a context, so it's a > much bigger change. I haven't tried it. > > Would the two-release removal be acceptable, or would you rather keep > the API and take on the bigger change? > > This only covers the catalog side. Parquet and ORC still need Hadoop > too, but that's a separate discussion. > > Thanks, > Andreas > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-555:+Flink+Native+S3+FileSystem > [2] https://github.com/apache/paimon/pull/8193 > [3] https://github.com/apache/paimon/issues/6654 > [4] https://github.com/apache/paimon/pull/6653 > > > -- > Andreas Bube > Senior Software Engineer II > [email protected] > Landskronagade 66, 2100 København Ø, Denmark > toogoodtogo.com
