Hi Andreas, Thanks for driving this! I think it is a reasonable direction.
However, I also believe that the considerations regarding Parquet are necessary; if we want to achieve complete isolation of all Hadoop dependencies, we must take every component into account. Best, Jingsong On Wed, Oct 28, 2026 at 9:11 PM Andreas Bube via dev <[email protected]> wrote: > > Hi again, > > A correction and an update on option B. > > I said around 19 classes hold a CatalogContext. On master it's 10 that > hold one in a serialisable field: the four catalog loaders, > CatalogEnvironment, FormatTableImpl, ResolvingFileIO, RESTTokenFileIO, > UriReaderFactory and CDCSource. There is also one serialisable > anonymous class that captures one (below). > > I've prototyped both options so they're easier to compare. > > A [1] moves the three methods to HadoopUtils. It's about 110 lines > across 22 files, with a test that clones CatalogContext and a > CatalogLoader while Hadoop classes are blocked. The test fails on > master and passes with the change. This branch removes the methods > outright. The deprecation step would be a separate, earlier PR. > > B [2] leaves CatalogContext as it is. A small carrier class holds the > context's options, Hadoop conf and FileIO loaders, and the > serialisable classes store that instead of the context, which is > rebuilt after deserialisation. It's about 160 lines across 13 files, > with a test that clones a catalog loader, a table and a > UriReaderFactory while Hadoop classes are blocked. The existing > catalog and table tests pass on both branches. > > B adds a public class where A removes methods. The bigger cost is that > B is easy to break again. Every serialisable class that holds or > captures a CatalogContext has to use the carrier instead. The compiler > won't catch a mistake, and neither will tests, unless they happen to > serialise that class without Hadoop. My first version of B missed one: > BlobViewLookup returns a serialisable BlobViewResolver as an anonymous > class, and it captures the context. The fix is a separate commit [3], > but the next class written this way would bring the problem back, with > nothing in the build to catch it. With A that can't happen, because > CatalogContext itself serialises without Hadoop. > > Andreas > > [1] > https://github.com/bube/paimon/compare/030b38f994...catalog-context-hadoop-free > [2] > https://github.com/bube/paimon/compare/3ffb4e4ce8...catalog-context-serial-holders > [3] > https://github.com/bube/paimon/commit/6389566d0dedc51ead7f25c3c8fd808ca02927eb > > On Wed, 28 Oct 2026 at 12:30, Andreas Bube <[email protected]> wrote: > > > > Hi all, > > > > Paimon jobs on Flink still need Hadoop on the classpath, even when the > > file system doesn't use it, for example Flink 2.3's new native S3 file > > system (FLIP-555 [1]). > > > > The first thing that breaks is CatalogContext. When Flink serializes > > Paimon's sink to submit the job, Java loads every type that appears in > > CatalogContext's method signatures. hadoopConf() returns a Hadoop > > Configuration, so the job fails with NoClassDefFoundError even though > > the context holds no Hadoop object. #8193 [2] made it possible to > > create a context without Hadoop, but it still can't be serialized. > > #6654 [3] > > tracks this. > > > > You only see it on Flink, since Spark and Hive always ship Hadoop, but > > the fix would be in paimon-common. > > > > #6653 [4] tried to fix it by moving the Hadoop methods into a > > subclass, and the review asked that existing integrations not break. > > As far as I can tell, any fix has to remove these three methods from > > CatalogContext, since serialization fails as long as one of them is there: > > > > create(Options, Configuration) > > create(Options, Configuration, FileIOLoader, FileIOLoader) > > hadoopConf() > > > > paimon-trino, Gravitino, Doris, Impala and XTable call them, so I'd > > like to agree on how to break them before opening a PR. > > > > My preference is to move them to HadoopUtils over two releases. In one > > release we add HadoopUtils.createCatalogContext(...) and > > HadoopUtils.hadoopConf(CatalogContext) and deprecate the old methods. > > In the next we > > remove the old methods. Callers change one line each. I have a > > prototype of about 110 lines, with a test that serializes > > CatalogContext and CatalogLoader while Hadoop classes are blocked. The > > test fails on > > master and passes with the change. Nothing changes when Hadoop is on > > the classpath. > > > > The alternative is to leave CatalogContext alone and stop serializing > > it. The classes that hold one would serialize its options, Hadoop conf > > and FileIO loaders instead, and rebuild the context on read. Nothing > > breaks for callers, but around 19 classes hold a context, so it's a > > much bigger change. I haven't tried it. > > > > Would the two-release removal be acceptable, or would you rather keep > > the API and take on the bigger change? > > > > This only covers the catalog side. Parquet and ORC still need Hadoop > > too, but that's a separate discussion. > > > > Thanks, > > Andreas > > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-555:+Flink+Native+S3+FileSystem > > [2] https://github.com/apache/paimon/pull/8193 > > [3] https://github.com/apache/paimon/issues/6654 > > [4] https://github.com/apache/paimon/pull/6653 > > > > > > -- > > Andreas Bube > > Senior Software Engineer II > > [email protected] > > Landskronagade 66, 2100 København Ø, Denmark > > toogoodtogo.com
