Hi Jingsong, Yes, I agree, we absolutely must deal with Parquet and any other usages Hadoop. My initial thought was to first deal with the paimon-core and keep formats etc. separate since one does not block the other.
I can imagine that dealing with file formats will be a much larger task - perhaps not in code, but primarily in terms of testing to ensure backwards and forwards compatibility in case we choose a new Parquet implementation (e.g. Hardwood [1]). Based on your feedback Jingsong, I'm left with two questions: 1. Should we first sketch out a full plan for dealing with all components before we start working on it? 2. What are your thoughts on the proposed solutions (A and B) to dealing with the Hadoop Configuration in paimon-core/CatalogContext? Best regards, Andreas [1] https://github.com/hardwood-hq/hardwood On Thu, 29 Oct 2026 at 02:01, Jingsong Li <[email protected]> wrote: > > Hi Andreas, > > Thanks for driving this! I think it is a reasonable direction. > > However, I also believe that the considerations regarding Parquet are > necessary; if we want to achieve complete isolation of all Hadoop > dependencies, we must take every component into account. > > Best, > Jingsong > > On Wed, Oct 28, 2026 at 9:11 PM Andreas Bube via dev > <[email protected]> wrote: > > > > Hi again, > > > > A correction and an update on option B. > > > > I said around 19 classes hold a CatalogContext. On master it's 10 that > > hold one in a serialisable field: the four catalog loaders, > > CatalogEnvironment, FormatTableImpl, ResolvingFileIO, RESTTokenFileIO, > > UriReaderFactory and CDCSource. There is also one serialisable > > anonymous class that captures one (below). > > > > I've prototyped both options so they're easier to compare. > > > > A [1] moves the three methods to HadoopUtils. It's about 110 lines > > across 22 files, with a test that clones CatalogContext and a > > CatalogLoader while Hadoop classes are blocked. The test fails on > > master and passes with the change. This branch removes the methods > > outright. The deprecation step would be a separate, earlier PR. > > > > B [2] leaves CatalogContext as it is. A small carrier class holds the > > context's options, Hadoop conf and FileIO loaders, and the > > serialisable classes store that instead of the context, which is > > rebuilt after deserialisation. It's about 160 lines across 13 files, > > with a test that clones a catalog loader, a table and a > > UriReaderFactory while Hadoop classes are blocked. The existing > > catalog and table tests pass on both branches. > > > > B adds a public class where A removes methods. The bigger cost is that > > B is easy to break again. Every serialisable class that holds or > > captures a CatalogContext has to use the carrier instead. The compiler > > won't catch a mistake, and neither will tests, unless they happen to > > serialise that class without Hadoop. My first version of B missed one: > > BlobViewLookup returns a serialisable BlobViewResolver as an anonymous > > class, and it captures the context. The fix is a separate commit [3], > > but the next class written this way would bring the problem back, with > > nothing in the build to catch it. With A that can't happen, because > > CatalogContext itself serialises without Hadoop. > > > > Andreas > > > > [1] > > https://github.com/bube/paimon/compare/030b38f994...catalog-context-hadoop-free > > [2] > > https://github.com/bube/paimon/compare/3ffb4e4ce8...catalog-context-serial-holders > > [3] > > https://github.com/bube/paimon/commit/6389566d0dedc51ead7f25c3c8fd808ca02927eb > > > > On Wed, 28 Oct 2026 at 12:30, Andreas Bube <[email protected]> wrote: > > > > > > Hi all, > > > > > > Paimon jobs on Flink still need Hadoop on the classpath, even when the > > > file system doesn't use it, for example Flink 2.3's new native S3 file > > > system (FLIP-555 [1]). > > > > > > The first thing that breaks is CatalogContext. When Flink serializes > > > Paimon's sink to submit the job, Java loads every type that appears in > > > CatalogContext's method signatures. hadoopConf() returns a Hadoop > > > Configuration, so the job fails with NoClassDefFoundError even though > > > the context holds no Hadoop object. #8193 [2] made it possible to > > > create a context without Hadoop, but it still can't be serialized. > > > #6654 [3] > > > tracks this. > > > > > > You only see it on Flink, since Spark and Hive always ship Hadoop, but > > > the fix would be in paimon-common. > > > > > > #6653 [4] tried to fix it by moving the Hadoop methods into a > > > subclass, and the review asked that existing integrations not break. > > > As far as I can tell, any fix has to remove these three methods from > > > CatalogContext, since serialization fails as long as one of them is there: > > > > > > create(Options, Configuration) > > > create(Options, Configuration, FileIOLoader, FileIOLoader) > > > hadoopConf() > > > > > > paimon-trino, Gravitino, Doris, Impala and XTable call them, so I'd > > > like to agree on how to break them before opening a PR. > > > > > > My preference is to move them to HadoopUtils over two releases. In one > > > release we add HadoopUtils.createCatalogContext(...) and > > > HadoopUtils.hadoopConf(CatalogContext) and deprecate the old methods. > > > In the next we > > > remove the old methods. Callers change one line each. I have a > > > prototype of about 110 lines, with a test that serializes > > > CatalogContext and CatalogLoader while Hadoop classes are blocked. The > > > test fails on > > > master and passes with the change. Nothing changes when Hadoop is on > > > the classpath. > > > > > > The alternative is to leave CatalogContext alone and stop serializing > > > it. The classes that hold one would serialize its options, Hadoop conf > > > and FileIO loaders instead, and rebuild the context on read. Nothing > > > breaks for callers, but around 19 classes hold a context, so it's a > > > much bigger change. I haven't tried it. > > > > > > Would the two-release removal be acceptable, or would you rather keep > > > the API and take on the bigger change? > > > > > > This only covers the catalog side. Parquet and ORC still need Hadoop > > > too, but that's a separate discussion. > > > > > > Thanks, > > > Andreas > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-555:+Flink+Native+S3+FileSystem > > > [2] https://github.com/apache/paimon/pull/8193 > > > [3] https://github.com/apache/paimon/issues/6654 > > > [4] https://github.com/apache/paimon/pull/6653 > > > > > > > > > -- > > > Andreas Bube > > > Senior Software Engineer II > > > [email protected] > > > Landskronagade 66, 2100 København Ø, Denmark > > > toogoodtogo.com
