Hi all, Paimon jobs on Flink still need Hadoop on the classpath, even when the file system doesn't use it, for example Flink 2.3's new native S3 file system (FLIP-555 [1]).
The first thing that breaks is CatalogContext. When Flink serializes Paimon's sink to submit the job, Java loads every type that appears in CatalogContext's method signatures. hadoopConf() returns a Hadoop Configuration, so the job fails with NoClassDefFoundError even though the context holds no Hadoop object. #8193 [2] made it possible to create a context without Hadoop, but it still can't be serialized. #6654 [3] tracks this. You only see it on Flink, since Spark and Hive always ship Hadoop, but the fix would be in paimon-common. #6653 [4] tried to fix it by moving the Hadoop methods into a subclass, and the review asked that existing integrations not break. As far as I can tell, any fix has to remove these three methods from CatalogContext, since serialization fails as long as one of them is there: create(Options, Configuration) create(Options, Configuration, FileIOLoader, FileIOLoader) hadoopConf() paimon-trino, Gravitino, Doris, Impala and XTable call them, so I'd like to agree on how to break them before opening a PR. My preference is to move them to HadoopUtils over two releases. In one release we add HadoopUtils.createCatalogContext(...) and HadoopUtils.hadoopConf(CatalogContext) and deprecate the old methods. In the next we remove the old methods. Callers change one line each. I have a prototype of about 110 lines, with a test that serializes CatalogContext and CatalogLoader while Hadoop classes are blocked. The test fails on master and passes with the change. Nothing changes when Hadoop is on the classpath. The alternative is to leave CatalogContext alone and stop serializing it. The classes that hold one would serialize its options, Hadoop conf and FileIO loaders instead, and rebuild the context on read. Nothing breaks for callers, but around 19 classes hold a context, so it's a much bigger change. I haven't tried it. Would the two-release removal be acceptable, or would you rather keep the API and take on the bigger change? This only covers the catalog side. Parquet and ORC still need Hadoop too, but that's a separate discussion. Thanks, Andreas [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-555:+Flink+Native+S3+FileSystem [2] https://github.com/apache/paimon/pull/8193 [3] https://github.com/apache/paimon/issues/6654 [4] https://github.com/apache/paimon/pull/6653 -- Andreas Bube Senior Software Engineer II [email protected] Landskronagade 66, 2100 København Ø, Denmark toogoodtogo.com
