Hi Jiwon, I'd vote for Option 2 — discontinuing Hadoop 2 across Amoro in the next release.
Key reasons: 1. Hadoop 3 clients are backward-compatible with Hadoop 2 clusters. Hadoop 3 RPC can connect to Hadoop 2 NameNodes, and Hive 3 metastore client can talk to Hive 2 metastore servers. Users on Hadoop 2 clusters can still use connectors built with Hadoop 3 — the change is on the build side, not a forced cluster upgrade. 2. Hadoop 2 and Hive 2 are coupled in our build. The hadoop2 profile sets both together, and the Trino module pins Hive 2.1.1 with CI running only under hadoop2. Keeping it means maintaining the entire chain. 3. AMS already requires Hadoop 3. Keeping Hadoop 2 only for connectors creates a mixed config that's hard to test end-to-end — worse than a clean break. 4. Hadoop 2.10 hasn't had a release in 4+ years. The ecosystem has moved on. If we want to be slightly conservative, we could mark it deprecated next release and remove in the following one. But I'm fine with a clean break either way. Happy to help with migration docs if we go with Option 2. Best, Jinsong On Mon, Sep 21, 2026 at 3:42 PM Xavier Bai <[email protected]> wrote: > +1 for discontinue Hadoop 2 in the next release > > Jiwon Park <[email protected]> 于2026年9月20日周日 19:30写道： > > > Hi all, > > > > Following Jinsong's suggestion on PR #4378, I would like to gather > feedback > > on the scope of Hadoop 2 support in Amoro. > > > > https://github.com/apache/amoro/pull/4378 > > > > The PR upgrades AMS and optimizers to Iceberg 1.10.2 while keeping > > mixed-format connectors on Iceberg 1.8.1 to preserve existing engine > > support, including Spark 3.3 and Flink 1.18. > > > > The new Parquet dependency requires Hadoop APIs unavailable in Hadoop 2, > so > > AMS and optimizers need a compatible Hadoop 3 runtime. The question is > > whether we should also discontinue Hadoop 2 support for mixed-format > > connectors. > > > > Amoro's hadoop2 Maven profile uses Hadoop 2.10.2. Although Hadoop 2.10 > has > > not been officially declared EOL, its last release was in May 2022, over > > four years ago. Given this gap and the cost of maintaining separate build > > and test configurations, I think it is worth considering retiring Hadoop > 2 > > support in Amoro. > > > > Either way, we need to retire Hadoop 2-based AMS distributions and update > > optimizer environments, including the Hadoop 2.8.3-based bundle in the > > Flink optimizer Docker image. > > > > For the connectors, I see two options: > > > > 1. Retain Hadoop 2 for mixed-format connectors only, maintaining both > > Hadoop 2 and Hadoop 3 connector build and test configurations. > > 2. Discontinue Hadoop 2 across Amoro in the next release, simplifying > > maintenance but requiring affected connector users to migrate. > > > > This is separate from the Iceberg version split: neither option would > drop > > existing Spark or Flink versions, though their Hadoop 3 configurations > > would need validation. > > > > Does anyone still require Hadoop 2 for mixed-format connectors? Which > > option would the community prefer? > > > > I can update the implementation and migration documentation based on the > > feedback. > > > > Thanks, > > Jiwon > > >
