Hi all, Following Jinsong's suggestion on PR #4378, I would like to gather feedback on the scope of Hadoop 2 support in Amoro.
https://github.com/apache/amoro/pull/4378 The PR upgrades AMS and optimizers to Iceberg 1.10.2 while keeping mixed-format connectors on Iceberg 1.8.1 to preserve existing engine support, including Spark 3.3 and Flink 1.18. The new Parquet dependency requires Hadoop APIs unavailable in Hadoop 2, so AMS and optimizers need a compatible Hadoop 3 runtime. The question is whether we should also discontinue Hadoop 2 support for mixed-format connectors. Amoro's hadoop2 Maven profile uses Hadoop 2.10.2. Although Hadoop 2.10 has not been officially declared EOL, its last release was in May 2022, over four years ago. Given this gap and the cost of maintaining separate build and test configurations, I think it is worth considering retiring Hadoop 2 support in Amoro. Either way, we need to retire Hadoop 2-based AMS distributions and update optimizer environments, including the Hadoop 2.8.3-based bundle in the Flink optimizer Docker image. For the connectors, I see two options: 1. Retain Hadoop 2 for mixed-format connectors only, maintaining both Hadoop 2 and Hadoop 3 connector build and test configurations. 2. Discontinue Hadoop 2 across Amoro in the next release, simplifying maintenance but requiring affected connector users to migrate. This is separate from the Iceberg version split: neither option would drop existing Spark or Flink versions, though their Hadoop 3 configurations would need validation. Does anyone still require Hadoop 2 for mixed-format connectors? Which option would the community prefer? I can update the implementation and migration documentation based on the feedback. Thanks, Jiwon
