Hi Fokko, Ryan, Netflix is still on Spark-2.4.4 with Iceberg-0.9. We are actively migrating to Spark-3.x and Iceberg 1.1 (or later). I do not anticipate us using Spark-2.4.4 with newer versions of Iceberg (>0.9). If the plan is to not support Spark-2.4.4 with Iceberg >= 1.X, that should be ok. @John Zhuge <jzh...@netflix.com> can you please chime in?
thanks, Anjali On Fri, Apr 14, 2023 at 10:56 AM Ryan Blue <b...@tabular.io> wrote: > Overall I'm +1, but could be convinced otherwise. > > Spark 2.4 is old and doesn't really function properly because the Spark > Catalog API was missing at the time. And people can still use older > versions of Iceberg that support Spark 2.4 if they need it because the > Iceberg spec guarantees forward compatibility. > > That said, I'd love to hear from more people on this. I think it would be > great to drop support, but I don't know how many people still use it. Is > upgrading Hadoop a good reason to drop support for an engine? Hadoop seems > like a minor concern to me unless it is blocking something. > > Ryan > > On Thu, Apr 13, 2023 at 12:54 PM Jack Ye <yezhao...@gmail.com> wrote: > >> +1 for dropping 2.4 support >> >> Best, >> Jack Ye >> >> On Thu, Apr 13, 2023 at 10:59 AM Fokko Driesprong <fo...@apache.org> >> wrote: >> >>> Hi all, >>> >>> I'm working on moving to Hadoop 3.x >>> <https://github.com/apache/iceberg/pull/7114>, and one thing is that it >>> seems to be incompatible with Spark 2.4. I wanted to ask if people are >>> still on Spark 2.4 and what we think of dropping the support. The last >>> release of Spark 2.4.8 was on 2021-05-17 and it also looks like the 2.4 >>> branch on the Spark Github repository is stale, so I don't expect any >>> further releases. >>> >>> Before creating a PR I would like to check on the mail-list if anyone >>> has any objections. If so, please let us know. >>> >>> Thanks, >>> Fokko Driesprong >>> >> > > -- > Ryan Blue > Tabular >
