+1 (non-binding) The Iceberg implementation of DataSourceV2 is passing all tests after updating to the 2.4 API, although I've had to disable ORC support because BufferHolder is no longer public.
One oddity is that the DSv2 API for batch sources now includes an epoch ID, which I think will be removed in the refactor before 2.5 or 3.0 and wasn't part of the 2.3 release. That's strange, but it's minor. rb On Tue, Oct 23, 2018 at 5:10 PM Sean Owen <sro...@gmail.com> wrote: > Hm, so you're trying to build a source release from a binary release? > I don't think that needs to work nor do I expect it to for reasons > like this. They just have fairly different things. > > On Tue, Oct 23, 2018 at 7:04 PM Dongjoon Hyun <dongjoon.h...@gmail.com> > wrote: > > > > Ur, Wenchen. > > > > Source distribution seems to fail by default. > > > > > https://dist.apache.org/repos/dist/dev/spark/v2.4.0-rc4-bin/spark-2.4.0.tgz > > > > $ dev/make-distribution.sh -Pyarn -Phadoop-2.7 -Pkinesis-asl -Phive > -Phive-thriftserver > > ... > > + cp /spark-2.4.0/LICENSE-binary /spark-2.4.0/dist/LICENSE > > cp: /spark-2.4.0/LICENSE-binary: No such file or directory > > > > > > The root cause seems to be the following fix. > > > > > https://github.com/apache/spark/pull/22436/files#diff-01ca42240614718522afde4d4885b40dR175 > > > > Although Apache Spark provides the binary distributions, it would be > great if this succeeds out of the box. > > > > Bests, > > Dongjoon. > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > > -- Ryan Blue Software Engineer Netflix
