I feel like this shouldn't be a big problem going forward since all new tables will be using the V2 format where the snapshot ID inheritance is enabled. There is currently a bug in our rewrite manifests action that checks the flag but doesn't check the format version. I have a fix for that locally, I'll submit it by the end of tomorrow.
On 2023/08/31 17:10:54 Pucheng Yang wrote: > Thanks Rayn, then I think I have a path forward. And I will file a feature > request on thread-safe appendManifest on github. Thanks again. > > On Thu, Aug 31, 2023 at 10:04 AM Ryan Blue <b...@tabular.io> wrote: > > > I think making the operation thread-safe and parallelizing is a good idea. > > It should be pretty easy. > > > > And yes, versions of Iceberg older than the one where that config property > > was added would be the ones where it is unsafe. It's probably safe for most > > people, but we still can't change the default. > > > > On Thu, Aug 31, 2023 at 9:59 AM Pucheng Yang <py...@pinterest.com.invalid> > > wrote: > > > >> Thanks Ryan, what might you consider an "older" version of Iceberg? Is it > >> fair to say any version before > >> https://github.com/apache/iceberg/commit/c3dc9824b381e5e479e356be5e0f4fcf61a9fc37 > >> ? If that is the case, my organization controls the Iceberg reader so might > >> be a less concern for me. > >> > >> Another option is to make appendManifests operation thread-safe so that > >> we can parallel the execution when needed (also I think we should add > >> comment to indicate appendManifests API is not thread-safe, this part seems > >> missing, slack discussion: > >> https://apache-iceberg.slack.com/archives/C03LG1D563F/p1693410897758349?thread_ts=1693355747.700759&cid=C03LG1D563F > >> ) > >> > >>> > > > > -- > > Ryan Blue > > Tabular > > >
