I ran tests on production data that have file size 10K~200M with CRC checksum enabled and disabled as a comparison. There is no significant difference is seen.
CRC Enabled Write used time: 14371 ms CRC Disabled Write used time: 14355 ms On Thu, Dec 5, 2019 at 11:09 AM Julien Le Dem <[email protected]> wrote: > I verified the signatures > ran the build and test > It looks like the compatibility changes being discussed are not blockers. > > +1 (binding) > > > On Wed, Nov 27, 2019 at 1:43 AM Gabor Szadovszky <[email protected]> wrote: > > > Thanks, Zoltan. > > > > I also vote +1 (binding) > > > > Cheers, > > Gabor > > > > On Tue, Nov 26, 2019 at 1:46 PM Zoltan Ivanfi <[email protected]> > > wrote: > > > > > +1 (binding) > > > > > > - I have read through the problem reports in this e-mail thread (one > > caused > > > by the use of a private method via reflection an another one caused by > > > having mixed versions of the libraries on the classpath) and I am > > convinced > > > that they do not block the release. > > > - Signature and hash of the source tarball are valid. > > > - The specified git hash matches the specified git tag. > > > - The contents of the source tarball match the contents of the git repo > > at > > > the specified tag. > > > > > > Br, > > > > > > Zoltan > > > > > > > > > On Tue, Nov 26, 2019 at 10:54 AM Gabor Szadovszky <[email protected]> > > > wrote: > > > > > > > Created > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_PARQUET-2D1703&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=sgUj-DJ6A0vGBMalFBKeIy_8PwUeNX39K31GnzWgcsI&e= > to track > > > this. > > > > > > > > Back to the RC. Anyone from the PMC willing to vote? > > > > > > > > Cheers, > > > > Gabor > > > > > > > > On Mon, Nov 25, 2019 at 6:45 PM Ryan Blue <[email protected] > > > > > > wrote: > > > > > > > > > Gabor, good point about not being able to check new APIs. Updating > > the > > > > > previous version would also allow us to get rid of temporary > > > exclusions, > > > > > like the one you pointed out for schema. It would be great to > improve > > > > what > > > > > we catch there. > > > > > > > > > > On Mon, Nov 25, 2019 at 1:56 AM Gabor Szadovszky <[email protected] > > > > > > wrote: > > > > > > > > > > > Hi Ryan, > > > > > > > > > > > > It is a different topic but would like to reflect shortly. > > > > > > I understand that 1.7.0 was the first apache release. The problem > > > with > > > > > > doing the compatibility checks comparing to 1.7.0 is that we can > > > easily > > > > > add > > > > > > incompatibilities in API which are added after 1.7.0. For > example: > > > > > Adding a > > > > > > new class for public use in 1.8.0 then removing it in 1.9.0. The > > > > > > compatibility check would not discover this breaking change. So, > I > > > > > think, a > > > > > > better approach would be to always compare to the previous minor > > > > release > > > > > > (e.g. comparing 1.9.0 to 1.8.0 etc.). > > > > > > As I've written before, even org/apache/parquet/schema/** is > > excluded > > > > > from > > > > > > the compatibility check. As far as I know this is public API. > So, I > > > am > > > > > not > > > > > > sure that only packages that are not part of the public API are > > > > excluded. > > > > > > > > > > > > Let's discuss this on the next parquet sync. > > > > > > > > > > > > Regards, > > > > > > Gabor > > > > > > > > > > > > On Fri, Nov 22, 2019 at 6:20 PM Ryan Blue > > <[email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > Gabor, > > > > > > > > > > > > > > 1.7.0 was the first version using the org.apache.parquet > > packages, > > > so > > > > > > > that's the correct base version for compatibility checks. The > > > > > exclusions > > > > > > in > > > > > > > the POM are classes that the Parquet community does not > consider > > > > > public. > > > > > > We > > > > > > > rely on these checks to highlight binary incompatibilities, and > > > then > > > > we > > > > > > > discuss them on this list or in the dev sync. If the class is > > > > internal, > > > > > > we > > > > > > > add an exclusion for it. > > > > > > > > > > > > > > I know you're familiar with this process since we've talked > about > > > it > > > > > > > before. I also know that you'd rather have more strict binary > > > > > > > compatibility, but until we have someone with the time to do > some > > > > > > > maintenance and build a public API module, I'm afraid that's > what > > > we > > > > > have > > > > > > > to work with. > > > > > > > > > > > > > > Michael, > > > > > > > > > > > > > > I hope the context above is helpful and explains why running a > > > binary > > > > > > > compatibility check tool will find incompatible changes. We > allow > > > > > binary > > > > > > > incompatible changes to internal classes and modules, like > > > > > > parquet-common. > > > > > > > > > > > > > > On Fri, Nov 22, 2019 at 12:23 AM Gabor Szadovszky < > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Ryan, > > > > > > > > I would not trust our compatibility checks (semver) too much. > > > > > > Currently, > > > > > > > it > > > > > > > > is configured to compare our current version to 1.7.0. It > means > > > > > > anything > > > > > > > > that is added since 1.7.0 and then broke in a later release > > won't > > > > be > > > > > > > > caught. In addition, many packages are excluded from the > check > > > > > because > > > > > > of > > > > > > > > different reasons. For example org/apache/parquet/schema/** > is > > > > > excluded > > > > > > > so > > > > > > > > if it would really be an API compatibility issue we certainly > > > > > wouldn't > > > > > > > > catch it. > > > > > > > > > > > > > > > > Michael, > > > > > > > > It fails because of a NoSuchMethodError pointing to a method > > that > > > > is > > > > > > > newly > > > > > > > > introduced in 1.11. Both the caller and the callee shipped by > > > > > > parquet-mr. > > > > > > > > So, I'm quite sure it is a classpath issue. It seems that the > > > 1.11 > > > > > > > version > > > > > > > > of the parquet-column jar is not on the classpath. > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Nov 22, 2019 at 1:44 AM Michael Heuer < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > The dependency versions are consistent in our artifact > > > > > > > > > > > > > > > > > > $ mvn dependency:tree | grep parquet > > > > > > > > > [INFO] | \- > > org.apache.parquet:parquet-avro:jar:1.11.0:compile > > > > > > > > > [INFO] | \- > > > > > > > > > > > org.apache.parquet:parquet-format-structures:jar:1.11.0:compile > > > > > > > > > [INFO] | +- > > > org.apache.parquet:parquet-column:jar:1.11.0:compile > > > > > > > > > [INFO] | | +- > > > > > org.apache.parquet:parquet-common:jar:1.11.0:compile > > > > > > > > > [INFO] | | \- > > > > > > org.apache.parquet:parquet-encoding:jar:1.11.0:compile > > > > > > > > > [INFO] | +- > > > org.apache.parquet:parquet-hadoop:jar:1.11.0:compile > > > > > > > > > [INFO] | | +- > > > > > org.apache.parquet:parquet-jackson:jar:1.11.0:compile > > > > > > > > > > > > > > > > > > The latter error > > > > > > > > > > > > > > > > > > Caused by: org.apache.spark.SparkException: Job aborted due > > to > > > > > stage > > > > > > > > > failure: Task 0 in stage 0.0 failed 1 times, most recent > > > failure: > > > > > > Lost > > > > > > > > task > > > > > > > > > 0.0 in stage 0.0 (TID 0, localhost, executor driver): > > > > > > > > > java.lang.NoSuchMethodError: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.schema.Types$PrimitiveBuilder.as(Lorg/apache/parquet/schema/LogicalTypeAnnotation;)Lorg/apache/parquet/schema/Types$Builder; > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroSchemaConverter.convertField( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroSchemaConverter.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=fPqvqtlS18V1SPu-P8rpbffYhgNzHwYy27UAQfp82y8&e= > :161) > > > > > > > > > > > > > > > > > > occurs when I attempt to run via spark-submit on Spark > 2.4.4 > > > > > > > > > > > > > > > > > > $ spark-submit --version > > > > > > > > > Welcome to > > > > > > > > > ____ __ > > > > > > > > > / __/__ ___ _____/ /__ > > > > > > > > > _\ \/ _ \/ _ `/ __/ '_/ > > > > > > > > > /___/ .__/\_,_/_/ /_/\_\ version 2.4.4 > > > > > > > > > /_/ > > > > > > > > > > > > > > > > > > Using Scala version 2.11.12, Java HotSpot(TM) 64-Bit Server > > VM, > > > > > > > 1.8.0_191 > > > > > > > > > Branch > > > > > > > > > Compiled by user on 2019-08-27T21:21:38Z > > > > > > > > > Revision > > > > > > > > > Url > > > > > > > > > Type --help for more information. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Nov 21, 2019, at 6:06 PM, Ryan Blue > > > > <[email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > Thanks for looking into it, Nandor. That doesn't sound > > like a > > > > > > problem > > > > > > > > > with > > > > > > > > > > Parquet, but a problem with the test environment since > > > > > parquet-avro > > > > > > > > > depends > > > > > > > > > > on a newer API method. > > > > > > > > > > > > > > > > > > > > On Thu, Nov 21, 2019 at 3:58 PM Nandor Kollar > > > > > > > > > <[email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> I'm not sure that this is a binary compatibility issue. > > The > > > > > > missing > > > > > > > > > builder > > > > > > > > > >> method was recently added in 1.11.0 with the > introduction > > of > > > > the > > > > > > new > > > > > > > > > >> logical type API, while the original version (one with a > > > > single > > > > > > > > > >> OriginalType input parameter called before by > > > > > AvroSchemaConverter) > > > > > > > of > > > > > > > > > this > > > > > > > > > >> method is kept untouched. It seems to me that the > Parquet > > > > > version > > > > > > on > > > > > > > > > Spark > > > > > > > > > >> executor mismatch: parquet-avro is on 1.11.0, but > > > > parquet-column > > > > > > is > > > > > > > > > still > > > > > > > > > >> on an older version. > > > > > > > > > >> > > > > > > > > > >> On Thu, Nov 21, 2019 at 11:41 PM Michael Heuer < > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >>> Perhaps not strictly necessary to say, but if this > > > particular > > > > > > > > > >>> compatibility break between 1.10 and 1.11 was > > intentional, > > > > and > > > > > no > > > > > > > > other > > > > > > > > > >>> compatibility breaks are found, I would vote -1 > > > (non-binding) > > > > > on > > > > > > > this > > > > > > > > > RC > > > > > > > > > >>> such that we might go back and revisit the changes to > > > > preserve > > > > > > > > > >>> compatibility. > > > > > > > > > >>> > > > > > > > > > >>> I am not sure there is presently enough motivation in > the > > > > Spark > > > > > > > > project > > > > > > > > > >>> for a release after 2.4.4 and before 3.0 in which to > bump > > > the > > > > > > > Parquet > > > > > > > > > >>> dependency version to 1.11.x. > > > > > > > > > >>> > > > > > > > > > >>> michael > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>>> On Nov 21, 2019, at 11:01 AM, Ryan Blue > > > > > > <[email protected] > > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > >>>> > > > > > > > > > >>>> Gabor, shouldn't Parquet be binary compatible for > public > > > > APIs? > > > > > > > From > > > > > > > > > the > > > > > > > > > >>>> stack trace, it looks like this 1.11.0 RC breaks > binary > > > > > > > > compatibility > > > > > > > > > >> in > > > > > > > > > >>>> the type builders. > > > > > > > > > >>>> > > > > > > > > > >>>> Looks like this should have been caught by the binary > > > > > > > compatibility > > > > > > > > > >>> checks. > > > > > > > > > >>>> > > > > > > > > > >>>> On Thu, Nov 21, 2019 at 8:56 AM Gabor Szadovszky < > > > > > > > [email protected]> > > > > > > > > > >>> wrote: > > > > > > > > > >>>> > > > > > > > > > >>>>> Hi Michael, > > > > > > > > > >>>>> > > > > > > > > > >>>>> Unfortunately, I don't have too much experience on > > Spark. > > > > But > > > > > > if > > > > > > > > > spark > > > > > > > > > >>> uses > > > > > > > > > >>>>> the parquet-mr library in an embedded way (that's how > > > Hive > > > > > uses > > > > > > > it) > > > > > > > > > it > > > > > > > > > >>> is > > > > > > > > > >>>>> required to re-build Spark with 1.11 RC parquet-mr. > > > > > > > > > >>>>> > > > > > > > > > >>>>> Regards, > > > > > > > > > >>>>> Gabor > > > > > > > > > >>>>> > > > > > > > > > >>>>> On Wed, Nov 20, 2019 at 5:44 PM Michael Heuer < > > > > > > [email protected] > > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > >>>>> > > > > > > > > > >>>>>> It appears a provided scope dependency on spark-sql > > > leaks > > > > > old > > > > > > > > > parquet > > > > > > > > > >>>>>> versions was causing the runtime error below. After > > > > > including > > > > > > > new > > > > > > > > > >>>>>> parquet-column and parquet-hadoop compile scope > > > > dependencies > > > > > > (in > > > > > > > > > >>> addition > > > > > > > > > >>>>>> to parquet-avro, which we already have at compile > > > scope), > > > > > our > > > > > > > > build > > > > > > > > > >>>>>> succeeds. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bigdatagenomics_adam_pull_2232&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=QR-gkcFEuOecntHA5v79xckD1k54e3Hcu7j0RiI78kE&e= > < > > > > > > > > > >>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_bigdatagenomics_adam_pull_2232&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=QR-gkcFEuOecntHA5v79xckD1k54e3Hcu7j0RiI78kE&e= > > > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> However, when running via spark-submit I run into a > > > > similar > > > > > > > > runtime > > > > > > > > > >>> error > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Caused by: java.lang.NoSuchMethodError: > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.schema.Types$PrimitiveBuilder.as(Lorg/apache/parquet/schema/LogicalTypeAnnotation;)Lorg/apache/parquet/schema/Types$Builder; > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroSchemaConverter.convertField( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroSchemaConverter.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=fPqvqtlS18V1SPu-P8rpbffYhgNzHwYy27UAQfp82y8&e= > :161) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroSchemaConverter.convertUnion( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroSchemaConverter.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=fPqvqtlS18V1SPu-P8rpbffYhgNzHwYy27UAQfp82y8&e= > :226) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroSchemaConverter.convertField( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroSchemaConverter.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=fPqvqtlS18V1SPu-P8rpbffYhgNzHwYy27UAQfp82y8&e= > :182) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroSchemaConverter.convertField( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroSchemaConverter.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=fPqvqtlS18V1SPu-P8rpbffYhgNzHwYy27UAQfp82y8&e= > :141) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroSchemaConverter.convertField( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroSchemaConverter.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=fPqvqtlS18V1SPu-P8rpbffYhgNzHwYy27UAQfp82y8&e= > :244) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroSchemaConverter.convertFields( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroSchemaConverter.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=fPqvqtlS18V1SPu-P8rpbffYhgNzHwYy27UAQfp82y8&e= > :135) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroSchemaConverter.convert( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroSchemaConverter.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=fPqvqtlS18V1SPu-P8rpbffYhgNzHwYy27UAQfp82y8&e= > :126) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroWriteSupport.init( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroWriteSupport.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=AnW5dIuzP3qugBtSelhftpM9wJ6qmyf3Jh4UOe84g2M&e= > :121) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ParquetOutputFormat.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=BphWJojY_H6zmfK7_APdH7EnAqPnbpe8HQR5c6FCdqc&e= > :388) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ParquetOutputFormat.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=BphWJojY_H6zmfK7_APdH7EnAqPnbpe8HQR5c6FCdqc&e= > :349) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.InstrumentedOutputFormat.getRecordWriter(InstrumentedOutputFormat.scala:35) > > > > > > > > > >>>>>> at > https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.internal.io&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=oUUc68R-A0q141eyiEb8tSiXUnMlK1RJQAFk2Y0AcC8&e= > > > > > > > > > >>>>>> > > > > > > > > > >> > > > > > > > > > > > > > > > .HadoopMapReduceWriteConfigUtil.initWriter(SparkHadoopWriter.scala:350) > > > > > > > > > >>>>>> at > https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.internal.io&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=oUUc68R-A0q141eyiEb8tSiXUnMlK1RJQAFk2Y0AcC8&e= > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > .SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:120) > > > > > > > > > >>>>>> at > https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.internal.io&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=oUUc68R-A0q141eyiEb8tSiXUnMlK1RJQAFk2Y0AcC8&e= > > > > > > > > > >>>>>> > > > > > > .SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83) > > > > > > > > > >>>>>> at > https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.internal.io&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=oUUc68R-A0q141eyiEb8tSiXUnMlK1RJQAFk2Y0AcC8&e= > > > > > > > > > >>>>>> > > > > > > .SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) > > > > > > > > > >>>>>> at > > > > org.apache.spark.scheduler.Task.run(Task.scala:123) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ThreadPoolExecutor.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=vqAmsqMs16YnJIzOu5RXbKPUar7cxKcZMYDAMmk5y8o&e= > :1149) > > > > > > > > > >>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ThreadPoolExecutor.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=vqAmsqMs16YnJIzOu5RXbKPUar7cxKcZMYDAMmk5y8o&e= > :624) > > > > > > > > > >>>>>> at java.lang.Thread.run( > https://urldefense.proofpoint.com/v2/url?u=http-3A__Thread.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=AIV_giot2889YjBVYzI9kK6hRjD7_zW6pluPyj7E9jg&e= > :748) > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Will bumping our library dependency version to 1.11 > > > > require > > > > > a > > > > > > > new > > > > > > > > > >>> version > > > > > > > > > >>>>>> of Spark, built against Parquet 1.11? > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Please accept my apologies if this is heading > > > out-of-scope > > > > > for > > > > > > > the > > > > > > > > > >>>>> Parquet > > > > > > > > > >>>>>> mailing list. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> michael > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>>> On Nov 20, 2019, at 10:00 AM, Michael Heuer < > > > > > > [email protected] > > > > > > > > > > > > > > > > > >>> wrote: > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> I am willing to do some benchmarking on genomic > data > > at > > > > > scale > > > > > > > but > > > > > > > > > am > > > > > > > > > >>>>> not > > > > > > > > > >>>>>> quite sure what the Spark target version for 1.11.0 > > > might > > > > > be. > > > > > > > > Will > > > > > > > > > >>>>> Parquet > > > > > > > > > >>>>>> 1.11.0 be compatible in Spark 2.4.x? > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> Updating from 1.10.1 to 1.11.0 breaks at runtime in > > our > > > > > build > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> … > > > > > > > > > >>>>>>> D 0, localhost, executor driver): > > > > > > > java.lang.NoClassDefFoundError: > > > > > > > > > >>>>>> org/apache/parquet/schema/LogicalTypeAnnotation > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.avro.AvroWriteSupport.init( > https://urldefense.proofpoint.com/v2/url?u=http-3A__AvroWriteSupport.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=AnW5dIuzP3qugBtSelhftpM9wJ6qmyf3Jh4UOe84g2M&e= > :121) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ParquetOutputFormat.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=BphWJojY_H6zmfK7_APdH7EnAqPnbpe8HQR5c6FCdqc&e= > :388) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ParquetOutputFormat.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=BphWJojY_H6zmfK7_APdH7EnAqPnbpe8HQR5c6FCdqc&e= > :349) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.rdd.InstrumentedOutputFormat.getRecordWriter(InstrumentedOutputFormat.scala:35) > > > > > > > > > >>>>>>> at > https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.internal.io&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=oUUc68R-A0q141eyiEb8tSiXUnMlK1RJQAFk2Y0AcC8&e= > > > > > > > > > >>>>>> > > > > > > > > > >> > > > > > > > > > > > > > > > .HadoopMapReduceWriteConfigUtil.initWriter(SparkHadoopWriter.scala:350) > > > > > > > > > >>>>>>> at > https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.internal.io&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=oUUc68R-A0q141eyiEb8tSiXUnMlK1RJQAFk2Y0AcC8&e= > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > .SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:120) > > > > > > > > > >>>>>>> at > https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.internal.io&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=oUUc68R-A0q141eyiEb8tSiXUnMlK1RJQAFk2Y0AcC8&e= > > > > > > > > > >>>>>> > > > > > > .SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:83) > > > > > > > > > >>>>>>> at > https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.internal.io&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=oUUc68R-A0q141eyiEb8tSiXUnMlK1RJQAFk2Y0AcC8&e= > > > > > > > > > >>>>>> > > > > > > .SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) > > > > > > > > > >>>>>>> at > > > > org.apache.spark.scheduler.Task.run(Task.scala:123) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > > > > > > > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ThreadPoolExecutor.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=vqAmsqMs16YnJIzOu5RXbKPUar7cxKcZMYDAMmk5y8o&e= > :1149) > > > > > > > > > >>>>>>> at > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ThreadPoolExecutor.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=vqAmsqMs16YnJIzOu5RXbKPUar7cxKcZMYDAMmk5y8o&e= > :624) > > > > > > > > > >>>>>>> at java.lang.Thread.run( > https://urldefense.proofpoint.com/v2/url?u=http-3A__Thread.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=AIV_giot2889YjBVYzI9kK6hRjD7_zW6pluPyj7E9jg&e= > :748) > > > > > > > > > >>>>>>> Caused by: java.lang.ClassNotFoundException: > > > > > > > > > >>>>>> org.apache.parquet.schema.LogicalTypeAnnotation > > > > > > > > > >>>>>>> at > > > > > > > java.net.URLClassLoader.findClass( > https://urldefense.proofpoint.com/v2/url?u=http-3A__URLClassLoader.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=l0rcPNQkDktHTJpSRyLDNA6oxD-DNFdG1sbmbXJHoBw&e= > :382) > > > > > > > > > >>>>>>> at > > > > > java.lang.ClassLoader.loadClass( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ClassLoader.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=_Y3zN39Q3XTEKHmzI7GlUTW_UnI27oCYeayHeeMa6N4&e= > :424) > > > > > > > > > >>>>>>> at > > > > > > > > > >> > > > sun.misc.Launcher$AppClassLoader.loadClass( > https://urldefense.proofpoint.com/v2/url?u=http-3A__Launcher.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=PiLkstETFC9XjKt60e7oupwqEazGD7KDBoHGUcwnTds&e= > :349) > > > > > > > > > >>>>>>> at > > > > > java.lang.ClassLoader.loadClass( > https://urldefense.proofpoint.com/v2/url?u=http-3A__ClassLoader.java&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=_Y3zN39Q3XTEKHmzI7GlUTW_UnI27oCYeayHeeMa6N4&e= > :357) > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> michael > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>>> On Nov 20, 2019, at 3:29 AM, Gabor Szadovszky < > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > >>>>> wrote: > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> Thanks, Fokko. > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> Ryan, we did not do such measurements yet. I'm > > > afraid, I > > > > > > won't > > > > > > > > > have > > > > > > > > > >>>>>> enough > > > > > > > > > >>>>>>>> time to do that in the next couple of weeks. > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> Cheers, > > > > > > > > > >>>>>>>> Gabor > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> On Tue, Nov 19, 2019 at 6:14 PM Driesprong, Fokko > > > > > > > > > >>>>> <[email protected] > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>>> wrote: > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>>> Thanks Gabor for the explanation. I'd like to > > change > > > my > > > > > > vote > > > > > > > to > > > > > > > > > +1 > > > > > > > > > >>>>>>>>> (non-binding). > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>> Cheers, Fokko > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>> Op di 19 nov. 2019 om 18:03 schreef Ryan Blue > > > > > > > > > >>>>>> <[email protected]> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>>>>> Gabor, what I meant was: have we tried this with > > > real > > > > > data > > > > > > > to > > > > > > > > > see > > > > > > > > > >>>>> the > > > > > > > > > >>>>>>>>>> effect? I think those results would be helpful. > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>>> On Mon, Nov 18, 2019 at 11:35 PM Gabor > Szadovszky > > < > > > > > > > > > >>> [email protected] > > > > > > > > > >>>>>> > > > > > > > > > >>>>>>>>>> wrote: > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>>>> Hi Ryan, > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> It is not easy to calculate. For the column > > indexes > > > > > > feature > > > > > > > > we > > > > > > > > > >>>>>>>>> introduced > > > > > > > > > >>>>>>>>>>> two new structures saved before the footer: > > column > > > > > > indexes > > > > > > > > and > > > > > > > > > >>>>> offset > > > > > > > > > >>>>>>>>>>> indexes. If the min/max values are not too > long, > > > then > > > > > the > > > > > > > > > >>>>> truncation > > > > > > > > > >>>>>>>>>> might > > > > > > > > > >>>>>>>>>>> not decrease the file size because of the > offset > > > > > indexes. > > > > > > > > > >>> Moreover, > > > > > > > > > >>>>>> we > > > > > > > > > >>>>>>>>>> also > > > > > > > > > >>>>>>>>>>> introduced parquet.page.row.count.limit which > > might > > > > > > > increase > > > > > > > > > the > > > > > > > > > >>>>>> number > > > > > > > > > >>>>>>>>>> of > > > > > > > > > >>>>>>>>>>> pages which leads to increasing the file size. > > > > > > > > > >>>>>>>>>>> The footer itself is also changed and we are > > saving > > > > > more > > > > > > > > values > > > > > > > > > >> in > > > > > > > > > >>>>>> it: > > > > > > > > > >>>>>>>>>> the > > > > > > > > > >>>>>>>>>>> offset values to the column/offset indexes, the > > new > > > > > > logical > > > > > > > > > type > > > > > > > > > >>>>>>>>>>> structures, the CRC checksums (we might have > some > > > > > > others). > > > > > > > > > >>>>>>>>>>> So, the size of the files with small amount of > > data > > > > > will > > > > > > be > > > > > > > > > >>>>> increased > > > > > > > > > >>>>>>>>>>> (because of the larger footer). The size of the > > > files > > > > > > where > > > > > > > > the > > > > > > > > > >>>>>> values > > > > > > > > > >>>>>>>>>> can > > > > > > > > > >>>>>>>>>>> be encoded very well (RLE) will probably be > > > increased > > > > > > > > (because > > > > > > > > > >> we > > > > > > > > > >>>>>> will > > > > > > > > > >>>>>>>>>> have > > > > > > > > > >>>>>>>>>>> more pages). The size of some files where the > > > values > > > > > are > > > > > > > long > > > > > > > > > >>>>>> (>64bytes > > > > > > > > > >>>>>>>>>> by > > > > > > > > > >>>>>>>>>>> default) might be decreased because of > truncating > > > the > > > > > > > min/max > > > > > > > > > >>>>> values. > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> Regards, > > > > > > > > > >>>>>>>>>>> Gabor > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> On Mon, Nov 18, 2019 at 6:46 PM Ryan Blue > > > > > > > > > >>>>> <[email protected] > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>>>>>> wrote: > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> Gabor, do we have an idea of the additional > > > overhead > > > > > > for a > > > > > > > > > >>>>> non-test > > > > > > > > > >>>>>>>>>> data > > > > > > > > > >>>>>>>>>>>> file? It should be easy to validate that this > > > > doesn't > > > > > > > > > introduce > > > > > > > > > >>> an > > > > > > > > > >>>>>>>>>>>> unreasonable amount of overhead. In some > cases, > > it > > > > > > should > > > > > > > > > >>> actually > > > > > > > > > >>>>>> be > > > > > > > > > >>>>>>>>>>>> smaller since the column indexes are truncated > > and > > > > > page > > > > > > > > stats > > > > > > > > > >> are > > > > > > > > > >>>>>>>>> not. > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> On Mon, Nov 18, 2019 at 1:00 AM Gabor > Szadovszky > > > > > > > > > >>>>>>>>>>>> <[email protected]> > wrote: > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> Hi Fokko, > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> For the first point. The referenced > constructor > > > is > > > > > > > private > > > > > > > > > and > > > > > > > > > >>>>>>>>>> Iceberg > > > > > > > > > >>>>>>>>>>>> uses > > > > > > > > > >>>>>>>>>>>>> it via reflection. It is not a breaking > > change. I > > > > > > think, > > > > > > > > > >>>>> parquet-mr > > > > > > > > > >>>>>>>>>>> shall > > > > > > > > > >>>>>>>>>>>>> not keep private methods only because of > > clients > > > > > might > > > > > > > use > > > > > > > > > >> them > > > > > > > > > >>>>> via > > > > > > > > > >>>>>>>>>>>>> reflection. > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> About the checksum. I've agreed on having the > > CRC > > > > > > > checksum > > > > > > > > > >> write > > > > > > > > > >>>>>>>>>>> enabled > > > > > > > > > >>>>>>>>>>>> by > > > > > > > > > >>>>>>>>>>>>> default because the benchmarks did not show > > > > > significant > > > > > > > > > >>>>> performance > > > > > > > > > >>>>>>>>>>>>> penalties. See > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_parquet-2Dmr_pull_647&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=qqrHpGSG1Ku1bD-rQwjZlR33DoKe-4NEwUNCw8eoUFk&e= > > > > > > > > > >>> for > > > > > > > > > >>>>>>>>>>>> details. > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> About the file size change. 1.11.0 is > > introducing > > > > > > column > > > > > > > > > >>> indexes, > > > > > > > > > >>>>>>>>> CRC > > > > > > > > > >>>>>>>>>>>>> checksum, removing the statistics from the > page > > > > > headers > > > > > > > and > > > > > > > > > >>> maybe > > > > > > > > > >>>>>>>>>> other > > > > > > > > > >>>>>>>>>>>>> changes that impact file size. If only file > > size > > > is > > > > > in > > > > > > > > > >> question > > > > > > > > > >>> I > > > > > > > > > >>>>>>>>>>> cannot > > > > > > > > > >>>>>>>>>>>>> see a breaking change here. > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> Regards, > > > > > > > > > >>>>>>>>>>>>> Gabor > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> On Sun, Nov 17, 2019 at 9:27 PM Driesprong, > > Fokko > > > > > > > > > >>>>>>>>>> <[email protected] > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> wrote: > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> Unfortunately, a -1 from my side > (non-binding) > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> I've updated Iceberg to Parquet 1.11.0, and > > > found > > > > > > three > > > > > > > > > >> things: > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> - We've broken backward compatibility of the > > > > > > constructor > > > > > > > > of > > > > > > > > > >>>>>>>>>>>>>> ColumnChunkPageWriteStore > > > > > > > > > >>>>>>>>>>>>>> < > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_parquet-2Dmr_commit_e7db9e20f52c925a207ea62d6dda6dc4e870294e-23diff-2Dd007a18083a2431c30a5416f248e0a4bR80&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=c47ZyLKesRF-F3eK_4G9aROolyJwQPZmdGbnjJQH_eU&e= > > > > > > > > > >>>>>>>>>>>>>>> . > > > > > > > > > >>>>>>>>>>>>>> This required a change > > > > > > > > > >>>>>>>>>>>>>> < > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Diceberg_pull_297_files-23diff-2Db877faa96f292b851c75fe8bcc1912f8R176&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=7jqIhgiNnkm8N8z8RMRU_TteDEABVZMk-JsO6aQjRXc&e= > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> to the code. This isn't a hard blocker, but > if > > > > there > > > > > > > will > > > > > > > > be > > > > > > > > > >> a > > > > > > > > > >>>>>>>>>> new > > > > > > > > > >>>>>>>>>>>> RC, > > > > > > > > > >>>>>>>>>>>>>> I've > > > > > > > > > >>>>>>>>>>>>>> submitted a patch: > > > > > > > > > >>>>>>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_parquet-2Dmr_pull_699&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=JrIvjB_4COadNSlkUcZaAewxvfSbTsTyBsFUiizFNJU&e= > > > > > > > > > >>>>>>>>>>>>>> - Related, that we need to put in the > > changelog, > > > > is > > > > > > that > > > > > > > > > >>>>>>>>>> checksums > > > > > > > > > >>>>>>>>>>>> are > > > > > > > > > >>>>>>>>>>>>>> enabled by default: > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_parquet-2Dmr_blob_master_parquet-2Dcolumn_src_main_java_org_apache_parquet_column_ParquetProperties.java-23L54&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=2j-7A0WLZLej2EcsO7z9DYl58kOwEtRey_MZYLLk3IQ&e= > > > > > > > > > >>>>>>>>>>>>>> This > > > > > > > > > >>>>>>>>>>>>>> will impact performance. I would suggest > > > disabling > > > > > it > > > > > > by > > > > > > > > > >>>>>>>>>> default: > > > > > > > > > >>>>>>>>>>>>>> > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_parquet-2Dmr_pull_700&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=j5UBc01oaFJO_FD_FbQ48ORjxJBgOMeVvSo8Yy-rqwo&e= > > > > > > > > > >>>>>>>>>>>>>> < > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_parquet-2Dmr_commit_e7db9e20f52c925a207ea62d6dda6dc4e870294e-23diff-2Dd007a18083a2431c30a5416f248e0a4bR277&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=AyzLxEWZPSZlnpaz3SE3GTUu5ouF-Inwj2zuWICZAYI&e= > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> - Binary compatibility. While updating > > Iceberg, > > > > I've > > > > > > > > noticed > > > > > > > > > >>>>>>>>>> that > > > > > > > > > >>>>>>>>>>>> the > > > > > > > > > >>>>>>>>>>>>>> split-test was failing: > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_incubator-2Diceberg_pull_297_files-23diff-2D4b64b7014f259be41b26cfb73d3e6e93L199&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=qE9EJHf5O1z1IxWBSUQVGGTEXPyEdoILjwfljOP9IE0&e= > > > > > > > > > >>>>>>>>>>>>>> The > > > > > > > > > >>>>>>>>>>>>>> two records are now divided over four Spark > > > > > > partitions. > > > > > > > > > >>>>>>>>>> Something > > > > > > > > > >>>>>>>>>>> in > > > > > > > > > >>>>>>>>>>>>> the > > > > > > > > > >>>>>>>>>>>>>> output has changed since the files are > bigger > > > now. > > > > > Has > > > > > > > > > anyone > > > > > > > > > >>>>>>>>>> any > > > > > > > > > >>>>>>>>>>>> idea > > > > > > > > > >>>>>>>>>>>>>> to > > > > > > > > > >>>>>>>>>>>>>> check what's changed, or a way to check > this? > > > The > > > > > only > > > > > > > > thing > > > > > > > > > >> I > > > > > > > > > >>>>>>>>>> can > > > > > > > > > >>>>>>>>>>>>>> think of > > > > > > > > > >>>>>>>>>>>>>> is the checksum mentioned above. > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> $ ls -lah ~/Desktop/parquet-1-1* > > > > > > > > > >>>>>>>>>>>>>> -rw-r--r-- 1 fokkodriesprong staff 562B > 17 > > > nov > > > > > > 21:09 > > > > > > > > > >>>>>>>>>>>>>> > > > > > /Users/fokkodriesprong/Desktop/parquet-1-10-1.parquet > > > > > > > > > >>>>>>>>>>>>>> -rw-r--r-- 1 fokkodriesprong staff 611B > 17 > > > nov > > > > > > 21:05 > > > > > > > > > >>>>>>>>>>>>>> > > > > > /Users/fokkodriesprong/Desktop/parquet-1-11-0.parquet > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> $ parquet-tools cat > > > > > > > > > >>>>>>>>>>>> > > > > /Users/fokkodriesprong/Desktop/parquet-1-10-1.parquet > > > > > > > > > >>>>>>>>>>>>>> id = 1 > > > > > > > > > >>>>>>>>>>>>>> data = a > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> $ parquet-tools cat > > > > > > > > > >>>>>>>>>>>> > > > > /Users/fokkodriesprong/Desktop/parquet-1-11-0.parquet > > > > > > > > > >>>>>>>>>>>>>> id = 1 > > > > > > > > > >>>>>>>>>>>>>> data = a > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> A binary diff here: > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >> > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_Fokko_1c209f158299dc2fb5878c5bae4bf6d8&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=cTzJCzw-nCXuEIWrYnEVoSq3rGSqLk0Y6pRSHowwFQs&e= > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> Cheers, Fokko > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> Op za 16 nov. 2019 om 04:18 schreef Junjie > > Chen > > > < > > > > > > > > > >>>>>>>>>>>>> [email protected] > > > > > > > > > >>>>>>>>>>>>>>> : > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>> +1 > > > > > > > > > >>>>>>>>>>>>>>> Verified signature, checksum and ran mvn > > > install > > > > > > > > > >> successfully. > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>> Wang, Yuming <[email protected]> > > > > > > 于2019年11月14日周四 > > > > > > > > > >>>>>>>>> 下午2:05写道： > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> +1 > > > > > > > > > >>>>>>>>>>>>>>>> Tested Parquet 1.11.0 with Spark SQL > module: > > > > > > build/sbt > > > > > > > > > >>>>>>>>>>>>> "sql/test-only" > > > > > > > > > >>>>>>>>>>>>>>> -Phadoop-3.2 > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> On 2019/11/13, 21:33, "Gabor Szadovszky" > < > > > > > > > > > >> [email protected]> > > > > > > > > > >>>>>>>>>>>> wrote: > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> Hi everyone, > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> I propose the following RC to be released > as > > > > > > official > > > > > > > > > >>>>>>>>>> Apache > > > > > > > > > >>>>>>>>>>>>>> Parquet > > > > > > > > > >>>>>>>>>>>>>>> 1.11.0 > > > > > > > > > >>>>>>>>>>>>>>>> release. > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> The commit id is > > > > > > > > 18519eb8e059865652eee3ff0e8593f126701da4 > > > > > > > > > >>>>>>>>>>>>>>>> * This corresponds to the tag: > > > > > > > apache-parquet-1.11.0-rc7 > > > > > > > > > >>>>>>>>>>>>>>>> * > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__nam01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fparquet-2Dmr-252Ftree-252F18519eb8e059865652eee3ff0e8593f126701da4-26amp-3Bdata-3D02-257C01-257Cyumwang-2540ebay.com-257C8d588ca5855842a94bed08d7683e1221-257C46326bff992841a0baca17c16c94ea99-257C0-257C0-257C637092488114756267-26amp-3Bsdata-3DToLFrTB9lU-252FGzH6UpXwy7PAY7kaupbyKAgdghESCfgg-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=SJU6SXHUbJvaYgjg95fEBM-7sdiuRWI7iAheLj6HPhw&e= > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> The release tarball, signature, and > > checksums > > > > are > > > > > > > here: > > > > > > > > > >>>>>>>>>>>>>>>> * > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__nam01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fdist.apache.org-252Frepos-252Fdist-252Fdev-252Fparquet-252Fapache-2Dparquet-2D1.11.0-2Drc7-26amp-3Bdata-3D02-257C01-257Cyumwang-2540ebay.com-257C8d588ca5855842a94bed08d7683e1221-257C46326bff992841a0baca17c16c94ea99-257C0-257C0-257C637092488114756267-26amp-3Bsdata-3DMPaHiYJT7ZcqreAYUkvDvZugthUhRPrySdXpN2ytT5k-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=xGqVIxUEoUapVwpqRhYkq7OJtoikK0XaLGft9Lblwgw&e= > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> You can find the KEYS file here: > > > > > > > > > >>>>>>>>>>>>>>>> * > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__nam01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fapache.org-252Fdist-252Fparquet-252FKEYS-26amp-3Bdata-3D02-257C01-257Cyumwang-2540ebay.com-257C8d588ca5855842a94bed08d7683e1221-257C46326bff992841a0baca17c16c94ea99-257C0-257C0-257C637092488114756267-26amp-3Bsdata-3DIwG4MUGsP2lVzlD4bwZUEPuEAPUg-252FHXRYtxc5CQupBM-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=yzWOrL6zPFXTus1SdheATIVwHiOBt0Qidq_Hyl8mdpk&e= > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> Binary artifacts are staged in Nexus here: > > > > > > > > > >>>>>>>>>>>>>>>> * > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__nam01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Frepository.apache.org-252Fcontent-252Fgroups-252Fstaging-252Forg-252Fapache-252Fparquet-252F-26amp-3Bdata-3D02-257C01-257Cyumwang-2540ebay.com-257C8d588ca5855842a94bed08d7683e1221-257C46326bff992841a0baca17c16c94ea99-257C0-257C0-257C637092488114756267-26amp-3Bsdata-3DlHtqLRQqQFwsyoaLSVaJuau5gxPKsCQFFVJaY8H0tZQ-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=20DYwYuaSWdv5A9c3konLGFKzeLDizly8Ol1C9ByqYA&e= > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> This release includes the changes listed > at: > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__nam01.safelinks.protection.outlook.com_-3Furl-3Dhttps-253A-252F-252Fgithub.com-252Fapache-252Fparquet-2Dmr-252Fblob-252Fapache-2Dparquet-2D1.11.0-2Drc7-252FCHANGES.md-26amp-3Bdata-3D02-257C01-257Cyumwang-2540ebay.com-257C8d588ca5855842a94bed08d7683e1221-257C46326bff992841a0baca17c16c94ea99-257C0-257C0-257C637092488114756267-26amp-3Bsdata-3D82BplI3bLAL6qArLHvVoYReZOk-252BboSP655rI8VX5Q5I-253D-26amp-3Breserved-3D0&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=FQ88AmOZ4TMjDdqNBGu-ag&m=UfFS1N1YHm7sJbXTdatgZuyK2eS186zlrHz6rA6WZYQ&s=y1GMeLPY2n0dFWRj9sjEapyXX9woFJLI0UE59YYRYfI&e= > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> Please download, verify, and test. > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> Please vote in the next 72 hours. > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> [ ] +1 Release this as Apache Parquet > 1.11.0 > > > > > > > > > >>>>>>>>>>>>>>>> [ ] +0 > > > > > > > > > >>>>>>>>>>>>>>>> [ ] -1 Do not release this because... > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>>> -- > > > > > > > > > >>>>>>>>>>>> Ryan Blue > > > > > > > > > >>>>>>>>>>>> Software Engineer > > > > > > > > > >>>>>>>>>>>> Netflix > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > >>>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>>> -- > > > > > > > > > >>>>>>>>>> Ryan Blue > > > > > > > > > >>>>>>>>>> Software Engineer > > > > > > > > > >>>>>>>>>> Netflix > > > > > > > > > >>>>>>>>>> > > > > > > > > > >>>>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> -- > > > > > > > > > >>>> Ryan Blue > > > > > > > > > >>>> Software Engineer > > > > > > > > > >>>> Netflix > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Ryan Blue > > > > > > > > > > Software Engineer > > > > > > > > > > Netflix > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Ryan Blue > > > > > > > Software Engineer > > > > > > > Netflix > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Ryan Blue > > > > > Software Engineer > > > > > Netflix > > > > > > > > > > > > > > > -- Xinli Shang
