Thanks Junjie, for testing. The issue is that some encryption interop tests uses parquet files from a separate git repository (parquet-testing). To achieve that the maven-exec-plugin is configured to use the "git submodule update" command for downloading the related files. Unfortunately, we did not think about this issue during implementing this idea. It all means that the tarball is not 100% standalone.
I've created the jira PARQUET-1992 <https://issues.apache.org/jira/browse/PARQUET-1992> about this topic. Any comments about the solution of this issue are welcomed. I am currently unsure if this issue is a show stopper or not. By editing the pom.xml file and removing related exec plugin the binaries are reproducible from the tarball. All the tests are also executable excluding the one that requires these parquet files. What are the exact requirements from a tarball? Thanks a lot, Gabor On Mon, Mar 1, 2021 at 11:19 AM Junjie Chen <[email protected]> wrote: > Hi > I downloaded the package and ran mvn clean install, it failed with message: > [ERROR] Failed to execute goal > org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (git submodule update) on > project parquet-format-structures: Command > execution failed.: Process exited with an error: 128 (Exit value: 128) -> > [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal org.codehaus.mojo:exec-maven-plugin:1.2.1:exec (git submodule updat > e) on project parquet-format-structures: Command execution failed. > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > (MojoExecutor.java:215) > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > (MojoExecutor.java:156) > at org.apache.maven.lifecycle.internal.MojoExecutor.execute > (MojoExecutor.java:148) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject > (LifecycleModuleBuilder.java:117) > at > org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject > (LifecycleModuleBuilder.java:81) > at > > org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build > (SingleThreadedBuilder.java:56) > at org.apache.maven.lifecycle.internal.LifecycleStarter.execute > (LifecycleStarter.java:128) > at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) > at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) > at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) > at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) > at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) > at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) > at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke > (NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke > (DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke (Method.java:498) > at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced > (Launcher.java:282) > at org.codehaus.plexus.classworlds.launcher.Launcher.launch > (Launcher.java:225) > at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode > (Launcher.java:406) > at org.codehaus.plexus.classworlds.launcher.Launcher.main > (Launcher.java:347) > > Should I install something before running the maven build? > > On Sun, Feb 28, 2021 at 11:50 AM Wang, Yuming <[email protected]> > wrote: > > > +1. Thank you Gabor. It works. > > > > On 2021/2/26, 19:50, "Gabor Szadovszky" <[email protected]> wrote: > > > > External Email > > > > Thanks for testing. It seems that the previous (rc1) was stuck on the > > staging maven repo. I've had to manually drop the previous upload. I > > don't > > remember we've had to do such things before. Anyway, the referenced > > staging > > maven repository now contains the rc2 build. Could you please try it > > again? > > > > Thanks a lot, > > Gabor > > > > On Fri, Feb 26, 2021 at 1:30 AM Wang, Yuming > <[email protected] > > > > > wrote: > > > > > -1. It still throws incompatible exceptions: > > > Caused by: sbt.ForkMain$ForkError: java.lang.NoSuchMethodError: > > > java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer; > > > at > > > > > > org.apache.parquet.hadoop.codec.SnappyCompressor.reset(SnappyCompressor.java:156) > > > at > > > > > > org.apache.hadoop.io.compress.CodecPool.returnCompressor(CodecPool.java:210) > > > at > > > > > > org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.release(CodecFactory.java:177) > > > at > > > > org.apache.parquet.hadoop.CodecFactory.release(CodecFactory.java:250) > > > at > > > > > > org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:168) > > > at > > > > > > org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:41) > > > at > > > > > > org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:58) > > > at > > > > > > org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:75) > > > at > > > > > > org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:281) > > > at > > > > > > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1471) > > > at > > > > > > org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:287) > > > ... 9 more > > > > > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fpull%2F31649&data=04%7C01%7Cyumwang%40ebay.com%7Cd34af2e6aa3749a295a408d8da4cc713%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637499370584039891%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Zm%2FlQaid9CeDDxig3RdWFzJ065lqOoT0s3T72hqFwc4%3D&reserved=0 > > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fruns%2F1983121842%3Fcheck_suite_focus%3Dtrue&data=04%7C01%7Cyumwang%40ebay.com%7Cd34af2e6aa3749a295a408d8da4cc713%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637499370584039891%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BV9GaP1SCYjkVyH42Rs1PbpFxLvS3AY9nVLJLXZL8tE%3D&reserved=0 > > > > > > > > > On 2021/2/25, 19:27, "Gabor Szadovszky" <[email protected]> wrote: > > > > > > External Email > > > > > > Hi everyone, > > > > > > I propose the following RC to be released as the official > Apache > > > Parquet > > > 1.12.0 release. > > > > > > The commit id is d859b4435d139f1979f46cf63d2afdf32935ad51 > > > * This corresponds to the tag: apache-parquet-1.12.0-rc2 > > > * > > > > > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Ftree%2Fd859b4435d139f1979f46cf63d2afdf32935ad51&data=04%7C01%7Cyumwang%40ebay.com%7Cd34af2e6aa3749a295a408d8da4cc713%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637499370584039891%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=S91VddHb1xV5XG5gMXSWU29UeF9H2yVbEjKv9koovYw%3D&reserved=0 > > > > > > The release tarball, signature, and checksums are here: > > > * > > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fparquet%2Fapache-parquet-1.12.0-rc2&data=04%7C01%7Cyumwang%40ebay.com%7Cd34af2e6aa3749a295a408d8da4cc713%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637499370584039891%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Z6gftDJWkIdxqJH0LB6zbvEkf9Pb25uXOScmwQ%2BwwOM%3D&reserved=0 > > > > > > You can find the KEYS file here: > > > * > > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdownloads.apache.org%2Fparquet%2FKEYS&data=04%7C01%7Cyumwang%40ebay.com%7Cd34af2e6aa3749a295a408d8da4cc713%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637499370584039891%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7gMa%2Fw%2F0PUujCwTBV%2F34F0s4JZwwIy%2FWKaO8pLsXfkw%3D&reserved=0 > > > > > > Binary artifacts are staged in Nexus here: > > > * > > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Fgroups%2Fstaging%2Forg%2Fapache%2Fparquet%2F&data=04%7C01%7Cyumwang%40ebay.com%7Cd34af2e6aa3749a295a408d8da4cc713%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637499370584049885%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=tNdrkiwxcx3lxkLr1wup%2FI8ohuEhoEuQJHjJ6gMJE9Q%3D&reserved=0 > > > > > > For release content see: > > > * > > > > > > > > > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fparquet-mr%2Fblob%2Fapache-parquet-1.12.0-rc2%2FCHANGES.md&data=04%7C01%7Cyumwang%40ebay.com%7Cd34af2e6aa3749a295a408d8da4cc713%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637499370584049885%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=%2BjQTQq5iRekn2Cs7hk9b%2BcJy%2BpBOLLTI3Se5iSYYwbM%3D&reserved=0 > > > > > > Please download, verify, and test. > > > > > > Please vote in the next 72 hours. > > > > > > [ ] +1 Release this as Apache Parquet 1.12.0 > > > [ ] +0 > > > [ ] -1 Do not release this because... > > > > > > > > > > > > -- > Best Regards >
