Thanks for verifying, Gabor. I have filed a report here: https://issues.apache.org/jira/browse/PARQUET-2104
p.s: I assumed `master` can't possibly be broken and spent days on this trying to find my mistake. Glad someone sees the same thing and its not just me :D On Tue, Nov 23, 2021 at 6:24 AM Gabor Szadovszky <[email protected]> wrote: > Hey, > > I can reproduce the same on master. It seems the same issue happens with > older versions as well. I don't know how we did not find it yet. (Or I am > making the same mistake as you :) ). > Could you please create a jira about it and continue the discussion there? > > Thanks a lot, > Gabor > > On Tue, Nov 23, 2021 at 1:08 AM gamaken k <[email protected]> wrote: > > > Hello, > > > > I'm relatively new to parquet and to the community, kindly bear with me > if > > I'm making simple mistakes. > > > > I cloned parquet-mr and built the repo locally on my ubuntu 20.04, When > > running parquet-cli, particularly when trying to read values (cat, head) > > from a parquet file, I run into NoSuchMethodError (full stack at the end > of > > message). > > > > *Here is what I have tried so far to understand this problem:* > > 1. clean the repo and clone again and retry > > 2. metadata inspection commands such as `meta`, `schema`, `pages` > > `dictionary` - they all work perfectly. check-stats command reports no > > corruption or errors, > > 3. I'm running parquet-cli without Hadoop (according to this ReadMe > > < > > > https://github.com/apache/parquet-mr/tree/master/parquet-cli#running-without-hadoop > > > > > ) > > 4. I have tried this on two different machines (ubuntu 18.04 and > > ubuntu 20.04) and gotten the same result. > > 5. The input file is produced by a library that I am writing to serialize > > parquet data in .net (i.e, not produced by parquet-mr). > > 6. parquet-cpp is able to read the file correctly and deserialize the > data. > > parquet-mr compatibility is important to me, so I'm trying to get this > > working. > > > > I could not solve this problem, so, I would really appreciate it if > someone > > who understands what's wrong here could kindly share insights. > > > > *Error stack:* > > ~/repos/parquet-mr/parquet-cli$ parquet cat > > ../../testdata/dictionaryEncodingSample.parquet > > WARNING: An illegal reflective access operation has occurred > > ...<trimmed some more WARNs>... > > Exception in thread "main" java.lang.NoSuchMethodError: > > 'org.apache.avro.Schema > > > > > org.apache.parquet.avro.AvroSchemaConverter.convert(org.apache.parquet.schema.MessageType)' > > at org.apache.parquet.cli.util.Schemas.fromParquet(Schemas.java:89) > > at org.apache.parquet.cli.BaseCommand.getAvroSchema(BaseCommand.java:405) > > at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:66) > > at org.apache.parquet.cli.Main.run(Main.java:157) > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) > > at org.apache.parquet.cli.Main.main(Main.java:187) > > > > *mvn dependency:tree output for parquet-cli* > > [INFO] -------------------< org.apache.parquet:parquet-cli > > >------------------- > > [INFO] Building Apache Parquet Command-line 1.13.0-SNAPSHOT > > [INFO] --------------------------------[ jar > > ]--------------------------------- > > [INFO] > > [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ parquet-cli > > --- > > [INFO] org.apache.parquet:parquet-cli:jar:1.13.0-SNAPSHOT > > [INFO] +- org.apache.parquet:parquet-avro:jar:1.13.0-SNAPSHOT:compile > > [INFO] +- > > org.apache.parquet:parquet-format-structures:jar:1.13.0-SNAPSHOT:compile > > [INFO] +- org.apache.parquet:parquet-common:jar:1.13.0-SNAPSHOT:compile > > [INFO] +- org.apache.parquet:parquet-column:jar:1.13.0-SNAPSHOT:compile > > [INFO] | +- > > org.apache.parquet:parquet-encoding:jar:1.13.0-SNAPSHOT:compile > > [INFO] | \- org.apache.yetus:audience-annotations:jar:0.13.0:compile > > [INFO] +- org.apache.parquet:parquet-hadoop:jar:1.13.0-SNAPSHOT:compile > > [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.8.3:compile > > [INFO] | \- commons-pool:commons-pool:jar:1.6:compile > > [INFO] +- org.apache.avro:avro:jar:1.10.2:compile > > [INFO] | \- org.apache.commons:commons-compress:jar:1.20:compile > > [INFO] +- com.github.luben:zstd-jni:jar:1.5.0-1:runtime > > [INFO] +- org.slf4j:slf4j-api:jar:1.7.22:compile > > [INFO] +- net.sf.opencsv:opencsv:jar:2.3:compile > > [INFO] +- org.apache.commons:commons-text:jar:1.8:compile > > [INFO] | \- org.apache.commons:commons-lang3:jar:3.9:compile > > [INFO] +- org.apache.parquet:parquet-jackson:jar:1.13.0-SNAPSHOT:runtime > > [INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.2:compile > > [INFO] +- com.fasterxml.jackson.core:jackson-core:jar:2.12.2:compile > > [INFO] +- > com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2:compile > > [INFO] +- com.beust:jcommander:jar:1.72:compile > > [INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.22:provided > > [INFO] +- com.google.guava:guava:jar:27.0.1-jre:provided > > [INFO] | +- com.google.guava:failureaccess:jar:1.0.1:provided > > [INFO] | +- > > > > > com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:provided > > [INFO] | +- org.checkerframework:checker-qual:jar:2.5.2:provided > > [INFO] | +- > > com.google.errorprone:error_prone_annotations:jar:2.2.0:provided > > [INFO] | +- com.google.j2objc:j2objc-annotations:jar:1.1:provided > > [INFO] | \- > org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:provided > > [INFO] +- org.apache.hadoop:hadoop-client:jar:2.10.1:provided > > [INFO] | +- org.apache.hadoop:hadoop-hdfs-client:jar:2.10.1:provided > > [INFO] | | \- com.squareup.okhttp:okhttp:jar:2.7.5:provided > > [INFO] | | \- com.squareup.okio:okio:jar:1.6.0:provided > > [INFO] | +- > > org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.10.1:provided > > [INFO] | | +- > > org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.10.1:provided > > [INFO] | | \- > > org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.10.1:provided > > [INFO] | | +- > > org.apache.hadoop:hadoop-yarn-server-common:jar:2.10.1:provided > > [INFO] | | | +- > > org.apache.hadoop:hadoop-yarn-registry:jar:2.10.1:provided > > [INFO] | | | +- > > > org.apache.geronimo.specs:geronimo-jcache_1.0_spec:jar:1.0-alpha-1:provided > > [INFO] | | | +- org.ehcache:ehcache:jar:3.3.1:provided > > [INFO] | | | +- com.zaxxer:HikariCP-java7:jar:2.4.12:provided > > [INFO] | | | \- > > com.microsoft.sqlserver:mssql-jdbc:jar:6.2.1.jre7:provided > > [INFO] | | \- > > org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided > > [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:2.10.1:provided > > [INFO] | | \- javax.xml.bind:jaxb-api:jar:2.2.2:provided > > [INFO] | | +- javax.xml.stream:stax-api:jar:1.0-2:provided > > [INFO] | | \- javax.activation:activation:jar:1.1:provided > > [INFO] | +- > > org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.10.1:provided > > [INFO] | | +- org.apache.hadoop:hadoop-yarn-client:jar:2.10.1:provided > > [INFO] | | \- org.apache.hadoop:hadoop-yarn-common:jar:2.10.1:provided > > [INFO] | | \- com.sun.jersey:jersey-client:jar:1.9:provided > > [INFO] | +- > > org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.10.1:provided > > [INFO] | \- org.apache.hadoop:hadoop-annotations:jar:2.10.1:provided > > [INFO] +- org.apache.hadoop:hadoop-common:jar:2.10.1:provided > > [INFO] | +- commons-cli:commons-cli:jar:1.2:provided > > [INFO] | +- org.apache.commons:commons-math3:jar:3.1.1:provided > > [INFO] | +- xmlenc:xmlenc:jar:0.52:provided > > [INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.2:provided > > [INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.4:provided > > [INFO] | +- commons-codec:commons-codec:jar:1.4:provided > > [INFO] | +- commons-net:commons-net:jar:3.1:provided > > [INFO] | +- commons-collections:commons-collections:jar:3.2.2:provided > > [INFO] | +- javax.servlet:servlet-api:jar:2.5:provided > > [INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26:provided > > [INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26:provided > > [INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:provided > > [INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:provided > > [INFO] | +- com.sun.jersey:jersey-core:jar:1.9:provided > > [INFO] | +- com.sun.jersey:jersey-json:jar:1.9:provided > > [INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:provided > > [INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:provided > > [INFO] | | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:provided > > [INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.8.3:provided > > [INFO] | +- com.sun.jersey:jersey-server:jar:1.9:provided > > [INFO] | | \- asm:asm:jar:3.1:provided > > [INFO] | +- net.java.dev.jets3t:jets3t:jar:0.9.0:provided > > [INFO] | | \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:provided > > [INFO] | +- commons-lang:commons-lang:jar:2.6:provided > > [INFO] | +- commons-configuration:commons-configuration:jar:1.6:provided > > [INFO] | +- commons-digester:commons-digester:jar:1.8:provided > > [INFO] | +- commons-beanutils:commons-beanutils:jar:1.9.4:provided > > [INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided > > [INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided > > [INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:provided > > [INFO] | +- com.google.code.gson:gson:jar:2.2.4:provided > > [INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.10.1:provided > > [INFO] | | +- com.nimbusds:nimbus-jose-jwt:jar:7.9:provided > > [INFO] | | | +- > > com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:provided > > [INFO] | | | \- net.minidev:json-smart:jar:2.3:provided (version > > selected from constraint [1.3.1,2.3]) > > [INFO] | | | \- net.minidev:accessors-smart:jar:1.2:provided > > [INFO] | | | \- org.ow2.asm:asm:jar:5.0.4:provided > > [INFO] | | +- > > > org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:provided > > [INFO] | | | +- > > org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:provided > > [INFO] | | | +- > > org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:provided > > [INFO] | | | \- > org.apache.directory.api:api-util:jar:1.0.0-M20:provided > > [INFO] | | \- org.apache.curator:curator-framework:jar:2.13.0:provided > > [INFO] | +- com.jcraft:jsch:jar:0.1.55:provided > > [INFO] | +- org.apache.curator:curator-client:jar:2.13.0:provided > > [INFO] | +- org.apache.curator:curator-recipes:jar:2.13.0:provided > > [INFO] | +- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:provided > > [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.14:provided > > [INFO] | | +- > com.github.spotbugs:spotbugs-annotations:jar:3.1.9:provided > > [INFO] | | \- io.netty:netty:jar:3.10.6.Final:provided > > [INFO] | +- org.codehaus.woodstox:stax2-api:jar:3.1.4:provided > > [INFO] | \- com.fasterxml.woodstox:woodstox-core:jar:5.0.3:provided > > [INFO] +- com.google.code.findbugs:jsr305:jar:3.0.2:provided > > [INFO] +- log4j:log4j:jar:1.2.17:provided > > [INFO] +- commons-io:commons-io:jar:2.4:provided > > [INFO] +- commons-logging:commons-logging:jar:1.1.3:provided > > [INFO] +- junit:junit:jar:4.13.1:test > > [INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test > > [INFO] \- org.easymock:easymock:jar:3.4:test > > [INFO] \- org.objenesis:objenesis:jar:2.2:test > > [INFO] > > ------------------------------------------------------------------------ > > [INFO] BUILD SUCCESS > > [INFO] > > ------------------------------------------------------------------------ > > > > Thank you > > >
