[jira] [Commented] (PARQUET-2153) Cannot read schema from parquet file
[ https://issues.apache.org/jira/browse/PARQUET-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17554037#comment-17554037 ] Nils Broman commented on PARQUET-2153: -- I just tested the fix you mention and I still got the same error, so unfortunately it does not fix this issue. > Cannot read schema from parquet file > > > Key: PARQUET-2153 > URL: https://issues.apache.org/jira/browse/PARQUET-2153 > Project: Parquet > Issue Type: Bug > Components: parquet-avro >Affects Versions: 1.12.3 >Reporter: Nils Broman >Priority: Major > > I'm trying to generate a Avro schema from a parquet file. I get the issue > both when using > [https://github.com/benwatson528/intellij-avro-parquet-plugin] in Intellij as > well as when I'm using my own implementation of generating a schema. > The parquet file contains nested entries and arrays: \{"a": a, "b": [{"c": > c}]} > I get the following error message: > {code:java} > org.apache.avro.SchemaParseException: Can't redefine: element > at org.apache.avro.Schema$Names.put(Schema.java:1547) > at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:810) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:972) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1134) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema.toString(Schema.java:424) > at org.apache.avro.Schema.toString(Schema.java:396) > at > uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getSchema(ParquetFileReader.java:65) > at > uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:196) > at > uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:184) > at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) {code} > It appears to be an issue similar to > [PARQUET-1441|https://issues.apache.org/jira/browse/PARQUET-1441] or > [PARQUET-1409|https://issues.apache.org/jira/browse/PARQUET-1409]. > Or could it possibly be something wrong in my parquet file? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2153) Cannot read schema from parquet file
[ https://issues.apache.org/jira/browse/PARQUET-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17553635#comment-17553635 ] Timothy Miller commented on PARQUET-2153: - Is this related to anything fixed by [https://github.com/apache/parquet-mr/pull/957] ? > Cannot read schema from parquet file > > > Key: PARQUET-2153 > URL: https://issues.apache.org/jira/browse/PARQUET-2153 > Project: Parquet > Issue Type: Bug > Components: parquet-avro >Affects Versions: 1.12.3 >Reporter: Nils Broman >Priority: Major > > I'm trying to generate a Avro schema from a parquet file. I get the issue > both when using > [https://github.com/benwatson528/intellij-avro-parquet-plugin] in Intellij as > well as when I'm using my own implementation of generating a schema. > The parquet file contains nested entries and arrays: \{"a": a, "b": [{"c": > c}]} > I get the following error message: > {code:java} > org.apache.avro.SchemaParseException: Can't redefine: element > at org.apache.avro.Schema$Names.put(Schema.java:1547) > at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:810) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:972) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1134) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema.toString(Schema.java:424) > at org.apache.avro.Schema.toString(Schema.java:396) > at > uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getSchema(ParquetFileReader.java:65) > at > uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:196) > at > uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:184) > at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) {code} > It appears to be an issue similar to > [PARQUET-1441|https://issues.apache.org/jira/browse/PARQUET-1441] or > [PARQUET-1409|https://issues.apache.org/jira/browse/PARQUET-1409]. > Or could it possibly be something wrong in my parquet file? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2153) Cannot read schema from parquet file
[ https://issues.apache.org/jira/browse/PARQUET-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17553558#comment-17553558 ] ASF GitHub Bot commented on PARQUET-2153: - NilsB44 opened a new pull request, #977: URL: https://github.com/apache/parquet-mr/pull/977 This extends the previous issue PARQUET-1441, where this issue was fixed for RecordSchema but not for FixedSchema. Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Parquet Jira](https://issues.apache.org/jira/browse/PARQUET/) issues and references them in the PR title. For example, "PARQUET-1234: My Parquet PR" - https://issues.apache.org/jira/browse/PARQUET-XXX - In case you are adding a dependency, check if the license complies with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: I'm unsure on how to add unit tests for FixedSchema, maybe someone could help me on this? ### Commits - [ ] My commits all reference Jira issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 1. Subject is limited to 50 characters (not including Jira issue reference) 1. Subject does not end with a period 1. Subject uses the imperative mood ("add", not "adding") 1. Body wraps at 72 characters 1. Body explains "what" and "why", not "how" ### Documentation - [ ] In case of new functionality, my PR adds documentation that describes how to use it. - All the public functions and the classes in the PR contain Javadoc that explain what it does > Cannot read schema from parquet file > > > Key: PARQUET-2153 > URL: https://issues.apache.org/jira/browse/PARQUET-2153 > Project: Parquet > Issue Type: Bug > Components: parquet-avro >Affects Versions: 1.12.3 >Reporter: Nils Broman >Priority: Major > > I'm trying to generate a Avro schema from a parquet file. I get the issue > both when using > [https://github.com/benwatson528/intellij-avro-parquet-plugin] in Intellij as > well as when I'm using my own implementation of generating a schema. > The parquet file contains nested entries and arrays: \{"a": a, "b": [{"c": > c}]} > I get the following error message: > {code:java} > org.apache.avro.SchemaParseException: Can't redefine: element > at org.apache.avro.Schema$Names.put(Schema.java:1547) > at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:810) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:972) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1134) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema.toString(Schema.java:424) > at org.apache.avro.Schema.toString(Schema.java:396) > at > uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getSchema(ParquetFileReader.java:65) > at > uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:196) > at > uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:184) > at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) {code} > It appears to be an issue similar to > [PARQUET-1441|https://issues.apache.org/jira/browse/PARQUET-1441] or > [PARQUET-1409|https://issues.apache.org/jira/browse/PARQUET-1409]. > Or could it possibly be something wrong in my parquet file? -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (PARQUET-2153) Cannot read schema from parquet file
[ https://issues.apache.org/jira/browse/PARQUET-2153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17553555#comment-17553555 ] Nils Broman commented on PARQUET-2153: -- I found out that this is indeed the same issue as for PARQUET-1441, however for FixedSchema. The fix is isolated to RecordSchema and the same fix needs to be applied to FixedSchema as well. > Cannot read schema from parquet file > > > Key: PARQUET-2153 > URL: https://issues.apache.org/jira/browse/PARQUET-2153 > Project: Parquet > Issue Type: Bug > Components: parquet-avro >Affects Versions: 1.12.3 >Reporter: Nils Broman >Priority: Major > > I'm trying to generate a Avro schema from a parquet file. I get the issue > both when using > [https://github.com/benwatson528/intellij-avro-parquet-plugin] in Intellij as > well as when I'm using my own implementation of generating a schema. > The parquet file contains nested entries and arrays: \{"a": a, "b": [{"c": > c}]} > I get the following error message: > {code:java} > org.apache.avro.SchemaParseException: Can't redefine: element > at org.apache.avro.Schema$Names.put(Schema.java:1547) > at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:810) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:972) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema$ArraySchema.toJson(Schema.java:1134) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1239) > at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1000) > at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:984) > at org.apache.avro.Schema.toString(Schema.java:424) > at org.apache.avro.Schema.toString(Schema.java:396) > at > uk.co.hadoopathome.intellij.viewer.fileformat.ParquetFileReader.getSchema(ParquetFileReader.java:65) > at > uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:196) > at > uk.co.hadoopathome.intellij.viewer.FileViewerToolWindow$2.doInBackground(FileViewerToolWindow.java:184) > at java.desktop/javax.swing.SwingWorker$1.call(SwingWorker.java:304) > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at java.desktop/javax.swing.SwingWorker.run(SwingWorker.java:343) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) {code} > It appears to be an issue similar to > [PARQUET-1441|https://issues.apache.org/jira/browse/PARQUET-1441] or > [PARQUET-1409|https://issues.apache.org/jira/browse/PARQUET-1409]. > Or could it possibly be something wrong in my parquet file? -- This message was sent by Atlassian Jira (v8.20.7#820007)