[jira] [Created] (BEAM-9350) Default value for beam schema types
MAKSIM TSYGAN created BEAM-9350: --- Summary: Default value for beam schema types Key: BEAM-9350 URL: https://issues.apache.org/jira/browse/BEAM-9350 Project: Beam Issue Type: Improvement Components: beam-model, sdk-java-core Reporter: MAKSIM TSYGAN The Beam schema types not support default value. It break backward compatibility if convert avro schema to beam schema and then back (use beam AvroUtils class). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9613) BigQuery IO not support convert double type for beam row
MAKSIM TSYGAN created BEAM-9613: --- Summary: BigQuery IO not support convert double type for beam row Key: BEAM-9613 URL: https://issues.apache.org/jira/browse/BEAM-9613 Project: Beam Issue Type: Bug Components: beam-model Reporter: MAKSIM TSYGAN If execute query with double column via BigQueryIO.readFrom(), I get exception: Caused by: java.lang.UnsupportedOperationException: Converting BigQuery type 'class java.lang.Double' to 'FieldType\{typeName=DOUBLE, nullable=true, logicalType=null, collectionElementType=null, mapKeyType=null, mapValueType=null, rowSchema=null, metadata={}}' is not supported at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamValue(BigQueryUtils.java:532) at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRowFieldValue(BigQueryUtils.java:483) at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.lambda$toBeamRow$6(BigQueryUtils.java:469) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) at org.apache.beam.sdk.io.gcp.bigquery.BigQueryUtils.toBeamRow(BigQueryUtils.java:470) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9361) NPE When putting Avro record with enum through SqlTransform
[ https://issues.apache.org/jira/browse/BEAM-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066111#comment-17066111 ] MAKSIM TSYGAN commented on BEAM-9361: - The first is better! > NPE When putting Avro record with enum through SqlTransform > --- > > Key: BEAM-9361 > URL: https://issues.apache.org/jira/browse/BEAM-9361 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.19.0 >Reporter: Niels Basjes >Priority: Major > > I ran into this problem when trying to put my Avro records through the > SqlTransform. > I was able to reduce the reproduction path to the code below. > This code fails on my machine (using Beam 2.19.0) with the following > NullPointerException > {code:java} > org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse > query SELECT name, direction FROM InputStreamat > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:175) > at > org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:125) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:83) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490) > at > org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:261) > at com.bol.analytics.m2.TestAvro2SQL.testAvro2SQL(TestAvro2SQL.java:99) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException: > java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:217) > at > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:144) > ... 31 more > Caused by: java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:45) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:280) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:287) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.lambda$toCalciteRowType$0(CalciteUtils.java:261) > at > java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110) > at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:581) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toCalciteRowType(CalciteUtils.java:258) >
[jira] [Commented] (BEAM-9361) NPE When putting Avro record with enum through SqlTransform
[ https://issues.apache.org/jira/browse/BEAM-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112875#comment-17112875 ] MAKSIM TSYGAN commented on BEAM-9361: - I think that need add Enum in SQL type mapping in org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils class for work it in SQL: !image-2020-05-21-10-36-00-375.png! I think that also need add mapping for Enum type for zeta-sql. > NPE When putting Avro record with enum through SqlTransform > --- > > Key: BEAM-9361 > URL: https://issues.apache.org/jira/browse/BEAM-9361 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.19.0 >Reporter: Niels Basjes >Priority: P2 > Attachments: image-2020-05-21-10-36-00-375.png > > > I ran into this problem when trying to put my Avro records through the > SqlTransform. > I was able to reduce the reproduction path to the code below. > This code fails on my machine (using Beam 2.19.0) with the following > NullPointerException > {code:java} > org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse > query SELECT name, direction FROM InputStreamat > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:175) > at > org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:125) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:83) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490) > at > org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:261) > at com.bol.analytics.m2.TestAvro2SQL.testAvro2SQL(TestAvro2SQL.java:99) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException: > java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:217) > at > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:144) > ... 31 more > Caused by: java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:45) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:280) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:287) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.lambda$toCalciteRowType$0(CalciteUtils.java:261) >
[jira] [Updated] (BEAM-9361) NPE When putting Avro record with enum through SqlTransform
[ https://issues.apache.org/jira/browse/BEAM-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MAKSIM TSYGAN updated BEAM-9361: Attachment: image-2020-05-21-10-36-00-375.png > NPE When putting Avro record with enum through SqlTransform > --- > > Key: BEAM-9361 > URL: https://issues.apache.org/jira/browse/BEAM-9361 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.19.0 >Reporter: Niels Basjes >Priority: P2 > Attachments: image-2020-05-21-10-36-00-375.png > > > I ran into this problem when trying to put my Avro records through the > SqlTransform. > I was able to reduce the reproduction path to the code below. > This code fails on my machine (using Beam 2.19.0) with the following > NullPointerException > {code:java} > org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse > query SELECT name, direction FROM InputStreamat > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:175) > at > org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:125) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:83) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490) > at > org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:261) > at com.bol.analytics.m2.TestAvro2SQL.testAvro2SQL(TestAvro2SQL.java:99) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException: > java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:217) > at > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:144) > ... 31 more > Caused by: java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:45) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:280) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.toRelDataType(CalciteUtils.java:287) > at > org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils.lambda$toCalciteRowType$0(CalciteUtils.java:261) > at > java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:110) > at java.util.stream.IntPipeline$Head.forEach(IntPipeline.java:581) > at >
[jira] [Comment Edited] (BEAM-9361) NPE When putting Avro record with enum through SqlTransform
[ https://issues.apache.org/jira/browse/BEAM-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112875#comment-17112875 ] MAKSIM TSYGAN edited comment on BEAM-9361 at 5/21/20, 7:43 AM: --- I think that need add SQL type mapping in org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils class for work it in SQL: !image-2020-05-21-10-36-00-375.png! I think that also need add mapping for Enum type for zeta-sql. was (Author: maksim.tsygan): I think that need add Enum in SQL type mapping in org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils class for work it in SQL: !image-2020-05-21-10-36-00-375.png! I think that also need add mapping for Enum type for zeta-sql. > NPE When putting Avro record with enum through SqlTransform > --- > > Key: BEAM-9361 > URL: https://issues.apache.org/jira/browse/BEAM-9361 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.19.0 >Reporter: Niels Basjes >Priority: P2 > Attachments: image-2020-05-21-10-36-00-375.png > > > I ran into this problem when trying to put my Avro records through the > SqlTransform. > I was able to reduce the reproduction path to the code below. > This code fails on my machine (using Beam 2.19.0) with the following > NullPointerException > {code:java} > org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse > query SELECT name, direction FROM InputStreamat > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:175) > at > org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:125) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:83) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490) > at > org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:261) > at com.bol.analytics.m2.TestAvro2SQL.testAvro2SQL(TestAvro2SQL.java:99) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException: > java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:217) > at > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:144) > ... 31 more > Caused by: java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:45) > at >
[jira] [Comment Edited] (BEAM-9361) NPE When putting Avro record with enum through SqlTransform
[ https://issues.apache.org/jira/browse/BEAM-9361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112875#comment-17112875 ] MAKSIM TSYGAN edited comment on BEAM-9361 at 5/21/20, 7:44 AM: --- I think that need add SQL type mapping in org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils class for work it in SQL: !image-2020-05-21-10-36-00-375.png! Also we need add mapping for zeta-sql. was (Author: maksim.tsygan): I think that need add SQL type mapping in org.apache.beam.sdk.extensions.sql.impl.utils.CalciteUtils class for work it in SQL: !image-2020-05-21-10-36-00-375.png! I think that also need add mapping for Enum type for zeta-sql. > NPE When putting Avro record with enum through SqlTransform > --- > > Key: BEAM-9361 > URL: https://issues.apache.org/jira/browse/BEAM-9361 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.19.0 >Reporter: Niels Basjes >Priority: P2 > Attachments: image-2020-05-21-10-36-00-375.png > > > I ran into this problem when trying to put my Avro records through the > SqlTransform. > I was able to reduce the reproduction path to the code below. > This code fails on my machine (using Beam 2.19.0) with the following > NullPointerException > {code:java} > org.apache.beam.sdk.extensions.sql.impl.ParseException: Unable to parse > query SELECT name, direction FROM InputStreamat > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:175) > at > org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:103) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:125) > at > org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:83) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:539) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:490) > at > org.apache.beam.sdk.values.PCollectionTuple.apply(PCollectionTuple.java:261) > at com.bol.analytics.m2.TestAvro2SQL.testAvro2SQL(TestAvro2SQL.java:99) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:319) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at org.junit.runner.JUnitCore.run(JUnitCore.java:137) > at > com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) > at > com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33) > at > com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230) > at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58) > Caused by: > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.ValidationException: > java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.prepare.PlannerImpl.validate(PlannerImpl.java:217) > at > org.apache.beam.sdk.extensions.sql.impl.CalciteQueryPlanner.convertToBeamRel(CalciteQueryPlanner.java:144) > ... 31 more > Caused by: java.lang.NullPointerException > at > org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.type.SqlTypeFactoryImpl.createSqlType(SqlTypeFactoryImpl.java:45) > at >