[
https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16815650#comment-16815650
]
Jozef Vilcek commented on BEAM-3386:
------------------------------------
Tried today to run SqlTransform with hive table provider on spark and got this
exeption .... Could it be related?
{code:java}
....
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Property
'org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem' not
valid for plugin type org.apache.calcite.rel.type.RelDataTypeSystem
at
org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:159)
at
org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:114)
at
org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.prepare.PlannerImpl.ready(PlannerImpl.java:143)
at
org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.prepare.PlannerImpl.parse(PlannerImpl.java:170)
at
org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.tools.Planner.parse(Planner.java:50)
at
org.apache.beam.sdk.extensions.sql.impl.BeamQueryPlanner.convertToBeamRel(BeamQueryPlanner.java:122)
at
org.apache.beam.sdk.extensions.sql.impl.BeamSqlEnv.parseQuery(BeamSqlEnv.java:143)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:111)
at
org.apache.beam.sdk.extensions.sql.SqlTransform.expand(SqlTransform.java:79)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
at
com.sizmek.dp.dsp.hive.ProtoHiveIO$SqlReadTransform.expand(ProtoHiveIO.scala:77)
at
com.sizmek.dp.dsp.hive.ProtoHiveIO$SqlReadTransform.expand(ProtoHiveIO.scala:60)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
at com.sizmek.dp.dsp.hive.ProtoHiveIO.readInternal(ProtoHiveIO.scala:48)
at com.sizmek.dp.dsp.hive.ProtoHiveIO.read(ProtoHiveIO.scala:40)
at
com.sizmek.dp.dsp.apollo.pricing.PricingPipeline$.buildPricingData(PricingPipeline.scala:57)
at
com.sizmek.dp.dsp.apollo.pricing.PricingDirectDayDriver$.main(PricingDirectDayDriver.scala:28)
at
com.sizmek.dp.dsp.apollo.pricing.PricingDirectDayDriver.main(PricingDirectDayDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:637)
Caused by: java.lang.RuntimeException: Property
'org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem' not
valid for plugin type org.apache.calcite.rel.type.RelDataTypeSystem
at
org.apache.calcite.avatica.ConnectionConfigImpl$4.apply(ConnectionConfigImpl.java:235)
at
org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.get_(ConnectionConfigImpl.java:93)
at
org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.getPlugin(ConnectionConfigImpl.java:160)
at
org.apache.calcite.avatica.ConnectionConfigImpl$PropEnv.getPlugin(ConnectionConfigImpl.java:146)
at
org.apache.calcite.config.CalciteConnectionConfigImpl.typeSystem(CalciteConnectionConfigImpl.java:92)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.<init>(CalciteConnectionImpl.java:110)
at
org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>(CalciteJdbc41Factory.java:113)
at
org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:58)
at
org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection(CalciteJdbc41Factory.java:43)
at
org.apache.calcite.jdbc.CalciteFactory.newConnection(CalciteFactory.java:53)
at
org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:143)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at
org.apache.beam.repackaged.beam_sdks_java_extensions_sql.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:153)
... 27 more
Caused by: java.lang.ClassNotFoundException:
org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.calcite.avatica.ConnectionConfigImpl$4.apply(ConnectionConfigImpl.java:223)
... 40 more
{code}
ClassNotFoundException part feels strange but when I got rid of calcite JARs
from Spark jars folder for debuging, it went a bit a further and crashed on
schema stuff (struct and arrays are not yet supported as I can see from code)
> Dependency conflict when Calcite is included in a project.
> ----------------------------------------------------------
>
> Key: BEAM-3386
> URL: https://issues.apache.org/jira/browse/BEAM-3386
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
> Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0
> Reporter: Austin Haas
> Assignee: Kai Jiang
> Priority: Critical
> Labels: triaged
>
> When Calcite (v. 1.13.0) is included in a project that also includes Beam and
> the Beam SQL extension, then the following error is thrown when trying to run
> Beam code.
> ClassCastException
> org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot
> be cast to org.apache.calcite.rel.type.RelDataTypeSystem
> org.apache.calcite.jdbc.CalciteConnectionImpl.<init>
> (CalciteConnectionImpl.java:120)
>
> org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection.<init>
> (CalciteJdbc41Factory.java:114)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection
> (CalciteJdbc41Factory.java:59)
> org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection
> (CalciteJdbc41Factory.java:44)
> org.apache.calcite.jdbc.CalciteFactory.newConnection
> (CalciteFactory.java:53)
> org.apache.calcite.avatica.UnregisteredDriver.connect
> (UnregisteredDriver.java:138)
> java.sql.DriverManager.getConnection (DriverManager.java:664)
> java.sql.DriverManager.getConnection (DriverManager.java:208)
>
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare
> (Frameworks.java:145)
>
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner
> (Frameworks.java:106)
>
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready
> (PlannerImpl.java:140)
>
> org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse
> (PlannerImpl.java:170)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
