[jira] [Comment Edited] (BEAM-9239) Dependency conflict with Spark using aws io
[
https://issues.apache.org/jira/browse/BEAM-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181630#comment-17181630
]
Kai Jiang edited comment on BEAM-9239 at 8/21/20, 5:53 AM:
---
I am not too familiar with Flink. However, is that possible to shade or
relocate `{{com.amazonaws`}} packages in your flink application jar? You could
take a look at Flink docs about how to do and follow this:
[https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html#resolving-dependency-conflicts-with-flink-using-the-maven-shade-plugin]
was (Author: vectorijk):
I am not too familiar with Flink. However, is that possible to shade or
relocate `{{com.amazonaws`}} packages in your flink application jar? You could
take a look at Flink docs about how to do this:
[https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html#resolving-dependency-conflicts-with-flink-using-the-maven-shade-plugin]
> Dependency conflict with Spark using aws io
> ---
>
> Key: BEAM-9239
> URL: https://issues.apache.org/jira/browse/BEAM-9239
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws, runner-spark
>Affects Versions: 2.17.0
>Reporter: David McIntosh
>Priority: P1
>
> Starting with beam 2.17.0 I get this error in the Spark 2.4.4 driver when aws
> io is also used:
> {noformat}
> java.lang.NoSuchMethodError:
> com.fasterxml.jackson.databind.jsontype.TypeSerializer.typeId(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonToken;)Lcom/fasterxml/jackson/core/type/WritableTypeId;
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:163)
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:134)
> at
> com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.ensureSerializable(ProxyInvocationHandler.java:721)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:647)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:635)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:67)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.(SerializablePipelineOptions.java:43)
> at
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:71)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:215)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:90)
> {noformat}
> The cause seems to be that the Spark driver environment uses an older version
> of Jackson. I tried to update jackson on the Spark cluster but that led to
> several other errors.
> The change that started causing this was:
> https://github.com/apache/beam/commit/b68d70a47b68ad84efcd9405c1799002739bd116
> After reverting that change I was able to successfully run my job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9239) Dependency conflict with Spark using aws io
[
https://issues.apache.org/jira/browse/BEAM-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181630#comment-17181630
]
Kai Jiang commented on BEAM-9239:
-
I am not too familiar with Flink. However, is that possible to shade or
relocate `{{com.amazonaws`}} packages in your flink application jar? You could
take a look at Flink docs about how to do this:
[https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html#resolving-dependency-conflicts-with-flink-using-the-maven-shade-plugin]
> Dependency conflict with Spark using aws io
> ---
>
> Key: BEAM-9239
> URL: https://issues.apache.org/jira/browse/BEAM-9239
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws, runner-spark
>Affects Versions: 2.17.0
>Reporter: David McIntosh
>Priority: P1
>
> Starting with beam 2.17.0 I get this error in the Spark 2.4.4 driver when aws
> io is also used:
> {noformat}
> java.lang.NoSuchMethodError:
> com.fasterxml.jackson.databind.jsontype.TypeSerializer.typeId(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonToken;)Lcom/fasterxml/jackson/core/type/WritableTypeId;
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:163)
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:134)
> at
> com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.ensureSerializable(ProxyInvocationHandler.java:721)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:647)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:635)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:67)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.(SerializablePipelineOptions.java:43)
> at
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:71)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:215)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:90)
> {noformat}
> The cause seems to be that the Spark driver environment uses an older version
> of Jackson. I tried to update jackson on the Spark cluster but that led to
> several other errors.
> The change that started causing this was:
> https://github.com/apache/beam/commit/b68d70a47b68ad84efcd9405c1799002739bd116
> After reverting that change I was able to successfully run my job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9239) Dependency conflict with Spark using aws io
[
https://issues.apache.org/jira/browse/BEAM-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120035#comment-17120035
]
Kai Jiang edited comment on BEAM-9239 at 5/30/20, 12:23 AM:
Spark has two experimental configs `spark.driver.userClassPathFirst` and
`spark.executor.userClassPathFirst`. If set both of these as true, it allows
user-added jars precedence over Spark's own jars when loading classes in
cluster mode.
was (Author: vectorijk):
Spark has two experimental configs `spark.driver.userClassPathFirst` and
`spark.executor.userClassPathFirst`. It allows user-added jars precedence over
Spark's own jars when loading classes in cluster mode.
> Dependency conflict with Spark using aws io
> ---
>
> Key: BEAM-9239
> URL: https://issues.apache.org/jira/browse/BEAM-9239
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws, runner-spark
>Affects Versions: 2.17.0
>Reporter: David McIntosh
>Priority: P1
>
> Starting with beam 2.17.0 I get this error in the Spark 2.4.4 driver when aws
> io is also used:
> {noformat}
> java.lang.NoSuchMethodError:
> com.fasterxml.jackson.databind.jsontype.TypeSerializer.typeId(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonToken;)Lcom/fasterxml/jackson/core/type/WritableTypeId;
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:163)
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:134)
> at
> com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.ensureSerializable(ProxyInvocationHandler.java:721)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:647)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:635)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:67)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.(SerializablePipelineOptions.java:43)
> at
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:71)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:215)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:90)
> {noformat}
> The cause seems to be that the Spark driver environment uses an older version
> of Jackson. I tried to update jackson on the Spark cluster but that led to
> several other errors.
> The change that started causing this was:
> https://github.com/apache/beam/commit/b68d70a47b68ad84efcd9405c1799002739bd116
> After reverting that change I was able to successfully run my job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9239) Dependency conflict with Spark using aws io
[
https://issues.apache.org/jira/browse/BEAM-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120038#comment-17120038
]
Kai Jiang commented on BEAM-9239:
-
Maybe vendor relocated com.fasterxml.jackson library could be long term
solution for this
> Dependency conflict with Spark using aws io
> ---
>
> Key: BEAM-9239
> URL: https://issues.apache.org/jira/browse/BEAM-9239
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws, runner-spark
>Affects Versions: 2.17.0
>Reporter: David McIntosh
>Priority: P1
>
> Starting with beam 2.17.0 I get this error in the Spark 2.4.4 driver when aws
> io is also used:
> {noformat}
> java.lang.NoSuchMethodError:
> com.fasterxml.jackson.databind.jsontype.TypeSerializer.typeId(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonToken;)Lcom/fasterxml/jackson/core/type/WritableTypeId;
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:163)
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:134)
> at
> com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.ensureSerializable(ProxyInvocationHandler.java:721)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:647)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:635)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:67)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.(SerializablePipelineOptions.java:43)
> at
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:71)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:215)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:90)
> {noformat}
> The cause seems to be that the Spark driver environment uses an older version
> of Jackson. I tried to update jackson on the Spark cluster but that led to
> several other errors.
> The change that started causing this was:
> https://github.com/apache/beam/commit/b68d70a47b68ad84efcd9405c1799002739bd116
> After reverting that change I was able to successfully run my job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9239) Dependency conflict with Spark using aws io
[
https://issues.apache.org/jira/browse/BEAM-9239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120035#comment-17120035
]
Kai Jiang commented on BEAM-9239:
-
Spark has two experimental configs `spark.driver.userClassPathFirst` and
`spark.executor.userClassPathFirst`. It allows user-added jars precedence over
Spark's own jars when loading classes in cluster mode.
> Dependency conflict with Spark using aws io
> ---
>
> Key: BEAM-9239
> URL: https://issues.apache.org/jira/browse/BEAM-9239
> Project: Beam
> Issue Type: Bug
> Components: io-java-aws, runner-spark
>Affects Versions: 2.17.0
>Reporter: David McIntosh
>Priority: P1
>
> Starting with beam 2.17.0 I get this error in the Spark 2.4.4 driver when aws
> io is also used:
> {noformat}
> java.lang.NoSuchMethodError:
> com.fasterxml.jackson.databind.jsontype.TypeSerializer.typeId(Ljava/lang/Object;Lcom/fasterxml/jackson/core/JsonToken;)Lcom/fasterxml/jackson/core/type/WritableTypeId;
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:163)
> at
> org.apache.beam.sdk.io.aws.options.AwsModule$AWSCredentialsProviderSerializer.serializeWithType(AwsModule.java:134)
> at
> com.fasterxml.jackson.databind.ser.impl.TypeWrappedSerializer.serialize(TypeWrappedSerializer.java:32)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.ensureSerializable(ProxyInvocationHandler.java:721)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:647)
> at
> org.apache.beam.sdk.options.ProxyInvocationHandler$Serializer.serialize(ProxyInvocationHandler.java:635)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3559)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2927)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.serializeToJson(SerializablePipelineOptions.java:67)
> at
> org.apache.beam.runners.core.construction.SerializablePipelineOptions.(SerializablePipelineOptions.java:43)
> at
> org.apache.beam.runners.spark.translation.EvaluationContext.(EvaluationContext.java:71)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:215)
> at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:90)
> {noformat}
> The cause seems to be that the Spark driver environment uses an older version
> of Jackson. I tried to update jackson on the Spark cluster but that led to
> several other errors.
> The change that started causing this was:
> https://github.com/apache/beam/commit/b68d70a47b68ad84efcd9405c1799002739bd116
> After reverting that change I was able to successfully run my job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-10000) Support BIT_XOR aggregation function in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107787#comment-17107787 ] Kai Jiang commented on BEAM-1: -- wow! #1 > Support BIT_XOR aggregation function in BeamSQL > --- > > Key: BEAM-1 > URL: https://issues.apache.org/jira/browse/BEAM-1 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Rui Wang >Priority: Major > > See reference: > https://cloud.google.com/bigquery/docs/reference/standard-sql/aggregate_functions#bit_xor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9121) Bump vendored calcite to 1.21.0
[ https://issues.apache.org/jira/browse/BEAM-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang resolved BEAM-9121. - Fix Version/s: 2.20.0 Resolution: Won't Fix > Bump vendored calcite to 1.21.0 > --- > > Key: BEAM-9121 > URL: https://issues.apache.org/jira/browse/BEAM-9121 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: 2.20.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9121) Bump vendored calcite to 1.21.0
[ https://issues.apache.org/jira/browse/BEAM-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang updated BEAM-9121: Summary: Bump vendored calcite to 1.21.0 (was: Bump vendored calcite to 1.22.0) > Bump vendored calcite to 1.21.0 > --- > > Key: BEAM-9121 > URL: https://issues.apache.org/jira/browse/BEAM-9121 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9121) Bump vendored calcite to 1.22.0
[ https://issues.apache.org/jira/browse/BEAM-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang reassigned BEAM-9121: --- Assignee: Kai Jiang > Bump vendored calcite to 1.22.0 > --- > > Key: BEAM-9121 > URL: https://issues.apache.org/jira/browse/BEAM-9121 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9121) Bump vendored calcite to 1.22.0
[ https://issues.apache.org/jira/browse/BEAM-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang updated BEAM-9121: Summary: Bump vendored calcite to 1.22.0 (was: Bump vendored calcite to 1.21.0) > Bump vendored calcite to 1.22.0 > --- > > Key: BEAM-9121 > URL: https://issues.apache.org/jira/browse/BEAM-9121 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Kai Jiang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9121) Bump vendored calcite to 1.21.0
[ https://issues.apache.org/jira/browse/BEAM-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang reassigned BEAM-9121: --- Assignee: (was: Kai Jiang) > Bump vendored calcite to 1.21.0 > --- > > Key: BEAM-9121 > URL: https://issues.apache.org/jira/browse/BEAM-9121 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Kai Jiang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9412) Fix linkage errors in vendored calcite
[
https://issues.apache.org/jira/browse/BEAM-9412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Jiang reassigned BEAM-9412:
---
Assignee: Kai Jiang
> Fix linkage errors in vendored calcite
> --
>
> Key: BEAM-9412
> URL: https://issues.apache.org/jira/browse/BEAM-9412
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
>Reporter: Luke Cwik
>Assignee: Kai Jiang
>Priority: Minor
>
> As of [https://github.com/apache/beam/pull/10559], the linkage errors are:
> {code:java}
> Class org.slf4j.LoggerFactory is not found;
> referenced by 29 class files
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.pretty.SqlPrettyWriter
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.rel.rules.AbstractMaterializedViewRule
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.Benchmark
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.RexImplicationChecker
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.plan.VisitorDataContext
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDialectFactoryImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.SqlDialect
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.sql.validate.SqlValidatorException
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.ResultSetEnumerable
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.util.trace.CalciteTrace
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.runtime.CalciteException
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.AvaticaHttpClientFactoryImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.RemoteProtobufService
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.KerberosConnection
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.ClientKeytabJaasConf
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientSpnegoImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.Driver
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.remote.ProtobufTranslationImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.filter.FilterCompiler
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.filter.RelationalExpressionNode
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.filter.ValueNode
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.JsonContext
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.path.ArrayPathToken
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.path.CompiledPath
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.internal.path.PredicateContextImpl
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.spi.json.JsonOrgJsonProvider
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.com.jayway.jsonpath.spi.mapper.GsonMappingProvider
> (beam-vendor-calcite-1_20_0-0.1-SNAPSHOT.jar)
>
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.avatica.metrics.MetricsSystemLoader
> (beam-vendor-c
[jira] [Updated] (BEAM-8331) Vendored calcite breaks if another calcite is on the class path
[
https://issues.apache.org/jira/browse/BEAM-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Jiang updated BEAM-8331:
Parent: BEAM-9121
Issue Type: Sub-task (was: Bug)
> Vendored calcite breaks if another calcite is on the class path
> ---
>
> Key: BEAM-8331
> URL: https://issues.apache.org/jira/browse/BEAM-8331
> Project: Beam
> Issue Type: Sub-task
> Components: dsl-sql
>Affects Versions: 2.15.0, 2.16.0
>Reporter: Andrew Pilloud
>Assignee: Kai Jiang
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> If the beam vendored calcite and a non-vendored calcite are both on the
> classpath, neither version works. This is because the non-JDBC calcite path
> uses JDBC as a easy way to perform reflection. (This affects the non-JDBC
> version of calcite.) We need to rewrite the calcite JDBC urls as part of our
> vendoring (for example 'jdbc:calcite:' to 'jdbc:beam-vendor-calcite:').
> Example of where this happens:
> [https://github.com/apache/calcite/blob/0cce229903a845a7b8ed36cf86d6078fd82d73d3/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L175]
>
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException: Property
> 'org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem' not
> valid for plugin type org.apache.calcite.rel.type.RelDataTypeSystem
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:160)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:115)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.(ZetaSQLPlannerImpl.java:86)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.(ZetaSQLQueryPlanner.java:55){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9121) Bump vendored calcite to 1.21.0
[ https://issues.apache.org/jira/browse/BEAM-9121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang updated BEAM-9121: Status: Open (was: Triage Needed) > Bump vendored calcite to 1.21.0 > --- > > Key: BEAM-9121 > URL: https://issues.apache.org/jira/browse/BEAM-9121 > Project: Beam > Issue Type: Task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9121) Bump vendored calcite to 1.21.0
Kai Jiang created BEAM-9121: --- Summary: Bump vendored calcite to 1.21.0 Key: BEAM-9121 URL: https://issues.apache.org/jira/browse/BEAM-9121 Project: Beam Issue Type: Task Components: dsl-sql Reporter: Kai Jiang Assignee: Kai Jiang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8331) Vendored calcite breaks if another calcite is on the class path
[
https://issues.apache.org/jira/browse/BEAM-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012693#comment-17012693
]
Kai Jiang edited comment on BEAM-8331 at 1/10/20 10:41 AM:
---
I have built a vendored calcite with rewriting jdbc url locally. However, I
have tested the issue mentioned BEAM-3386. (ran a Beam SQL job on Spark runner)
Vendored calcite with rewriting JDBC url or without rewriting JDBC url both
works for Beam SQL job on Spark runner. (ran successfully)
[~apilloud] Would you mind sharing how to reproduce this issue in your case? so
that I can validate vendored calcite with rewriting jdbc url.
was (Author: vectorijk):
I have built a vendored calcite with rewriting jdbc url locally. However, I
have tested the issue mentioned BEAM-3386. (ran a Beam SQL job on Spark runner)
Vendored calcite with rewriting JDBC url or without rewriting JDBC url both
works for Beam SQL job on Spark runner
[~apilloud] Would you mind sharing how to reproduce this issue in your case? so
that I can validate vendored calcite with rewriting jdbc url.
> Vendored calcite breaks if another calcite is on the class path
> ---
>
> Key: BEAM-8331
> URL: https://issues.apache.org/jira/browse/BEAM-8331
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.15.0, 2.16.0
>Reporter: Andrew Pilloud
>Assignee: Kai Jiang
>Priority: Major
>
> If the beam vendored calcite and a non-vendored calcite are both on the
> classpath, neither version works. This is because the non-JDBC calcite path
> uses JDBC as a easy way to perform reflection. (This affects the non-JDBC
> version of calcite.) We need to rewrite the calcite JDBC urls as part of our
> vendoring (for example 'jdbc:calcite:' to 'jdbc:beam-vendor-calcite:').
> Example of where this happens:
> [https://github.com/apache/calcite/blob/0cce229903a845a7b8ed36cf86d6078fd82d73d3/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L175]
>
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException: Property
> 'org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem' not
> valid for plugin type org.apache.calcite.rel.type.RelDataTypeSystem
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:160)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:115)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.(ZetaSQLPlannerImpl.java:86)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.(ZetaSQLQueryPlanner.java:55){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8331) Vendored calcite breaks if another calcite is on the class path
[
https://issues.apache.org/jira/browse/BEAM-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17012693#comment-17012693
]
Kai Jiang commented on BEAM-8331:
-
I have built a vendored calcite with rewriting jdbc url locally. However, I
have tested the issue mentioned BEAM-3386. (ran a Beam SQL job on Spark runner)
Vendored calcite with rewriting JDBC url or without rewriting JDBC url both
works for Beam SQL job on Spark runner
[~apilloud] Would you mind sharing how to reproduce this issue in your case? so
that I can validate vendored calcite with rewriting jdbc url.
> Vendored calcite breaks if another calcite is on the class path
> ---
>
> Key: BEAM-8331
> URL: https://issues.apache.org/jira/browse/BEAM-8331
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.15.0, 2.16.0
>Reporter: Andrew Pilloud
>Assignee: Kai Jiang
>Priority: Major
>
> If the beam vendored calcite and a non-vendored calcite are both on the
> classpath, neither version works. This is because the non-JDBC calcite path
> uses JDBC as a easy way to perform reflection. (This affects the non-JDBC
> version of calcite.) We need to rewrite the calcite JDBC urls as part of our
> vendoring (for example 'jdbc:calcite:' to 'jdbc:beam-vendor-calcite:').
> Example of where this happens:
> [https://github.com/apache/calcite/blob/0cce229903a845a7b8ed36cf86d6078fd82d73d3/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L175]
>
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException: Property
> 'org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem' not
> valid for plugin type org.apache.calcite.rel.type.RelDataTypeSystem
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:160)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:115)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.(ZetaSQLPlannerImpl.java:86)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.(ZetaSQLQueryPlanner.java:55){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8496) remove SDF translators in flink streaming transform translator
[ https://issues.apache.org/jira/browse/BEAM-8496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011288#comment-17011288 ] Kai Jiang commented on BEAM-8496: - For failed nexmark on flink job, it was fixed by [https://github.com/apache/beam/commit/4af507f567605723af940c2af3b2912d0948adf5] actually. > remove SDF translators in flink streaming transform translator > -- > > Key: BEAM-8496 > URL: https://issues.apache.org/jira/browse/BEAM-8496 > Project: Beam > Issue Type: Bug > Components: runner-flink >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: 2.19.0 > > Time Spent: 7.5h > Remaining Estimate: 0h > > Since URN of SDF has been moved to runners-core-construction-java, we need to > remove it. > Otherwise, in failed nexmark Jenkins > [job|https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Flink/4128/console], > it causes duplicated transformer registered in > [PTransformTranslation.KnownTransformPayloadTranslator()|https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java#L290] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-4663) Implement Cost calculations for Cost-Based Optimization (CBO)
[ https://issues.apache.org/jira/browse/BEAM-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang resolved BEAM-4663. - Fix Version/s: Not applicable Resolution: Invalid > Implement Cost calculations for Cost-Based Optimization (CBO) > -- > > Key: BEAM-4663 > URL: https://issues.apache.org/jira/browse/BEAM-4663 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 2h 20m > Remaining Estimate: 0h > > To support CBO, we should implement methods in each Beam*Rel.java. > computeSelfCost(...) as our first step. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-8496) remove SDF translators in flink streaming transform translator
Kai Jiang created BEAM-8496: --- Summary: remove SDF translators in flink streaming transform translator Key: BEAM-8496 URL: https://issues.apache.org/jira/browse/BEAM-8496 Project: Beam Issue Type: Bug Components: runner-flink Reporter: Kai Jiang Assignee: Kai Jiang Since URN of SDF has been moved to runners-core-construction-java, we need to remove it. Otherwise, in failed nexmark Jenkins [job|https://builds.apache.org/job/beam_PostCommit_Java_Nexmark_Flink/4128/console], it causes duplicated transformer registered in [PTransformTranslation.KnownTransformPayloadTranslator()|https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/PTransformTranslation.java#L290] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-8331) Vendored calcite breaks if another calcite is on the class path
[
https://issues.apache.org/jira/browse/BEAM-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Jiang reassigned BEAM-8331:
---
Assignee: Kai Jiang
> Vendored calcite breaks if another calcite is on the class path
> ---
>
> Key: BEAM-8331
> URL: https://issues.apache.org/jira/browse/BEAM-8331
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.15.0, 2.16.0
>Reporter: Andrew Pilloud
>Assignee: Kai Jiang
>Priority: Major
>
> If the beam vendored calcite and a non-vendored calcite are both on the
> classpath, neither version works. This is because the non-JDBC calcite path
> uses JDBC as a easy way to perform reflection. (This affects the non-JDBC
> version of calcite.) We need to rewrite the calcite JDBC urls as part of our
> vendoring (for example 'jdbc:calcite:' to 'jdbc:beam-vendor-calcite:').
> Example of where this happens:
> [https://github.com/apache/calcite/blob/0cce229903a845a7b8ed36cf86d6078fd82d73d3/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L175]
>
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException: Property
> 'org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem' not
> valid for plugin type org.apache.calcite.rel.type.RelDataTypeSystem
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:160)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:115)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.(ZetaSQLPlannerImpl.java:86)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.(ZetaSQLQueryPlanner.java:55){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8331) Vendored calcite breaks if another calcite is on the class path
[
https://issues.apache.org/jira/browse/BEAM-8331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943340#comment-16943340
]
Kai Jiang commented on BEAM-8331:
-
sure. I assigned to myself.
> Vendored calcite breaks if another calcite is on the class path
> ---
>
> Key: BEAM-8331
> URL: https://issues.apache.org/jira/browse/BEAM-8331
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Affects Versions: 2.15.0, 2.16.0
>Reporter: Andrew Pilloud
>Priority: Major
>
> If the beam vendored calcite and a non-vendored calcite are both on the
> classpath, neither version works. This is because the non-JDBC calcite path
> uses JDBC as a easy way to perform reflection. (This affects the non-JDBC
> version of calcite.) We need to rewrite the calcite JDBC urls as part of our
> vendoring (for example 'jdbc:calcite:' to 'jdbc:beam-vendor-calcite:').
> Example of where this happens:
> [https://github.com/apache/calcite/blob/0cce229903a845a7b8ed36cf86d6078fd82d73d3/core/src/main/java/org/apache/calcite/tools/Frameworks.java#L175]
>
> {code:java}
> java.lang.RuntimeException: java.lang.RuntimeException: Property
> 'org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem' not
> valid for plugin type org.apache.calcite.rel.type.RelDataTypeSystem
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:160)
> at
> org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:115)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLPlannerImpl.(ZetaSQLPlannerImpl.java:86)
> at
> org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner.(ZetaSQLQueryPlanner.java:55){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-2466) Add Kafka Streams runner
[ https://issues.apache.org/jira/browse/BEAM-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918157#comment-16918157 ] Kai Jiang commented on BEAM-2466: - Nice progress. I reassigned the ticket to [~winkelman.kyle]. Definitely, I am very interested in collaborating in more features and testing. > Add Kafka Streams runner > > > Key: BEAM-2466 > URL: https://issues.apache.org/jira/browse/BEAM-2466 > Project: Beam > Issue Type: Wish > Components: runner-ideas >Reporter: Lorand Peter Kasler >Assignee: Kyle Winkelman >Priority: Minor > > Kafka Streams (https://kafka.apache.org/documentation/streams) has more and > more features that could make it a viable candidate for a streaming runner. > It uses DataFlow-like model -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (BEAM-2466) Add Kafka Streams runner
[ https://issues.apache.org/jira/browse/BEAM-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang reassigned BEAM-2466: --- Assignee: Kyle Winkelman (was: Kai Jiang) > Add Kafka Streams runner > > > Key: BEAM-2466 > URL: https://issues.apache.org/jira/browse/BEAM-2466 > Project: Beam > Issue Type: Wish > Components: runner-ideas >Reporter: Lorand Peter Kasler >Assignee: Kyle Winkelman >Priority: Minor > > Kafka Streams (https://kafka.apache.org/documentation/streams) has more and > more features that could make it a viable candidate for a streaming runner. > It uses DataFlow-like model -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (BEAM-5820) Vendor Calcite
[ https://issues.apache.org/jira/browse/BEAM-5820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang reassigned BEAM-5820: --- Assignee: Kai Jiang > Vendor Calcite > -- > > Key: BEAM-5820 > URL: https://issues.apache.org/jira/browse/BEAM-5820 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (BEAM-7728) Support ParquetTable in SQL
[ https://issues.apache.org/jira/browse/BEAM-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang resolved BEAM-7728. - Resolution: Fixed Fix Version/s: 2.15.0 > Support ParquetTable in SQL > --- > > Key: BEAM-7728 > URL: https://issues.apache.org/jira/browse/BEAM-7728 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Fix For: 2.15.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (BEAM-7728) Support ParquetTable in SQL
[ https://issues.apache.org/jira/browse/BEAM-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896822#comment-16896822 ] Kai Jiang edited comment on BEAM-7728 at 7/31/19 6:49 AM: -- to be implemented writer was (Author: vectorijk): to be implement writer > Support ParquetTable in SQL > --- > > Key: BEAM-7728 > URL: https://issues.apache.org/jira/browse/BEAM-7728 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (BEAM-7728) Support ParquetTable in SQL
[ https://issues.apache.org/jira/browse/BEAM-7728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896822#comment-16896822 ] Kai Jiang commented on BEAM-7728: - to be implement writer > Support ParquetTable in SQL > --- > > Key: BEAM-7728 > URL: https://issues.apache.org/jira/browse/BEAM-7728 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (BEAM-7728) Support ParquetTable in SQL
Kai Jiang created BEAM-7728: --- Summary: Support ParquetTable in SQL Key: BEAM-7728 URL: https://issues.apache.org/jira/browse/BEAM-7728 Project: Beam Issue Type: New Feature Components: dsl-sql Reporter: Kai Jiang Assignee: Kai Jiang -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (BEAM-5822) Vendor bytebuddy
[ https://issues.apache.org/jira/browse/BEAM-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang reassigned BEAM-5822: --- Assignee: Kai Jiang > Vendor bytebuddy > > > Key: BEAM-5822 > URL: https://issues.apache.org/jira/browse/BEAM-5822 > Project: Beam > Issue Type: Sub-task > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kai Jiang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-2466) Add Kafka Streams runner
[ https://issues.apache.org/jira/browse/BEAM-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16767394#comment-16767394 ] Kai Jiang commented on BEAM-2466: - [~teabot] IMHO, There should not exist a 'batch processing' concept for KStreams. I think we needs limit Beam only in streaming mode for KafkaStreamsRunner. [~aromanenko] I think it requires kafka topic as streaming input. Internally, KStream source utilized kafka topics as input. PoC branch: https://github.com/vectorijk/beam/tree/kafka-stream Welcome any ideas! > Add Kafka Streams runner > > > Key: BEAM-2466 > URL: https://issues.apache.org/jira/browse/BEAM-2466 > Project: Beam > Issue Type: Wish > Components: runner-ideas >Reporter: Lorand Peter Kasler >Assignee: Kai Jiang >Priority: Minor > Labels: triaged > > Kafka Streams (https://kafka.apache.org/documentation/streams) has more and > more features that could make it a viable candidate for a streaming runner. > It uses DataFlow-like model -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3386) Dependency conflict when Calcite is included in a project.
[ https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751942#comment-16751942 ] Kai Jiang commented on BEAM-3386: - [~kenn] this (without relocation) sounds pretty useful. let me checkout the thread quickly. > Dependency conflict when Calcite is included in a project. > -- > > Key: BEAM-3386 > URL: https://issues.apache.org/jira/browse/BEAM-3386 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0 >Reporter: Austin Haas >Assignee: Kai Jiang >Priority: Critical > > When Calcite (v. 1.13.0) is included in a project that also includes Beam and > the Beam SQL extension, then the following error is thrown when trying to run > Beam code. > ClassCastException > org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot > be cast to org.apache.calcite.rel.type.RelDataTypeSystem > org.apache.calcite.jdbc.CalciteConnectionImpl. > (CalciteConnectionImpl.java:120) > > org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. > (CalciteJdbc41Factory.java:114) > org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection > (CalciteJdbc41Factory.java:59) > org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection > (CalciteJdbc41Factory.java:44) > org.apache.calcite.jdbc.CalciteFactory.newConnection > (CalciteFactory.java:53) > org.apache.calcite.avatica.UnregisteredDriver.connect > (UnregisteredDriver.java:138) > java.sql.DriverManager.getConnection (DriverManager.java:664) > java.sql.DriverManager.getConnection (DriverManager.java:208) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare > (Frameworks.java:145) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner > (Frameworks.java:106) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready > (PlannerImpl.java:140) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse > (PlannerImpl.java:170) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3386) Dependency conflict when Calcite is included in a project.
[ https://issues.apache.org/jira/browse/BEAM-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16751928#comment-16751928 ] Kai Jiang commented on BEAM-3386: - [~iemejia] Thanks! I will take some time trying to work on vendor calcite. > Dependency conflict when Calcite is included in a project. > -- > > Key: BEAM-3386 > URL: https://issues.apache.org/jira/browse/BEAM-3386 > Project: Beam > Issue Type: Bug > Components: dsl-sql >Affects Versions: 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0 >Reporter: Austin Haas >Assignee: Kai Jiang >Priority: Critical > > When Calcite (v. 1.13.0) is included in a project that also includes Beam and > the Beam SQL extension, then the following error is thrown when trying to run > Beam code. > ClassCastException > org.apache.beam.sdk.extensions.sql.impl.planner.BeamRelDataTypeSystem cannot > be cast to org.apache.calcite.rel.type.RelDataTypeSystem > org.apache.calcite.jdbc.CalciteConnectionImpl. > (CalciteConnectionImpl.java:120) > > org.apache.calcite.jdbc.CalciteJdbc41Factory$CalciteJdbc41Connection. > (CalciteJdbc41Factory.java:114) > org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection > (CalciteJdbc41Factory.java:59) > org.apache.calcite.jdbc.CalciteJdbc41Factory.newConnection > (CalciteJdbc41Factory.java:44) > org.apache.calcite.jdbc.CalciteFactory.newConnection > (CalciteFactory.java:53) > org.apache.calcite.avatica.UnregisteredDriver.connect > (UnregisteredDriver.java:138) > java.sql.DriverManager.getConnection (DriverManager.java:664) > java.sql.DriverManager.getConnection (DriverManager.java:208) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPrepare > (Frameworks.java:145) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.tools.Frameworks.withPlanner > (Frameworks.java:106) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.ready > (PlannerImpl.java:140) > > org.apache.beam.sdks.java.extensions.sql.repackaged.org.apache.calcite.prepare.PlannerImpl.parse > (PlannerImpl.java:170) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5130) Support ALL (sub query)
[ https://issues.apache.org/jira/browse/BEAM-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Jiang reassigned BEAM-5130: --- Assignee: Kai Jiang > Support ALL (sub query) > --- > > Key: BEAM-5130 > URL: https://issues.apache.org/jira/browse/BEAM-5130 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Rui Wang >Assignee: Kai Jiang >Priority: Major > > Example: > > SELECT order_id FROM order WHERE item_id = ALL (SELECT item_id FROM > special_item_table). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
