[
https://issues.apache.org/jira/browse/BEAM-13294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Pilloud updated BEAM-13294:
----------------------------------
Description:
The join always fail when given a batch column that is Nullable and a streaming
column that is Not Nullable. This was discovered in BeamSideInputJoinRel but is
likely an issue in the Beam core join library.
Trivial reproduction with a small test diff:
./gradlew :runners:direct-java:needsRunnerTests --tests
org.apache.beam.sdk.schemas.transforms.JoinTest.testInnerJoinDifferentKeys
{code}
---
a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
+++
b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
@@ -51,7 +51,7 @@ public class JoinTest {
.build();
private static final Schema CG_SCHEMA_2 =
Schema.builder()
- .addStringField("user2")
+ .addNullableField("user2", Schema.FieldType.STRING)
.addInt32Field("count2")
.addStringField("country2")
.build();
{code}
was:
The join always fail when given a batch column that is Nullable and a streaming
column that is Not Nullable. This was discovered in BeamSideInputJoinRel but is
likely an issue in the Beam core join library.
Trivial reproduction with a small test diff:
./gradlew :runners:direct-java:needsRunnerTests --tests
org.apache.beam.sdk.schemas.transforms.JoinTest
{code}
---
a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
+++
b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
@@ -51,7 +51,7 @@ public class JoinTest {
.build();
private static final Schema CG_SCHEMA_2 =
Schema.builder()
- .addStringField("user2")
+ .addNullableField("user2", Schema.FieldType.STRING)
.addInt32Field("count2")
.addStringField("country2")
.build();
{code}
> innerBroadcastJoin fails when batch column Nullable, streaming column not
> -------------------------------------------------------------------------
>
> Key: BEAM-13294
> URL: https://issues.apache.org/jira/browse/BEAM-13294
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql, dsl-sql-zetasql, sdk-java-core
> Reporter: Andrew Pilloud
> Assignee: Andrew Pilloud
> Priority: P1
>
> The join always fail when given a batch column that is Nullable and a
> streaming column that is Not Nullable. This was discovered in
> BeamSideInputJoinRel but is likely an issue in the Beam core join library.
> Trivial reproduction with a small test diff:
> ./gradlew :runners:direct-java:needsRunnerTests --tests
> org.apache.beam.sdk.schemas.transforms.JoinTest.testInnerJoinDifferentKeys
> {code}
> ---
> a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
> +++
> b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/JoinTest.java
> @@ -51,7 +51,7 @@ public class JoinTest {
> .build();
> private static final Schema CG_SCHEMA_2 =
> Schema.builder()
> - .addStringField("user2")
> + .addNullableField("user2", Schema.FieldType.STRING)
> .addInt32Field("count2")
> .addStringField("country2")
> .build();
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
