[jira] [Commented] (BEAM-5420) BigtableIO tries to get runtime parameters when collecting display data at pipeline construction time
[
https://issues.apache.org/jira/browse/BEAM-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629346#comment-16629346
]
Kevin Si commented on BEAM-5420:
Or we use the following method to get display data from runtime parameters. The
point is it shouldn't call the get() method.
/** Create a display item for the specified key and \{@link ValueProvider}. */
public static ItemSpec item(String key, @Nullable ValueProvider value) {
if (value == null) {
return item(key, Type.STRING, null);
}
if (value.isAccessible()) {
Object got = value.get();
if (got == null) {
return item(key, Type.STRING, null);
}
Type type = inferType(got);
if (type != null) {
return item(key, type, got);
}
}
// General case: not null and type not inferable. Fall back to toString of the
VP itself.
return item(key, Type.STRING, String.valueOf(value));
}
> BigtableIO tries to get runtime parameters when collecting display data at
> pipeline construction time
> -
>
> Key: BEAM-5420
> URL: https://issues.apache.org/jira/browse/BEAM-5420
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For example:
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java#L165]
> At Dataflow pipeline construction time calling getProjectId() gives an error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5420) BigtableIO tries to get runtime parameters when collecting display data at pipeline construction time
[ https://issues.apache.org/jira/browse/BEAM-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16629256#comment-16629256 ] Kevin Si commented on BEAM-5420: The reason is that the display data is read at pipeline construction time only and never get updated at runtime. This means that the UI will show these dummy values, which is super ugly. I am going to update the logic and remove runtime parameters from the display data. > BigtableIO tries to get runtime parameters when collecting display data at > pipeline construction time > - > > Key: BEAM-5420 > URL: https://issues.apache.org/jira/browse/BEAM-5420 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Kevin Si >Assignee: Chamikara Jayalath >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > For example: > [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java#L165] > At Dataflow pipeline construction time calling getProjectId() gives an error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5420) BigtableIO tries to get runtime parameters when collecting display data at pipeline construction time
[
https://issues.apache.org/jira/browse/BEAM-5420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16627530#comment-16627530
]
Kevin Si commented on BEAM-5420:
This is the error we get if we try to access runtime parameters when getting
display data:
2018-09-25 08:15:55 INFO DataflowRunner:706 - Executing pipeline on the
Dataflow Service, which will have billing implications related to Google
Compute Engine usage and other Google Cloud Services.
2018-09-25 08:15:55 INFO PackageUtil:315 - Uploading 1 files from
PipelineOptions.filesToStage to staging location to prepare for execution.
2018-09-25 08:15:55 INFO PackageUtil:242 - Uploading
/usr/local/google/_blaze_kevinsi/f06b53e3e81897f185aef98ca1141c77/execroot/google3/blaze-out/k8-fastbuild/bin/cloud/dataflow/testing/integration/teleport/teleport_bigtable_export_import_avro_file_uif_generate_template.runfiles/google3/third_party/java_src/cloud/teleport/teleport-all-bundled.jar
to
gs://deft-guitar-sandbox-e2e-test/df2-teleport-bigtable-to-avro-file-generate-template-prod/09250815492298/staging/teleport-all-bundled-Lato7X_J6JWX3aVWaViZHg.jar
2018-09-25 08:16:12 INFO PackageUtil:389 - Staging files complete: 0 files
cached, 1 files newly uploaded
Exception in thread "main"
org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder$PopulateDisplayDataException:
Error while populating display data for component
'org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource': Value only
available at runtime, but accessed from a non-runtime context:
RuntimeValueProvider\{propertyName=bigtableTableId, default=null}
at
org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:792)
at
org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:741)
at org.apache.beam.sdk.io.Read$Bounded.populateDisplayData(Read.java:119)
at
org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:783)
at
org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.access$100(DisplayData.java:711)
at org.apache.beam.sdk.transforms.display.DisplayData.from(DisplayData.java:78)
at
org.apache.beam.runners.core.construction.PTransformTranslation.translateAppliedPTransform(PTransformTranslation.java:381)
at
org.apache.beam.runners.core.construction.PTransformTranslation$KnownTransformPayloadTranslator.translate(PTransformTranslation.java:323)
at
org.apache.beam.runners.core.construction.PTransformTranslation.toProto(PTransformTranslation.java:141)
at
org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:97)
at
org.apache.beam.runners.core.construction.PipelineTranslation$1.visitPrimitiveTransform(PipelineTranslation.java:72)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at
org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:44)
at
org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:164)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:732)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:177)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
at
com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:95)
Caused by: java.lang.IllegalStateException: Value only available at runtime,
but accessed from a non-runtime context:
RuntimeValueProvider\{propertyName=bigtableTableId, default=null}
at
org.apache.beam.sdk.options.ValueProvider$RuntimeValueProvider.get(ValueProvider.java:228)
at
org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.populateDisplayData(BigtableIO.java:1098)
at
org.apache.beam.sdk.transforms.display.DisplayData$InternalBuilder.include(DisplayData.java:783)
... 23 more
> BigtableIO tries to get runtime parameters when collecting display data at
> pipeline construction time
> -
>
> Key: BEAM-5420
> URL: https://issues.apache.org/jira/browse/BEAM-5420
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>
> For example:
>
[jira] [Created] (BEAM-5420) BigtableIO tries to get runtime parameters when collecting display data at pipeline construction time
Kevin Si created BEAM-5420: -- Summary: BigtableIO tries to get runtime parameters when collecting display data at pipeline construction time Key: BEAM-5420 URL: https://issues.apache.org/jira/browse/BEAM-5420 Project: Beam Issue Type: Bug Components: io-java-gcp Reporter: Kevin Si Assignee: Chamikara Jayalath For example: [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableConfig.java#L165] At Dataflow pipeline construction time calling getProjectId() gives an error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off
[ https://issues.apache.org/jira/browse/BEAM-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612000#comment-16612000 ] Kevin Si commented on BEAM-5364: I will send out a PR to fix this. The change should be small. > BigtableIO source tries to validate table ID even though validation is > turned off > -- > > Key: BEAM-5364 > URL: https://issues.apache.org/jira/browse/BEAM-5364 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Kevin Si >Assignee: Chamikara Jayalath >Priority: Minor > > [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg] > The validation can be turned off with following: > BigtableIO.read() > .withoutValidation() // skip validation when constructing the > pipelline. > A Dataflow template cannot be constructed due to this validation failure. > > Error log when trying to construct a template: > Exception in thread "main" java.lang.IllegalArgumentException: tableId was > not supplied > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084) > at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95) > at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44) > at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56) > at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182) > at > com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off
[ https://issues.apache.org/jira/browse/BEAM-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611998#comment-16611998 ] Kevin Si commented on BEAM-5364: Sorry! I meant to say a blocker for "Cloud Bigtable Avro import and export Dataflow Template". I set the priority to "Minor". > BigtableIO source tries to validate table ID even though validation is > turned off > -- > > Key: BEAM-5364 > URL: https://issues.apache.org/jira/browse/BEAM-5364 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Kevin Si >Assignee: Chamikara Jayalath >Priority: Minor > > [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg] > The validation can be turned off with following: > BigtableIO.read() > .withoutValidation() // skip validation when constructing the > pipelline. > A Dataflow template cannot be constructed due to this validation failure. > > Error log when trying to construct a template: > Exception in thread "main" java.lang.IllegalArgumentException: tableId was > not supplied > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084) > at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95) > at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44) > at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56) > at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182) > at > com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off
[ https://issues.apache.org/jira/browse/BEAM-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Si updated BEAM-5364: --- Priority: Minor (was: Blocker) > BigtableIO source tries to validate table ID even though validation is > turned off > -- > > Key: BEAM-5364 > URL: https://issues.apache.org/jira/browse/BEAM-5364 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Kevin Si >Assignee: Chamikara Jayalath >Priority: Minor > > [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg] > The validation can be turned off with following: > BigtableIO.read() > .withoutValidation() // skip validation when constructing the > pipelline. > A Dataflow template cannot be constructed due to this validation failure. > > Error log when trying to construct a template: > Exception in thread "main" java.lang.IllegalArgumentException: tableId was > not supplied > at > com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084) > at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95) > at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44) > at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423) > at > org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179) > at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) > at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488) > at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56) > at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182) > at > com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off
Kevin Si created BEAM-5364: -- Summary: BigtableIO source tries to validate table ID even though validation is turned off Key: BEAM-5364 URL: https://issues.apache.org/jira/browse/BEAM-5364 Project: Beam Issue Type: Bug Components: io-java-gcp Reporter: Kevin Si Assignee: Chamikara Jayalath [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg] The validation can be turned off with following: BigtableIO.read() .withoutValidation() // skip validation when constructing the pipelline. A Dataflow template cannot be constructed due to this validation failure. Error log when trying to construct a template: Exception in thread "main" java.lang.IllegalArgumentException: tableId was not supplied at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122) at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084) at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95) at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85) at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471) at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44) at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167) at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423) at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179) at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537) at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488) at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56) at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182) at com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5272) Randomize the reduced splits in BigtableIO so that multiple workers may not hit the same tablet server
[ https://issues.apache.org/jira/browse/BEAM-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598663#comment-16598663 ] Kevin Si commented on BEAM-5272: https://github.com/apache/beam/pull/6308 > Randomize the reduced splits in BigtableIO so that multiple workers may not > hit the same tablet server > -- > > Key: BEAM-5272 > URL: https://issues.apache.org/jira/browse/BEAM-5272 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Kevin Si >Assignee: Chamikara Jayalath >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Randomize the reduced splits in BigtableIO so that multiple workers may not > hit the same tablet server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5272) Randomize the reduced splits in BigtableIO so that multiple workers may not hit the same tablet server
Kevin Si created BEAM-5272: -- Summary: Randomize the reduced splits in BigtableIO so that multiple workers may not hit the same tablet server Key: BEAM-5272 URL: https://issues.apache.org/jira/browse/BEAM-5272 Project: Beam Issue Type: Improvement Components: io-java-gcp Reporter: Kevin Si Assignee: Chamikara Jayalath Randomize the reduced splits in BigtableIO so that multiple workers may not hit the same tablet server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (BEAM-5256) Update Bigtable dependency
[ https://issues.apache.org/jira/browse/BEAM-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Si closed BEAM-5256. -- Resolution: Won't Fix > Update Bigtable dependency > -- > > Key: BEAM-5256 > URL: https://issues.apache.org/jira/browse/BEAM-5256 > Project: Beam > Issue Type: Improvement > Components: dependencies >Affects Versions: 2.6.0 >Reporter: Kevin Si >Priority: Minor > Fix For: 2.7.0 > > > According to the following dependency tree, > "beam-sdks-java-io-google-cloud-platform" is depending on "bigtable-protos". > [INFO] Verbose not supported since maven-dependency-plugin 3.0 [INFO] > com.google.cloud.teleport:google-cloud-teleport-java:jar:0.1-SNAPSHOT [INFO] > \- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.6.0:compile > [INFO] \- com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3:compile > > Instead, it should depends on: > > com.google.api.grpc > proto-google-cloud-bigtable-v2 > 0.24.0 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5256) Update Bigtable dependency
[ https://issues.apache.org/jira/browse/BEAM-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596427#comment-16596427 ] Kevin Si commented on BEAM-5256: Second thought, we may want to use a different version of Bigtable client, so put this change on hold. > Update Bigtable dependency > -- > > Key: BEAM-5256 > URL: https://issues.apache.org/jira/browse/BEAM-5256 > Project: Beam > Issue Type: Improvement > Components: dependencies >Affects Versions: 2.6.0 >Reporter: Kevin Si >Priority: Minor > Fix For: 2.7.0 > > > According to the following dependency tree, > "beam-sdks-java-io-google-cloud-platform" is depending on "bigtable-protos". > [INFO] Verbose not supported since maven-dependency-plugin 3.0 [INFO] > com.google.cloud.teleport:google-cloud-teleport-java:jar:0.1-SNAPSHOT [INFO] > \- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.6.0:compile > [INFO] \- com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3:compile > > Instead, it should depends on: > > com.google.api.grpc > proto-google-cloud-bigtable-v2 > 0.24.0 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5256) Update Bigtable dependency
Kevin Si created BEAM-5256: -- Summary: Update Bigtable dependency Key: BEAM-5256 URL: https://issues.apache.org/jira/browse/BEAM-5256 Project: Beam Issue Type: Improvement Components: dependencies Affects Versions: 2.6.0 Reporter: Kevin Si Fix For: 2.7.0 According to the following dependency tree, "beam-sdks-java-io-google-cloud-platform" is depending on "bigtable-protos". [INFO] Verbose not supported since maven-dependency-plugin 3.0 [INFO] com.google.cloud.teleport:google-cloud-teleport-java:jar:0.1-SNAPSHOT [INFO] \- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.6.0:compile [INFO] \- com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3:compile Instead, it should depends on: com.google.api.grpc proto-google-cloud-bigtable-v2 0.24.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4261) BigtableIO should not try to validate runtime parameters at construction time.
[
https://issues.apache.org/jira/browse/BEAM-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Si updated BEAM-4261:
---
Summary: BigtableIO should not try to validate runtime parameters at
construction time. (was: CloudBigtableIO should not try to validate runtime
parameters at construction time.)
> BigtableIO should not try to validate runtime parameters at construction time.
> --
>
> Key: BEAM-4261
> URL: https://issues.apache.org/jira/browse/BEAM-4261
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The workaround for user is to have some default values set and override them
> at runtime.
> One example of validating runtime parameter at construction time is
> following, and there are could be more.
>
> @Override
> public void validate() {
> ValueProvider tableId = config.getTableId();
> checkArgument(tableId != null && tableId.isAccessible() &&
> !tableId.get().isEmpty(),
> "tableId was not supplied");
> }
>
> A reported issue on stackoverflow:
> [https://stackoverflow.com/questions/49595921/valueprovider-type-parameters-not-getting-honored-at-the-template-execution-time]
>
> One concern I have is that if we disable the validation at construction time,
> how do we validate it at runtime? Ideally, users should use template
> parameter metadata for validation, but that is optional.
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4261) CloudBigtableIO should not try to validate runtime parameters at construction time.
[
https://issues.apache.org/jira/browse/BEAM-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469481#comment-16469481
]
Kevin Si commented on BEAM-4261:
I am not sure. Trying to create template without any runtime parameter
specified should tell us what parameters are used at construction time.
I found this code:
static void validateTableExists(BigtableConfig config, PipelineOptions options)
{
if (config.getValidate() && config.isDataAccessible()) {
checkNotNull(config.getTableId().get());
...
}
}
This is probably something we should be doing at places where we try to
validate runtime parameters.
Places we shouldn't try to get runtime parameters in BigtableIO.java.
# Line#995: checkArgument(tableId != null && tableId.isAccessible() &&
!tableId.get().isEmpty(),
# Line#1003: builder.add(DisplayData.item("tableId", config.getTableId().get())
> CloudBigtableIO should not try to validate runtime parameters at construction
> time.
> ---
>
> Key: BEAM-4261
> URL: https://issues.apache.org/jira/browse/BEAM-4261
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The workaround for user is to have some default values set and override them
> at runtime.
> One example of validating runtime parameter at construction time is
> following, and there are could be more.
>
> @Override
> public void validate() {
> ValueProvider tableId = config.getTableId();
> checkArgument(tableId != null && tableId.isAccessible() &&
> !tableId.get().isEmpty(),
> "tableId was not supplied");
> }
>
> A reported issue on stackoverflow:
> [https://stackoverflow.com/questions/49595921/valueprovider-type-parameters-not-getting-honored-at-the-template-execution-time]
>
> One concern I have is that if we disable the validation at construction time,
> how do we validate it at runtime? Ideally, users should use template
> parameter metadata for validation, but that is optional.
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-4261) CloudBigtableIO should not try to validate runtime parameters at construction time.
[
https://issues.apache.org/jira/browse/BEAM-4261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469424#comment-16469424
]
Kevin Si commented on BEAM-4261:
I have not tried withoutValidation(). Maybe that can be a workaround.
I think the point is that that code should *just* work; without extra thinking
from our user. This means switching off validation should be automated.
> CloudBigtableIO should not try to validate runtime parameters at construction
> time.
> ---
>
> Key: BEAM-4261
> URL: https://issues.apache.org/jira/browse/BEAM-4261
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The workaround for user is to have some default values set and override them
> at runtime.
> One example of validating runtime parameter at construction time is
> following, and there are could be more.
>
> @Override
> public void validate() {
> ValueProvider tableId = config.getTableId();
> checkArgument(tableId != null && tableId.isAccessible() &&
> !tableId.get().isEmpty(),
> "tableId was not supplied");
> }
>
> A reported issue on stackoverflow:
> [https://stackoverflow.com/questions/49595921/valueprovider-type-parameters-not-getting-honored-at-the-template-execution-time]
>
> One concern I have is that if we disable the validation at construction time,
> how do we validate it at runtime? Ideally, users should use template
> parameter metadata for validation, but that is optional.
>
>
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-4261) CloudBigtableIO should not try to validate runtime parameters at construction time.
Kevin Si created BEAM-4261:
--
Summary: CloudBigtableIO should not try to validate runtime
parameters at construction time.
Key: BEAM-4261
URL: https://issues.apache.org/jira/browse/BEAM-4261
Project: Beam
Issue Type: Bug
Components: io-java-gcp
Reporter: Kevin Si
Assignee: Chamikara Jayalath
The workaround for user is to have some default values set and override them at
runtime.
One example of validating runtime parameter at construction time is following,
and there are could be more.
@Override
public void validate() {
ValueProvider tableId = config.getTableId();
checkArgument(tableId != null && tableId.isAccessible() &&
!tableId.get().isEmpty(),
"tableId was not supplied");
}
A reported issue on stackoverflow:
[https://stackoverflow.com/questions/49595921/valueprovider-type-parameters-not-getting-honored-at-the-template-execution-time]
One concern I have is that if we disable the validation at construction time,
how do we validate it at runtime? Ideally, users should use template parameter
metadata for validation, but that is optional.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
