tonytanger commented on code in PR #25364:
URL: https://github.com/apache/beam/pull/25364#discussion_r1100757973
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/changestreams/dofn/InitializeDoFn.java:
##########
@@ -53,6 +52,25 @@ public void
processElement(OutputReceiver<com.google.cloud.Timestamp> receiver)
LOG.info(daoFactory.getStreamTableDebugString());
LOG.info(daoFactory.getMetadataTableDebugString());
LOG.info("ChangeStreamName: " + daoFactory.getChangeStreamName());
+ if (!daoFactory
+ .getMetadataTableAdminDao()
+
.isAppProfileSingleClusterAndTransactional(this.metadataTableAppProfileId)) {
+ LOG.error(
+ "App profile id '"
+ + metadataTableAppProfileId
+ + "' provided to access metadata table needs to use
single-cluster routing policy"
+ + " and allow single-row transactions.");
+ // Terminate this pipeline now.
+ return;
Review Comment:
That's a good suggestion. I think fail fast is a good idea. The biggest
worry that I have is the pipeline submitter might not have the necessary
permissions to interact with Cloud Bigtable. Whereas the service account
attached to the dataflow workers will.
I think I will need some extra time to evaluate this approach. So for now, I
think we should go ahead for now without checking in `expand`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
