[
https://issues.apache.org/jira/browse/BEAM-10416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Webb updated BEAM-10416:
-----------------------------
Resolution: Won't Fix
Status: Resolved (was: Triage Needed)
old issue - resolving
> Getting error "message" : "The KMS key does not contain a location." while
> using withKmsKey() option at runtime in dataflow job
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-10416
> URL: https://issues.apache.org/jira/browse/BEAM-10416
> Project: Beam
> Issue Type: Bug
> Components: beam-model, runner-dataflow, test-failures
> Environment: windows 10 (dataflow runner)
> Reporter: Arzoo Khan
> Priority: P3
> Labels: GCP
>
>
> I have dataflow job that is written using apache beam.Here I am loading the
> data from one table to another table the mode of table is write truncate and
> the table contains sensitive data so table configured with kms key. I am
> following this code and trying to fetch the kms key associated with table and
> setting this key at the time of (BigQueryIO.write) while writing data into
> table.
> [https://github.com/apache/beam/blob/49403c00499f7f87c2bf9a4c63dec8ebd68d640d/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java#L91-L100]
>
>
> {{finalOutput.apply("Write success rows to Sensitive BigQuery",
> BigQueryIO.writeTableRows().withoutValidation()
> .to(options.getTargetTable())
> .withKmsKey(NestedValueProvider.of(options.getTargetTable(),new
> FetchingKMSKey(options.getProject())).toString()).withoutValidation()
> .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
> .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));}}
>
>
> {{public class FetchingKMSKey implements SerializableFunction<String, String>
> {
> private static final BigqueryClient BQ_CLIENT = new
> BigqueryClient("dataMainPipeline");
> public static final Logger LOG =
> LoggerFactory.getLogger(FetchingKMSKey.class);
> Table table ;
> String tableName;
> String project;
> public FetchingKMSKey(String project)
> { this.project=project; }
> public String apply(String tableName)
> {
> String[] name= tableName.split(Pattern.quote("."));
> try
> { table = BQ_CLIENT.getTableResource(project,name[0],name[1]); }
> catch (IOException e)
> {
> LOG.error(String.format("exception occured: %s", e.getMessage()));
> }
> catch (InterruptedException e)
> {
> LOG.error(String.format("exception occured: %s", e.getMessage()));
> }
> String kmsKey =table.getEncryptionConfiguration().getKmsKeyName();
> return kmsKey;
> }
> }}}
> {{}}
>
> After creating a template when I am running the job I am getting error.
>
> {{"location" : "US",
> "errors" : [ {
> "message" : "The KMS key does not contain a location.",
> "reason" : "invalid"
> } ],}}
>
> I am completely stuck. anyone could help?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
