[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655094#comment-15655094
 ] 

ASF GitHub Bot commented on BEAM-939:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1336


> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654909#comment-15654909
 ] 

ASF GitHub Bot commented on BEAM-939:
-

GitHub user tgroh opened a pull request:

https://github.com/apache/incubator-beam/pull/1336

[BEAM-939] Update BigtableWriteIT to set Credentials

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/incubator-beam triage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1336.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1336


commit 97c883de70cb8eea16c437fc64c6411de7daeea8
Author: Thomas Groh 
Date:   2016-11-10T19:32:55Z

Update BigtableWriteIT to set Credentials




> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654776#comment-15654776
 ] 

ASF GitHub Bot commented on BEAM-939:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1308


> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646506#comment-15646506
 ] 

Luke Cwik commented on BEAM-939:


Yes, did that in https://github.com/apache/incubator-beam/pull/1308

Unfortunately, the fact that we had to pass around the BigtableService instance 
for testing reasons made this more difficult than just deferring for 
PipelineOptions when its available in the few places we need the service.

Added you and Thomas Groh for review.

> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646406#comment-15646406
 ] 

ASF GitHub Bot commented on BEAM-939:
-

GitHub user lukecwik opened a pull request:

https://github.com/apache/incubator-beam/pull/1308

[BEAM-939] Use the credentials from GcpOptions instead of relying on 
BigtableOptions

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

This allows for us to create a version of the service depending on the 
PipelineOptions passed in at this time. This uses a function which takes in the 
contextual PipelineOptions and creates a service from it allowing us to not 
serialize the Credentials within the BigtableOptions.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lukecwik/incubator-beam beam939

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1308.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1308


commit 5fd9cdc0c843cf126ce0f2e6c6c7563471d6b335
Author: Luke Cwik 
Date:   2016-11-08T03:37:42Z

[BEAM-939] Use the credentials from GcpOptions instead of relying on the 
BigtableOptions to provide them.




> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
>Priority: Minor
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> 

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[Daniel Halperin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646161#comment-15646161
 ] 

Daniel Halperin commented on BEAM-939:
--

So we need to in our tests and in our IO set the {{credentialOptions}} in the 
{{BigtableOptions}}: 
https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/master/bigtable-client-core-parent/bigtable-client-core/src/main/java/com/google/cloud/bigtable/config/CredentialOptions.java

> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646147#comment-15646147
 ] 

Luke Cwik commented on BEAM-939:


Turns out that BigtableIO was never using the user specified credentials and 
relying on the application default via com.google.auth.GoogleCredentials. 
Unfortunately between 0.4.0 and 0.6.0, com.google.auth.oauth2.GoogleCredentials 
had a backwards incompatible change (removed the method 
com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials).

It seems like the proper way to fix this is to pass through the 
com.google.auth.Credentials object from pipeline options through.

> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-939) New credentials code broke Dataflow runner

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646116#comment-15646116
 ] 

Luke Cwik commented on BEAM-939:


Taking a look

> New credentials code broke Dataflow runner
> --
>
> Key: BEAM-939
> URL: https://issues.apache.org/jira/browse/BEAM-939
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Affects Versions: Not applicable
>Reporter: Daniel Halperin
>Assignee: Luke Cwik
> Fix For: 0.4.0-incubating
>
>
> https://builds.apache.org/view/Beam/job/beam_PostCommit_MavenVerify/1753/
> {code}
> java.lang.NoSuchMethodError: 
> com.google.auth.oauth2.GoogleCredentials.getApplicationDefault(Lcom/google/api/client/http/HttpTransport;)Lcom/google/auth/oauth2/GoogleCredentials;
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getApplicationDefaultCredential(CredentialFactory.java:207)
>   at 
> com.google.cloud.bigtable.config.CredentialFactory.getCredentials(CredentialFactory.java:112)
>   at 
> com.google.cloud.bigtable.grpc.io.CredentialInterceptorCache.getCredentialsInterceptor(CredentialInterceptorCache.java:94)
>   at 
> com.google.cloud.bigtable.grpc.BigtableSession.(BigtableSession.java:272)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableServiceImpl.tableExists(BigtableServiceImpl.java:81)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:296)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.validate(BigtableIO.java:185)
>   at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:399)
>   at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:307)
>   at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:47)
>   at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:158)
>   at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableReadIT.testE2EBigtableRead(BigtableReadIT.java:53)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at 
> org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)