The complete story is, TestPubsub invokes TestPipeline so it requires configuration in build.gradle. BeamSQL JDBC driver does not invoke TestPipeline so it requires "SET project".
Thanks Andrew. -Rui On Thu, Jul 26, 2018 at 6:01 PM Andrew Pilloud <[email protected]> wrote: > Your second stacktrace isn't going through SQL. It looks like you are > using the normal test path there. Have you tried setting in both places? > > On Thu, Jul 26, 2018, 5:48 PM Rui Wang <[email protected]> wrote: > >> Ah, SET project = apache-beam-testing; gives the following exception: >> >> io.grpc.StatusRuntimeException: PERMISSION_DENIED: User not authorized to >> perform this action. >> at >> io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:222) >> at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:203) >> at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:132) >> at >> com.google.pubsub.v1.PublisherGrpc$PublisherBlockingStub.createTopic(PublisherGrpc.java:666) >> at >> org.apache.beam.sdk.io.gcp.pubsub.PubsubGrpcClient.createTopic(PubsubGrpcClient.java:302) >> at >> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.initializePubsub(TestPubsub.java:102) >> at >> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.access$200(TestPubsub.java:42) >> at >> org.apache.beam.sdk.io.gcp.pubsub.TestPubsub$1.evaluate(TestPubsub.java:85) >> >> >> which should not happen because jenkins should already have credentials >> to access GCP. >> >> -Rui >> >> On Thu, Jul 26, 2018 at 3:30 PM Andrew Pilloud <[email protected]> >> wrote: >> >>> Beam SQL CLI does not accept beamTestPipelineOptions. Also, gradle is >>> invoking your test not the Beam SQL CLI. You'll need to set the options in >>> your integration test by executing 'SET project = ...' in the Beam SQL >>> connection you've launched for test. >>> >>> Andrew >>> >>> On Thu, Jul 26, 2018 at 3:06 PM Rui Wang <[email protected]> wrote: >>> >>>> The code path of reading pubsub through BeamSQL goes through this line >>>> of code: >>>> >>>> PubsubIO.Read<PubsubMessage> read = >>>> PubsubIO.readMessagesWithAttributes().fromTopic(getTopic()); >>>> >>>> -Rui >>>> >>>> On Thu, Jul 26, 2018 at 2:58 PM Rui Wang <[email protected]> wrote: >>>> >>>>> Hi Community, >>>>> >>>>> I am facing a runtime exception when I try to read from pubsub by Beam >>>>> SQL in JUnit tests (PR: https://github.com/apache/beam/pull/6006). >>>>> The exception is "Cannot create subscription because pipeline option >>>>> 'project' not specified". Based on existing JUnit tests which also read >>>>> from pubsub by Beam SQL, I added the following code to my >>>>> build.gradle and it seems didn't work. Is there someone who could know >>>>> what's wrong in my .gradle file? >>>>> >>>>> >>>>> task endToEndTest(type: Test) { >>>>> group = "Verification" >>>>> def gcpProject = project.findProperty('gcpProject') ?: >>>>> 'apache-beam-testing' >>>>> def gcsTempRoot = project.findProperty('gcsTempRoot') ?: >>>>> 'gs://temp-storage-for-end-to-end-tests/' >>>>> >>>>> // Disable Gradle cache (it should not be used because the IT's won't >>>>> run). >>>>> outputs.upToDateWhen { false } >>>>> >>>>> def pipelineOptions = [ >>>>> "--project=${gcpProject}", >>>>> "--tempLocation=${gcsTempRoot}", >>>>> "--blockOnRun=false"] >>>>> >>>>> systemProperty "beamTestPipelineOptions", >>>>> JsonOutput.toJson(pipelineOptions) >>>>> >>>>> include '**/BeamSqlLineIT.class' >>>>> classpath = >>>>> project(":beam-sdks-java-extensions-sql-jdbc").sourceSets.test.runtimeClasspath >>>>> testClassesDirs = >>>>> files(project(":beam-sdks-java-extensions-sql-jdbc").sourceSets.test.output.classesDirs) >>>>> useJUnit { } >>>>> } >>>>> >>>>> task postCommit { >>>>> group = "Verification" >>>>> description = "Various integration tests" >>>>> dependsOn endToEndTest >>>>> } >>>>> >>>>> >>>>> -Rui >>>>> >>>>
