After running tasks with --info and --rerun-tasks, I got to know that application default credentials were not set in Docker. I have set GOOGLE_APPLICATION_CREDENTIALS and now it is working fine. I can find the job on the dataflow cloud. Thank you for your guidance.
*Thanks & Regards* *Rehman Murad Ali* Software Engineer Mobile: +92 3452076766 Skype: rehman,muradali <http://venturedive.com/> On Thu, Nov 28, 2019 at 5:29 PM Michał Walenia <[email protected]> wrote: > It seems that Windows CMD doesn't play nice with JSON. I'm not sure what's > the problem in your Docker though, try running gradle with --info and > --rerun-tasks. You'll see more output this way > > On Thu, Nov 28, 2019 at 1:02 PM Rehman Murad Ali < > [email protected]> wrote: > >> Above mentioned screenshot and output logs are run on docker using bash >> shell on Windows. Yes, I have the permission and I was able to upload a >> file on cloud storage as well. >> >> On the other hand, I am getting JsonParsingException when running on >> Window's CMD. See the attached screenshots. >> >> >> >> >> *Thanks & Regards* >> >> >> *Rehman Murad Ali* >> >> Software Engineer >> Mobile: +92 3452076766 >> Skype: rehman,muradali >> >> <http://venturedive.com/> >> >> >> On Thu, Nov 28, 2019 at 4:29 PM Michał Walenia < >> [email protected]> wrote: >> >>> What kind of a shell are you using? Screenshot suggests it's not CMD, >>> but a bash shell on Windows(?) BTW, do you have permissions on the >>> apache-beam-testing project that allow you to start dataflow jobs directly? >>> If you don't, change the project to your org project, get the >>> permissions and try again :) >>> >>> On Thu, Nov 28, 2019 at 11:19 AM Rehman Murad Ali < >>> [email protected]> wrote: >>> >>>> I am using Windows CLI for this command. Moreover, I have tried setting >>>> up docker and run this command which results in "Build Successful" but I >>>> cannot find any jobs running on the cloud console. Here are the output >>>> logs. Is there another way to run Dataflow test case on Windows? >>>> >>>> root@cfbdbfdf72ea:/beam# ./gradlew >>>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>>> -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing >>>> -PgcpProject=apache-beam-testing >>>> -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>> -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>> > Task :buildSrc:compileJava NO-SOURCE >>>> > Task :buildSrc:compileGroovy UP-TO-DATE >>>> > Task :buildSrc:pluginDescriptors UP-TO-DATE >>>> > Task :buildSrc:processResources UP-TO-DATE >>>> > Task :buildSrc:classes UP-TO-DATE >>>> > Task :buildSrc:jar UP-TO-DATE >>>> > Task :buildSrc:assemble UP-TO-DATE >>>> > Task :buildSrc:spotlessGroovy UP-TO-DATE >>>> > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE >>>> > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE >>>> > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE >>>> > Task :buildSrc:spotlessCheck UP-TO-DATE >>>> > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE >>>> > Task :buildSrc:compileTestJava NO-SOURCE >>>> > Task :buildSrc:compileTestGroovy NO-SOURCE >>>> > Task :buildSrc:processTestResources NO-SOURCE >>>> > Task :buildSrc:testClasses UP-TO-DATE >>>> > Task :buildSrc:test NO-SOURCE >>>> > Task :buildSrc:validateTaskProperties UP-TO-DATE >>>> > Task :buildSrc:check UP-TO-DATE >>>> > Task :buildSrc:build UP-TO-DATE >>>> Configuration on demand is an incubating feature. >>>> > Task :model:job-management:extractProto UP-TO-DATE >>>> > Task :model:job-management:processResources UP-TO-DATE >>>> > Task :sdks:java:core:generateAvroProtocol NO-SOURCE >>>> > Task :sdks:java:core:generateAvroJava NO-SOURCE >>>> > Task :sdks:java:core:generateGrammarSource UP-TO-DATE >>>> > Task :sdks:java:core:processResources UP-TO-DATE >>>> > Task >>>> :sdks:java:extensions:google-cloud-platform-core:processResources NO-SOURCE >>>> > Task :sdks:java:extensions:protobuf:extractProto UP-TO-DATE >>>> > Task :sdks:java:extensions:protobuf:processResources NO-SOURCE >>>> > Task :model:pipeline:extractIncludeProto UP-TO-DATE >>>> > Task :model:pipeline:extractProto UP-TO-DATE >>>> > Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE >>>> > Task :examples:java:processResources NO-SOURCE >>>> > Task :examples:java:processTestResources UP-TO-DATE >>>> > Task :runners:core-construction-java:processResources NO-SOURCE >>>> > Task :model:fn-execution:extractProto UP-TO-DATE >>>> > Task :model:pipeline:generateProto UP-TO-DATE >>>> > Task :model:fn-execution:processResources UP-TO-DATE >>>> > Task :sdks:java:fn-execution:processResources NO-SOURCE >>>> > Task :runners:core-java:processResources NO-SOURCE >>>> > Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE >>>> > Task :sdks:java:harness:processResources NO-SOURCE >>>> > Task :runners:java-fn-execution:processResources NO-SOURCE >>>> > Task :runners:local-java:processResources NO-SOURCE >>>> > Task :runners:direct-java:processResources NO-SOURCE >>>> > Task :runners:google-cloud-dataflow-java:processResources UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:examples:compileJava >>>> NO-SOURCE >>>> > Task :runners:google-cloud-dataflow-java:examples:processResources >>>> NO-SOURCE >>>> > Task :runners:google-cloud-dataflow-java:examples:classes UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:examples:compileTestJava >>>> NO-SOURCE >>>> > Task :model:pipeline:compileJava UP-TO-DATE >>>> > Task >>>> :runners:google-cloud-dataflow-java:examples:processTestResources NO-SOURCE >>>> > Task :runners:google-cloud-dataflow-java:examples:testClasses >>>> UP-TO-DATE >>>> > Task :model:pipeline:processResources UP-TO-DATE >>>> > Task :model:pipeline:classes UP-TO-DATE >>>> > Task :model:pipeline:jar UP-TO-DATE >>>> > Task >>>> :runners:google-cloud-dataflow-java:worker:windmill:extractIncludeProto >>>> UP-TO-DATE >>>> > Task :model:pipeline:shadowJar UP-TO-DATE >>>> > Task :model:job-management:extractIncludeProto UP-TO-DATE >>>> > Task :model:job-management:generateProto UP-TO-DATE >>>> > Task :model:fn-execution:extractIncludeProto UP-TO-DATE >>>> > Task :model:fn-execution:generateProto UP-TO-DATE >>>> > Task :model:job-management:compileJava UP-TO-DATE >>>> > Task :model:job-management:classes UP-TO-DATE >>>> > Task :model:job-management:shadowJar UP-TO-DATE >>>> > Task :model:fn-execution:compileJava UP-TO-DATE >>>> > Task :model:fn-execution:classes UP-TO-DATE >>>> > Task :model:fn-execution:shadowJar UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:worker:windmill:extractProto >>>> UP-TO-DATE >>>> > Task >>>> :runners:google-cloud-dataflow-java:worker:windmill:generateProto >>>> UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:worker:windmill:compileJava >>>> UP-TO-DATE >>>> > Task >>>> :runners:google-cloud-dataflow-java:worker:windmill:processResources >>>> UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:worker:windmill:classes >>>> UP-TO-DATE >>>> > Task :sdks:java:core:compileJava UP-TO-DATE >>>> > Task :sdks:java:core:classes UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:worker:windmill:shadowJar >>>> UP-TO-DATE >>>> > Task >>>> :runners:google-cloud-dataflow-java:worker:legacy-worker:processResources >>>> NO-SOURCE >>>> > Task :sdks:java:core:shadowJar UP-TO-DATE >>>> > Task :sdks:java:extensions:google-cloud-platform-core:compileJava >>>> UP-TO-DATE >>>> > Task :sdks:java:extensions:google-cloud-platform-core:classes >>>> UP-TO-DATE >>>> > Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE >>>> > Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE >>>> > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE >>>> > Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE >>>> > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE >>>> > Task :sdks:java:extensions:protobuf:jar UP-TO-DATE >>>> > Task :runners:core-construction-java:compileJava UP-TO-DATE >>>> > Task :runners:core-construction-java:classes UP-TO-DATE >>>> > Task :runners:core-construction-java:jar UP-TO-DATE >>>> > Task :sdks:java:fn-execution:compileJava UP-TO-DATE >>>> > Task :sdks:java:fn-execution:classes UP-TO-DATE >>>> > Task :sdks:java:fn-execution:jar UP-TO-DATE >>>> > Task :runners:core-java:compileJava UP-TO-DATE >>>> > Task :runners:core-java:classes UP-TO-DATE >>>> > Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE >>>> > Task :runners:core-java:jar UP-TO-DATE >>>> > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE >>>> > Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE >>>> > Task :vendor:sdks-java-extensions-protobuf:compileJava UP-TO-DATE >>>> > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE >>>> > Task :vendor:sdks-java-extensions-protobuf:shadowJar UP-TO-DATE >>>> > Task :sdks:java:harness:compileJava UP-TO-DATE >>>> > Task :sdks:java:harness:classes UP-TO-DATE >>>> > Task :sdks:java:harness:jar UP-TO-DATE >>>> > Task :examples:java:compileJava UP-TO-DATE >>>> > Task :examples:java:classes UP-TO-DATE >>>> > Task :examples:java:jar UP-TO-DATE >>>> > Task :sdks:java:harness:shadowJar UP-TO-DATE >>>> > Task :examples:java:compileTestJava UP-TO-DATE >>>> > Task :examples:java:testClasses UP-TO-DATE >>>> > Task :examples:java:testJar UP-TO-DATE >>>> > Task :runners:local-java:compileJava UP-TO-DATE >>>> > Task :runners:local-java:classes UP-TO-DATE >>>> > Task :runners:java-fn-execution:compileJava UP-TO-DATE >>>> > Task :runners:java-fn-execution:classes UP-TO-DATE >>>> > Task :runners:java-fn-execution:jar UP-TO-DATE >>>> > Task :runners:local-java:jar UP-TO-DATE >>>> > Task :runners:direct-java:compileJava UP-TO-DATE >>>> > Task :runners:direct-java:classes UP-TO-DATE >>>> > Task :runners:direct-java:shadowJar UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:compileJava UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:classes UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:jar UP-TO-DATE >>>> > Task >>>> :runners:google-cloud-dataflow-java:worker:legacy-worker:compileJava >>>> UP-TO-DATE >>>> > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:classes >>>> UP-TO-DATE >>>> > Task >>>> :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar >>>> UP-TO-DATE >>>> > Task >>>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>>> FROM-CACHE >>>> >>>> Deprecated Gradle features were used in this build, making it >>>> incompatible with Gradle 6.0. >>>> Use '--warning-mode all' to show the individual deprecation warnings. >>>> See >>>> https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings >>>> >>>> BUILD SUCCESSFUL in 2s >>>> 65 actionable tasks: 1 from cache, 64 up-to-date >>>> root@cfbdbfdf72ea:/beam# >>>> >>>> >>>> >>>> *Thanks & Regards* >>>> >>>> >>>> *Rehman Murad Ali* >>>> >>>> Software Engineer >>>> Mobile: +92 3452076766 >>>> Skype: rehman,muradali >>>> >>>> <http://venturedive.com/> >>>> >>>> >>>> On Thu, Nov 28, 2019 at 12:37 PM Michał Walenia < >>>> [email protected]> wrote: >>>> >>>>> That's interesting, the command works for me (it crashes on >>>>> communication with GCP, but that was expected). From the log output it >>>>> seems you're using Windows. >>>>> Did you use WSL to run this? Do you have an option to use Linux to >>>>> check this command? The JSON created in the Gradle script may be treated >>>>> differently in CMD (due to quoting differences between it and Bash/ZSH/ >>>>> other shells). >>>>> >>>>> I'd try to run this somewhere else other than CMD. >>>>> Good luck! >>>>> >>>>> @Brian: -DbeamPipelineOptions parameter is constructed by Gradle from >>>>> an object via JsonOutput, so it should be a valid object after reading >>>>> back. >>>>> >>>>> On Thu, Nov 28, 2019 at 7:39 AM Rehman Murad Ali < >>>>> [email protected]> wrote: >>>>> >>>>>> Thank you, Brian and Michal, for replying. Here is the full command: >>>>>> >>>>>> ./gradlew >>>>>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>>>>> -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing >>>>>> -PgcpProject=apache-beam-testing >>>>>> -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>>>> -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>>>> >>>>>> >>>>>> Source: Confluence Java Tips >>>>>> <https://cwiki.apache.org/confluence/display/BEAM/Java+Tips> >>>>>> >>>>>> >>>>>> >>>>>> *Thanks & Regards* >>>>>> >>>>>> >>>>>> *Rehman Murad Ali* >>>>>> >>>>>> Software Engineer >>>>>> Mobile: +92 3452076766 >>>>>> Skype: rehman,muradali >>>>>> >>>>>> <http://venturedive.com/> >>>>>> >>>>>> >>>>>> On Thu, Nov 28, 2019 at 10:42 AM Zohaib Baig < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> +Rehman Murad Ali <[email protected]> >>>>>>> >>>>>>> On Thu, Nov 28, 2019 at 2:58 AM Brian Hulette <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> It looks like you passed an argument like >>>>>>>> -DbeamTestPipelineOptions >>>>>>>> "[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]", >>>>>>>> but the string inside the quotes needs to be a valid JSON array of >>>>>>>> strings. >>>>>>>> If you change it to something like -DbeamTestPipelineOptions >>>>>>>> '["--project=apache-beam-testing",...]' you should get past that error. >>>>>>>> >>>>>>>> Agree with Michał though that we could help best if you share your >>>>>>>> full command line. >>>>>>>> >>>>>>>> Brian >>>>>>>> >>>>>>>> On Wed, Nov 27, 2019 at 8:20 AM Michał Walenia < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> can you please post the command you used in the terminal? It seems >>>>>>>>> you used a wrong combination of quotes, but I'd need to see it to be >>>>>>>>> sure. >>>>>>>>> Cheers, >>>>>>>>> Michal >>>>>>>>> >>>>>>>>> On Wed, Nov 27, 2019 at 5:11 PM Rehman Murad Ali < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Community, >>>>>>>>>> >>>>>>>>>> I have been recently trying to test data flow jobs with the beam. >>>>>>>>>> I have set up Gcloud account and tried to copy a file from the local >>>>>>>>>> system >>>>>>>>>> to the Gcloud storage (which works fine). >>>>>>>>>> >>>>>>>>>> Now I am trying to run preCommitLegacyWorker task on local and I >>>>>>>>>> am getting the following error: >>>>>>>>>> >>>>>>>>>> Unable to instantiate test options from system property >>>>>>>>>> beamTestPipelineOptions >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Caused by: com.fasterxml.jackson.core.JsonParseException: >>>>>>>>>> Unexpected character ('-' (code 45)) in numeric value: expected >>>>>>>>>> digit (0-9) >>>>>>>>>> to follow minus sign, for valid numeric value >>>>>>>>>> at [Source: >>>>>>>>>> (String)"[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]"; >>>>>>>>>> line: 1, column: 4] >>>>>>>>>> at >>>>>>>>>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) >>>>>>>>>> at >>>>>>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693) >>>>>>>>>> at >>>>>>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase.reportUnexpectedNumberChar(ParserMinimalBase.java:541) >>>>>>>>>> at >>>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleInvalidNumberStart(ReaderBasedJsonParser.java:1637) >>>>>>>>>> at >>>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseNegNumber(ReaderBasedJsonParser.java:1391) >>>>>>>>>> at >>>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:742) >>>>>>>>>> at >>>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextTextValue(ReaderBasedJsonParser.java:1160) >>>>>>>>>> at >>>>>>>>>> com.fasterxml.jackson.databind.deser.std.StringArrayDeserializer.deserialize(StringArrayDeserializer.java:145) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Any help would be appreciable. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Thanks & Regards* >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Rehman Murad Ali* >>>>>>>>>> >>>>>>>>>> Software Engineer >>>>>>>>>> Mobile: +92 3452076766 <+92%20345%202076766> >>>>>>>>>> Skype: rehman,muradali >>>>>>>>>> >>>>>>>>>> <http://venturedive.com/> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Michał Walenia >>>>>>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>>>>>> >>>>>>>>> M: +48 791 432 002 <+48791432002> >>>>>>>>> E: [email protected] >>>>>>>>> >>>>>>>>> Unique Tech >>>>>>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> *Muhammad Zohaib Baig* >>>>>>> Senior Software Engineer >>>>>>> Mobile: +92 3443060266 >>>>>>> Skype: mzobii.baig >>>>>>> >>>>>>> <http://venturedive.com/> >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> Michał Walenia >>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>> >>>>> M: +48 791 432 002 <+48791432002> >>>>> E: [email protected] >>>>> >>>>> Unique Tech >>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>> >>>> >>> >>> -- >>> >>> Michał Walenia >>> Polidea <https://www.polidea.com/> | Software Engineer >>> >>> M: +48 791 432 002 <+48791432002> >>> E: [email protected] >>> >>> Unique Tech >>> Check out our projects! <https://www.polidea.com/our-work> >>> >> > > -- > > Michał Walenia > Polidea <https://www.polidea.com/> | Software Engineer > > M: +48 791 432 002 <+48791432002> > E: [email protected] > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work> >
