I'm glad I was able to help :) And you might consider using WSL, it will allow you to run commands in Bash without starting a Docker container or a VM :) Good luck! Michal
On Thu, Nov 28, 2019 at 2:46 PM Rehman Murad Ali < [email protected]> wrote: > After running tasks with --info and --rerun-tasks, I got to know that > application default credentials were not set in Docker. I have set > GOOGLE_APPLICATION_CREDENTIALS and now it is working fine. I can find the > job on the dataflow cloud. Thank you for your guidance. > > > *Thanks & Regards* > > > *Rehman Murad Ali* > > Software Engineer > Mobile: +92 3452076766 > Skype: rehman,muradali > > <http://venturedive.com/> > > > On Thu, Nov 28, 2019 at 5:29 PM Michał Walenia <[email protected]> > wrote: > >> It seems that Windows CMD doesn't play nice with JSON. I'm not sure >> what's the problem in your Docker though, try running gradle with --info >> and --rerun-tasks. You'll see more output this way >> >> On Thu, Nov 28, 2019 at 1:02 PM Rehman Murad Ali < >> [email protected]> wrote: >> >>> Above mentioned screenshot and output logs are run on docker using bash >>> shell on Windows. Yes, I have the permission and I was able to upload a >>> file on cloud storage as well. >>> >>> On the other hand, I am getting JsonParsingException when running on >>> Window's CMD. See the attached screenshots. >>> >>> >>> >>> >>> *Thanks & Regards* >>> >>> >>> *Rehman Murad Ali* >>> >>> Software Engineer >>> Mobile: +92 3452076766 >>> Skype: rehman,muradali >>> >>> <http://venturedive.com/> >>> >>> >>> On Thu, Nov 28, 2019 at 4:29 PM Michał Walenia < >>> [email protected]> wrote: >>> >>>> What kind of a shell are you using? Screenshot suggests it's not CMD, >>>> but a bash shell on Windows(?) BTW, do you have permissions on the >>>> apache-beam-testing project that allow you to start dataflow jobs directly? >>>> If you don't, change the project to your org project, get the >>>> permissions and try again :) >>>> >>>> On Thu, Nov 28, 2019 at 11:19 AM Rehman Murad Ali < >>>> [email protected]> wrote: >>>> >>>>> I am using Windows CLI for this command. Moreover, I have tried >>>>> setting up docker and run this command which results in "Build Successful" >>>>> but I cannot find any jobs running on the cloud console. Here are the >>>>> output logs. Is there another way to run Dataflow test case on Windows? >>>>> >>>>> root@cfbdbfdf72ea:/beam# ./gradlew >>>>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>>>> -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing >>>>> -PgcpProject=apache-beam-testing >>>>> -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>>> -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>>> > Task :buildSrc:compileJava NO-SOURCE >>>>> > Task :buildSrc:compileGroovy UP-TO-DATE >>>>> > Task :buildSrc:pluginDescriptors UP-TO-DATE >>>>> > Task :buildSrc:processResources UP-TO-DATE >>>>> > Task :buildSrc:classes UP-TO-DATE >>>>> > Task :buildSrc:jar UP-TO-DATE >>>>> > Task :buildSrc:assemble UP-TO-DATE >>>>> > Task :buildSrc:spotlessGroovy UP-TO-DATE >>>>> > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE >>>>> > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE >>>>> > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE >>>>> > Task :buildSrc:spotlessCheck UP-TO-DATE >>>>> > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE >>>>> > Task :buildSrc:compileTestJava NO-SOURCE >>>>> > Task :buildSrc:compileTestGroovy NO-SOURCE >>>>> > Task :buildSrc:processTestResources NO-SOURCE >>>>> > Task :buildSrc:testClasses UP-TO-DATE >>>>> > Task :buildSrc:test NO-SOURCE >>>>> > Task :buildSrc:validateTaskProperties UP-TO-DATE >>>>> > Task :buildSrc:check UP-TO-DATE >>>>> > Task :buildSrc:build UP-TO-DATE >>>>> Configuration on demand is an incubating feature. >>>>> > Task :model:job-management:extractProto UP-TO-DATE >>>>> > Task :model:job-management:processResources UP-TO-DATE >>>>> > Task :sdks:java:core:generateAvroProtocol NO-SOURCE >>>>> > Task :sdks:java:core:generateAvroJava NO-SOURCE >>>>> > Task :sdks:java:core:generateGrammarSource UP-TO-DATE >>>>> > Task :sdks:java:core:processResources UP-TO-DATE >>>>> > Task >>>>> :sdks:java:extensions:google-cloud-platform-core:processResources >>>>> NO-SOURCE >>>>> > Task :sdks:java:extensions:protobuf:extractProto UP-TO-DATE >>>>> > Task :sdks:java:extensions:protobuf:processResources NO-SOURCE >>>>> > Task :model:pipeline:extractIncludeProto UP-TO-DATE >>>>> > Task :model:pipeline:extractProto UP-TO-DATE >>>>> > Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE >>>>> > Task :examples:java:processResources NO-SOURCE >>>>> > Task :examples:java:processTestResources UP-TO-DATE >>>>> > Task :runners:core-construction-java:processResources NO-SOURCE >>>>> > Task :model:fn-execution:extractProto UP-TO-DATE >>>>> > Task :model:pipeline:generateProto UP-TO-DATE >>>>> > Task :model:fn-execution:processResources UP-TO-DATE >>>>> > Task :sdks:java:fn-execution:processResources NO-SOURCE >>>>> > Task :runners:core-java:processResources NO-SOURCE >>>>> > Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE >>>>> > Task :sdks:java:harness:processResources NO-SOURCE >>>>> > Task :runners:java-fn-execution:processResources NO-SOURCE >>>>> > Task :runners:local-java:processResources NO-SOURCE >>>>> > Task :runners:direct-java:processResources NO-SOURCE >>>>> > Task :runners:google-cloud-dataflow-java:processResources UP-TO-DATE >>>>> > Task :runners:google-cloud-dataflow-java:examples:compileJava >>>>> NO-SOURCE >>>>> > Task :runners:google-cloud-dataflow-java:examples:processResources >>>>> NO-SOURCE >>>>> > Task :runners:google-cloud-dataflow-java:examples:classes UP-TO-DATE >>>>> > Task :runners:google-cloud-dataflow-java:examples:compileTestJava >>>>> NO-SOURCE >>>>> > Task :model:pipeline:compileJava UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:examples:processTestResources >>>>> NO-SOURCE >>>>> > Task :runners:google-cloud-dataflow-java:examples:testClasses >>>>> UP-TO-DATE >>>>> > Task :model:pipeline:processResources UP-TO-DATE >>>>> > Task :model:pipeline:classes UP-TO-DATE >>>>> > Task :model:pipeline:jar UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:worker:windmill:extractIncludeProto >>>>> UP-TO-DATE >>>>> > Task :model:pipeline:shadowJar UP-TO-DATE >>>>> > Task :model:job-management:extractIncludeProto UP-TO-DATE >>>>> > Task :model:job-management:generateProto UP-TO-DATE >>>>> > Task :model:fn-execution:extractIncludeProto UP-TO-DATE >>>>> > Task :model:fn-execution:generateProto UP-TO-DATE >>>>> > Task :model:job-management:compileJava UP-TO-DATE >>>>> > Task :model:job-management:classes UP-TO-DATE >>>>> > Task :model:job-management:shadowJar UP-TO-DATE >>>>> > Task :model:fn-execution:compileJava UP-TO-DATE >>>>> > Task :model:fn-execution:classes UP-TO-DATE >>>>> > Task :model:fn-execution:shadowJar UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:worker:windmill:extractProto >>>>> UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:worker:windmill:generateProto >>>>> UP-TO-DATE >>>>> > Task :runners:google-cloud-dataflow-java:worker:windmill:compileJava >>>>> UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:worker:windmill:processResources >>>>> UP-TO-DATE >>>>> > Task :runners:google-cloud-dataflow-java:worker:windmill:classes >>>>> UP-TO-DATE >>>>> > Task :sdks:java:core:compileJava UP-TO-DATE >>>>> > Task :sdks:java:core:classes UP-TO-DATE >>>>> > Task :runners:google-cloud-dataflow-java:worker:windmill:shadowJar >>>>> UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:worker:legacy-worker:processResources >>>>> NO-SOURCE >>>>> > Task :sdks:java:core:shadowJar UP-TO-DATE >>>>> > Task :sdks:java:extensions:google-cloud-platform-core:compileJava >>>>> UP-TO-DATE >>>>> > Task :sdks:java:extensions:google-cloud-platform-core:classes >>>>> UP-TO-DATE >>>>> > Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE >>>>> > Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE >>>>> > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE >>>>> > Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE >>>>> > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE >>>>> > Task :sdks:java:extensions:protobuf:jar UP-TO-DATE >>>>> > Task :runners:core-construction-java:compileJava UP-TO-DATE >>>>> > Task :runners:core-construction-java:classes UP-TO-DATE >>>>> > Task :runners:core-construction-java:jar UP-TO-DATE >>>>> > Task :sdks:java:fn-execution:compileJava UP-TO-DATE >>>>> > Task :sdks:java:fn-execution:classes UP-TO-DATE >>>>> > Task :sdks:java:fn-execution:jar UP-TO-DATE >>>>> > Task :runners:core-java:compileJava UP-TO-DATE >>>>> > Task :runners:core-java:classes UP-TO-DATE >>>>> > Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE >>>>> > Task :runners:core-java:jar UP-TO-DATE >>>>> > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE >>>>> > Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE >>>>> > Task :vendor:sdks-java-extensions-protobuf:compileJava UP-TO-DATE >>>>> > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE >>>>> > Task :vendor:sdks-java-extensions-protobuf:shadowJar UP-TO-DATE >>>>> > Task :sdks:java:harness:compileJava UP-TO-DATE >>>>> > Task :sdks:java:harness:classes UP-TO-DATE >>>>> > Task :sdks:java:harness:jar UP-TO-DATE >>>>> > Task :examples:java:compileJava UP-TO-DATE >>>>> > Task :examples:java:classes UP-TO-DATE >>>>> > Task :examples:java:jar UP-TO-DATE >>>>> > Task :sdks:java:harness:shadowJar UP-TO-DATE >>>>> > Task :examples:java:compileTestJava UP-TO-DATE >>>>> > Task :examples:java:testClasses UP-TO-DATE >>>>> > Task :examples:java:testJar UP-TO-DATE >>>>> > Task :runners:local-java:compileJava UP-TO-DATE >>>>> > Task :runners:local-java:classes UP-TO-DATE >>>>> > Task :runners:java-fn-execution:compileJava UP-TO-DATE >>>>> > Task :runners:java-fn-execution:classes UP-TO-DATE >>>>> > Task :runners:java-fn-execution:jar UP-TO-DATE >>>>> > Task :runners:local-java:jar UP-TO-DATE >>>>> > Task :runners:direct-java:compileJava UP-TO-DATE >>>>> > Task :runners:direct-java:classes UP-TO-DATE >>>>> > Task :runners:direct-java:shadowJar UP-TO-DATE >>>>> > Task :runners:google-cloud-dataflow-java:compileJava UP-TO-DATE >>>>> > Task :runners:google-cloud-dataflow-java:classes UP-TO-DATE >>>>> > Task :runners:google-cloud-dataflow-java:jar UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:worker:legacy-worker:compileJava >>>>> UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:worker:legacy-worker:classes >>>>> UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar >>>>> UP-TO-DATE >>>>> > Task >>>>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>>>> FROM-CACHE >>>>> >>>>> Deprecated Gradle features were used in this build, making it >>>>> incompatible with Gradle 6.0. >>>>> Use '--warning-mode all' to show the individual deprecation warnings. >>>>> See >>>>> https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings >>>>> >>>>> BUILD SUCCESSFUL in 2s >>>>> 65 actionable tasks: 1 from cache, 64 up-to-date >>>>> root@cfbdbfdf72ea:/beam# >>>>> >>>>> >>>>> >>>>> *Thanks & Regards* >>>>> >>>>> >>>>> *Rehman Murad Ali* >>>>> >>>>> Software Engineer >>>>> Mobile: +92 3452076766 >>>>> Skype: rehman,muradali >>>>> >>>>> <http://venturedive.com/> >>>>> >>>>> >>>>> On Thu, Nov 28, 2019 at 12:37 PM Michał Walenia < >>>>> [email protected]> wrote: >>>>> >>>>>> That's interesting, the command works for me (it crashes on >>>>>> communication with GCP, but that was expected). From the log output it >>>>>> seems you're using Windows. >>>>>> Did you use WSL to run this? Do you have an option to use Linux to >>>>>> check this command? The JSON created in the Gradle script may be treated >>>>>> differently in CMD (due to quoting differences between it and Bash/ZSH/ >>>>>> other shells). >>>>>> >>>>>> I'd try to run this somewhere else other than CMD. >>>>>> Good luck! >>>>>> >>>>>> @Brian: -DbeamPipelineOptions parameter is constructed by Gradle from >>>>>> an object via JsonOutput, so it should be a valid object after reading >>>>>> back. >>>>>> >>>>>> On Thu, Nov 28, 2019 at 7:39 AM Rehman Murad Ali < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Thank you, Brian and Michal, for replying. Here is the full command: >>>>>>> >>>>>>> ./gradlew >>>>>>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>>>>>> -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing >>>>>>> -PgcpProject=apache-beam-testing >>>>>>> -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>>>>> -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>>>>> >>>>>>> >>>>>>> Source: Confluence Java Tips >>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/Java+Tips> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Thanks & Regards* >>>>>>> >>>>>>> >>>>>>> *Rehman Murad Ali* >>>>>>> >>>>>>> Software Engineer >>>>>>> Mobile: +92 3452076766 >>>>>>> Skype: rehman,muradali >>>>>>> >>>>>>> <http://venturedive.com/> >>>>>>> >>>>>>> >>>>>>> On Thu, Nov 28, 2019 at 10:42 AM Zohaib Baig < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> +Rehman Murad Ali <[email protected]> >>>>>>>> >>>>>>>> On Thu, Nov 28, 2019 at 2:58 AM Brian Hulette <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> It looks like you passed an argument like >>>>>>>>> -DbeamTestPipelineOptions >>>>>>>>> "[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]", >>>>>>>>> but the string inside the quotes needs to be a valid JSON array of >>>>>>>>> strings. >>>>>>>>> If you change it to something like -DbeamTestPipelineOptions >>>>>>>>> '["--project=apache-beam-testing",...]' you should get past that >>>>>>>>> error. >>>>>>>>> >>>>>>>>> Agree with Michał though that we could help best if you share your >>>>>>>>> full command line. >>>>>>>>> >>>>>>>>> Brian >>>>>>>>> >>>>>>>>> On Wed, Nov 27, 2019 at 8:20 AM Michał Walenia < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> can you please post the command you used in the terminal? It >>>>>>>>>> seems you used a wrong combination of quotes, but I'd need to see it >>>>>>>>>> to be >>>>>>>>>> sure. >>>>>>>>>> Cheers, >>>>>>>>>> Michal >>>>>>>>>> >>>>>>>>>> On Wed, Nov 27, 2019 at 5:11 PM Rehman Murad Ali < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Community, >>>>>>>>>>> >>>>>>>>>>> I have been recently trying to test data flow jobs with the >>>>>>>>>>> beam. I have set up Gcloud account and tried to copy a file from >>>>>>>>>>> the local >>>>>>>>>>> system to the Gcloud storage (which works fine). >>>>>>>>>>> >>>>>>>>>>> Now I am trying to run preCommitLegacyWorker task on local and I >>>>>>>>>>> am getting the following error: >>>>>>>>>>> >>>>>>>>>>> Unable to instantiate test options from system property >>>>>>>>>>> beamTestPipelineOptions >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Caused by: com.fasterxml.jackson.core.JsonParseException: >>>>>>>>>>> Unexpected character ('-' (code 45)) in numeric value: expected >>>>>>>>>>> digit (0-9) >>>>>>>>>>> to follow minus sign, for valid numeric value >>>>>>>>>>> at [Source: >>>>>>>>>>> (String)"[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]"; >>>>>>>>>>> line: 1, column: 4] >>>>>>>>>>> at >>>>>>>>>>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) >>>>>>>>>>> at >>>>>>>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693) >>>>>>>>>>> at >>>>>>>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase.reportUnexpectedNumberChar(ParserMinimalBase.java:541) >>>>>>>>>>> at >>>>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleInvalidNumberStart(ReaderBasedJsonParser.java:1637) >>>>>>>>>>> at >>>>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseNegNumber(ReaderBasedJsonParser.java:1391) >>>>>>>>>>> at >>>>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:742) >>>>>>>>>>> at >>>>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextTextValue(ReaderBasedJsonParser.java:1160) >>>>>>>>>>> at >>>>>>>>>>> com.fasterxml.jackson.databind.deser.std.StringArrayDeserializer.deserialize(StringArrayDeserializer.java:145) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Any help would be appreciable. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Thanks & Regards* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Rehman Murad Ali* >>>>>>>>>>> >>>>>>>>>>> Software Engineer >>>>>>>>>>> Mobile: +92 3452076766 <+92%20345%202076766> >>>>>>>>>>> Skype: rehman,muradali >>>>>>>>>>> >>>>>>>>>>> <http://venturedive.com/> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Michał Walenia >>>>>>>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>>>>>>> >>>>>>>>>> M: +48 791 432 002 <+48791432002> >>>>>>>>>> E: [email protected] >>>>>>>>>> >>>>>>>>>> Unique Tech >>>>>>>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> *Muhammad Zohaib Baig* >>>>>>>> Senior Software Engineer >>>>>>>> Mobile: +92 3443060266 >>>>>>>> Skype: mzobii.baig >>>>>>>> >>>>>>>> <http://venturedive.com/> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Michał Walenia >>>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>>> >>>>>> M: +48 791 432 002 <+48791432002> >>>>>> E: [email protected] >>>>>> >>>>>> Unique Tech >>>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>>> >>>>> >>>> >>>> -- >>>> >>>> Michał Walenia >>>> Polidea <https://www.polidea.com/> | Software Engineer >>>> >>>> M: +48 791 432 002 <+48791432002> >>>> E: [email protected] >>>> >>>> Unique Tech >>>> Check out our projects! <https://www.polidea.com/our-work> >>>> >>> >> >> -- >> >> Michał Walenia >> Polidea <https://www.polidea.com/> | Software Engineer >> >> M: +48 791 432 002 <+48791432002> >> E: [email protected] >> >> Unique Tech >> Check out our projects! <https://www.polidea.com/our-work> >> > -- Michał Walenia Polidea <https://www.polidea.com/> | Software Engineer M: +48 791 432 002 <+48791432002> E: [email protected] Unique Tech Check out our projects! <https://www.polidea.com/our-work>
