It seems that Windows CMD doesn't play nice with JSON. I'm not sure what's the problem in your Docker though, try running gradle with --info and --rerun-tasks. You'll see more output this way
On Thu, Nov 28, 2019 at 1:02 PM Rehman Murad Ali < [email protected]> wrote: > Above mentioned screenshot and output logs are run on docker using bash > shell on Windows. Yes, I have the permission and I was able to upload a > file on cloud storage as well. > > On the other hand, I am getting JsonParsingException when running on > Window's CMD. See the attached screenshots. > > > > > *Thanks & Regards* > > > *Rehman Murad Ali* > > Software Engineer > Mobile: +92 3452076766 > Skype: rehman,muradali > > <http://venturedive.com/> > > > On Thu, Nov 28, 2019 at 4:29 PM Michał Walenia <[email protected]> > wrote: > >> What kind of a shell are you using? Screenshot suggests it's not CMD, but >> a bash shell on Windows(?) BTW, do you have permissions on the >> apache-beam-testing project that allow you to start dataflow jobs directly? >> If you don't, change the project to your org project, get the permissions >> and try again :) >> >> On Thu, Nov 28, 2019 at 11:19 AM Rehman Murad Ali < >> [email protected]> wrote: >> >>> I am using Windows CLI for this command. Moreover, I have tried setting >>> up docker and run this command which results in "Build Successful" but I >>> cannot find any jobs running on the cloud console. Here are the output >>> logs. Is there another way to run Dataflow test case on Windows? >>> >>> root@cfbdbfdf72ea:/beam# ./gradlew >>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>> -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing >>> -PgcpProject=apache-beam-testing >>> -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>> -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>> > Task :buildSrc:compileJava NO-SOURCE >>> > Task :buildSrc:compileGroovy UP-TO-DATE >>> > Task :buildSrc:pluginDescriptors UP-TO-DATE >>> > Task :buildSrc:processResources UP-TO-DATE >>> > Task :buildSrc:classes UP-TO-DATE >>> > Task :buildSrc:jar UP-TO-DATE >>> > Task :buildSrc:assemble UP-TO-DATE >>> > Task :buildSrc:spotlessGroovy UP-TO-DATE >>> > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE >>> > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE >>> > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE >>> > Task :buildSrc:spotlessCheck UP-TO-DATE >>> > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE >>> > Task :buildSrc:compileTestJava NO-SOURCE >>> > Task :buildSrc:compileTestGroovy NO-SOURCE >>> > Task :buildSrc:processTestResources NO-SOURCE >>> > Task :buildSrc:testClasses UP-TO-DATE >>> > Task :buildSrc:test NO-SOURCE >>> > Task :buildSrc:validateTaskProperties UP-TO-DATE >>> > Task :buildSrc:check UP-TO-DATE >>> > Task :buildSrc:build UP-TO-DATE >>> Configuration on demand is an incubating feature. >>> > Task :model:job-management:extractProto UP-TO-DATE >>> > Task :model:job-management:processResources UP-TO-DATE >>> > Task :sdks:java:core:generateAvroProtocol NO-SOURCE >>> > Task :sdks:java:core:generateAvroJava NO-SOURCE >>> > Task :sdks:java:core:generateGrammarSource UP-TO-DATE >>> > Task :sdks:java:core:processResources UP-TO-DATE >>> > Task :sdks:java:extensions:google-cloud-platform-core:processResources >>> NO-SOURCE >>> > Task :sdks:java:extensions:protobuf:extractProto UP-TO-DATE >>> > Task :sdks:java:extensions:protobuf:processResources NO-SOURCE >>> > Task :model:pipeline:extractIncludeProto UP-TO-DATE >>> > Task :model:pipeline:extractProto UP-TO-DATE >>> > Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE >>> > Task :examples:java:processResources NO-SOURCE >>> > Task :examples:java:processTestResources UP-TO-DATE >>> > Task :runners:core-construction-java:processResources NO-SOURCE >>> > Task :model:fn-execution:extractProto UP-TO-DATE >>> > Task :model:pipeline:generateProto UP-TO-DATE >>> > Task :model:fn-execution:processResources UP-TO-DATE >>> > Task :sdks:java:fn-execution:processResources NO-SOURCE >>> > Task :runners:core-java:processResources NO-SOURCE >>> > Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE >>> > Task :sdks:java:harness:processResources NO-SOURCE >>> > Task :runners:java-fn-execution:processResources NO-SOURCE >>> > Task :runners:local-java:processResources NO-SOURCE >>> > Task :runners:direct-java:processResources NO-SOURCE >>> > Task :runners:google-cloud-dataflow-java:processResources UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:examples:compileJava NO-SOURCE >>> > Task :runners:google-cloud-dataflow-java:examples:processResources >>> NO-SOURCE >>> > Task :runners:google-cloud-dataflow-java:examples:classes UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:examples:compileTestJava >>> NO-SOURCE >>> > Task :model:pipeline:compileJava UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:examples:processTestResources >>> NO-SOURCE >>> > Task :runners:google-cloud-dataflow-java:examples:testClasses >>> UP-TO-DATE >>> > Task :model:pipeline:processResources UP-TO-DATE >>> > Task :model:pipeline:classes UP-TO-DATE >>> > Task :model:pipeline:jar UP-TO-DATE >>> > Task >>> :runners:google-cloud-dataflow-java:worker:windmill:extractIncludeProto >>> UP-TO-DATE >>> > Task :model:pipeline:shadowJar UP-TO-DATE >>> > Task :model:job-management:extractIncludeProto UP-TO-DATE >>> > Task :model:job-management:generateProto UP-TO-DATE >>> > Task :model:fn-execution:extractIncludeProto UP-TO-DATE >>> > Task :model:fn-execution:generateProto UP-TO-DATE >>> > Task :model:job-management:compileJava UP-TO-DATE >>> > Task :model:job-management:classes UP-TO-DATE >>> > Task :model:job-management:shadowJar UP-TO-DATE >>> > Task :model:fn-execution:compileJava UP-TO-DATE >>> > Task :model:fn-execution:classes UP-TO-DATE >>> > Task :model:fn-execution:shadowJar UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:worker:windmill:extractProto >>> UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:worker:windmill:generateProto >>> UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:worker:windmill:compileJava >>> UP-TO-DATE >>> > Task >>> :runners:google-cloud-dataflow-java:worker:windmill:processResources >>> UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:worker:windmill:classes >>> UP-TO-DATE >>> > Task :sdks:java:core:compileJava UP-TO-DATE >>> > Task :sdks:java:core:classes UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:worker:windmill:shadowJar >>> UP-TO-DATE >>> > Task >>> :runners:google-cloud-dataflow-java:worker:legacy-worker:processResources >>> NO-SOURCE >>> > Task :sdks:java:core:shadowJar UP-TO-DATE >>> > Task :sdks:java:extensions:google-cloud-platform-core:compileJava >>> UP-TO-DATE >>> > Task :sdks:java:extensions:google-cloud-platform-core:classes >>> UP-TO-DATE >>> > Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE >>> > Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE >>> > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE >>> > Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE >>> > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE >>> > Task :sdks:java:extensions:protobuf:jar UP-TO-DATE >>> > Task :runners:core-construction-java:compileJava UP-TO-DATE >>> > Task :runners:core-construction-java:classes UP-TO-DATE >>> > Task :runners:core-construction-java:jar UP-TO-DATE >>> > Task :sdks:java:fn-execution:compileJava UP-TO-DATE >>> > Task :sdks:java:fn-execution:classes UP-TO-DATE >>> > Task :sdks:java:fn-execution:jar UP-TO-DATE >>> > Task :runners:core-java:compileJava UP-TO-DATE >>> > Task :runners:core-java:classes UP-TO-DATE >>> > Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE >>> > Task :runners:core-java:jar UP-TO-DATE >>> > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE >>> > Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE >>> > Task :vendor:sdks-java-extensions-protobuf:compileJava UP-TO-DATE >>> > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE >>> > Task :vendor:sdks-java-extensions-protobuf:shadowJar UP-TO-DATE >>> > Task :sdks:java:harness:compileJava UP-TO-DATE >>> > Task :sdks:java:harness:classes UP-TO-DATE >>> > Task :sdks:java:harness:jar UP-TO-DATE >>> > Task :examples:java:compileJava UP-TO-DATE >>> > Task :examples:java:classes UP-TO-DATE >>> > Task :examples:java:jar UP-TO-DATE >>> > Task :sdks:java:harness:shadowJar UP-TO-DATE >>> > Task :examples:java:compileTestJava UP-TO-DATE >>> > Task :examples:java:testClasses UP-TO-DATE >>> > Task :examples:java:testJar UP-TO-DATE >>> > Task :runners:local-java:compileJava UP-TO-DATE >>> > Task :runners:local-java:classes UP-TO-DATE >>> > Task :runners:java-fn-execution:compileJava UP-TO-DATE >>> > Task :runners:java-fn-execution:classes UP-TO-DATE >>> > Task :runners:java-fn-execution:jar UP-TO-DATE >>> > Task :runners:local-java:jar UP-TO-DATE >>> > Task :runners:direct-java:compileJava UP-TO-DATE >>> > Task :runners:direct-java:classes UP-TO-DATE >>> > Task :runners:direct-java:shadowJar UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:compileJava UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:classes UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:jar UP-TO-DATE >>> > Task >>> :runners:google-cloud-dataflow-java:worker:legacy-worker:compileJava >>> UP-TO-DATE >>> > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:classes >>> UP-TO-DATE >>> > Task >>> :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar >>> UP-TO-DATE >>> > Task >>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>> FROM-CACHE >>> >>> Deprecated Gradle features were used in this build, making it >>> incompatible with Gradle 6.0. >>> Use '--warning-mode all' to show the individual deprecation warnings. >>> See >>> https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings >>> >>> BUILD SUCCESSFUL in 2s >>> 65 actionable tasks: 1 from cache, 64 up-to-date >>> root@cfbdbfdf72ea:/beam# >>> >>> >>> >>> *Thanks & Regards* >>> >>> >>> *Rehman Murad Ali* >>> >>> Software Engineer >>> Mobile: +92 3452076766 >>> Skype: rehman,muradali >>> >>> <http://venturedive.com/> >>> >>> >>> On Thu, Nov 28, 2019 at 12:37 PM Michał Walenia < >>> [email protected]> wrote: >>> >>>> That's interesting, the command works for me (it crashes on >>>> communication with GCP, but that was expected). From the log output it >>>> seems you're using Windows. >>>> Did you use WSL to run this? Do you have an option to use Linux to >>>> check this command? The JSON created in the Gradle script may be treated >>>> differently in CMD (due to quoting differences between it and Bash/ZSH/ >>>> other shells). >>>> >>>> I'd try to run this somewhere else other than CMD. >>>> Good luck! >>>> >>>> @Brian: -DbeamPipelineOptions parameter is constructed by Gradle from >>>> an object via JsonOutput, so it should be a valid object after reading >>>> back. >>>> >>>> On Thu, Nov 28, 2019 at 7:39 AM Rehman Murad Ali < >>>> [email protected]> wrote: >>>> >>>>> Thank you, Brian and Michal, for replying. Here is the full command: >>>>> >>>>> ./gradlew >>>>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>>>> -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing >>>>> -PgcpProject=apache-beam-testing >>>>> -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>>> -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>>>> >>>>> >>>>> Source: Confluence Java Tips >>>>> <https://cwiki.apache.org/confluence/display/BEAM/Java+Tips> >>>>> >>>>> >>>>> >>>>> *Thanks & Regards* >>>>> >>>>> >>>>> *Rehman Murad Ali* >>>>> >>>>> Software Engineer >>>>> Mobile: +92 3452076766 >>>>> Skype: rehman,muradali >>>>> >>>>> <http://venturedive.com/> >>>>> >>>>> >>>>> On Thu, Nov 28, 2019 at 10:42 AM Zohaib Baig < >>>>> [email protected]> wrote: >>>>> >>>>>> +Rehman Murad Ali <[email protected]> >>>>>> >>>>>> On Thu, Nov 28, 2019 at 2:58 AM Brian Hulette <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> It looks like you passed an argument like >>>>>>> -DbeamTestPipelineOptions >>>>>>> "[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]", >>>>>>> but the string inside the quotes needs to be a valid JSON array of >>>>>>> strings. >>>>>>> If you change it to something like -DbeamTestPipelineOptions >>>>>>> '["--project=apache-beam-testing",...]' you should get past that error. >>>>>>> >>>>>>> Agree with Michał though that we could help best if you share your >>>>>>> full command line. >>>>>>> >>>>>>> Brian >>>>>>> >>>>>>> On Wed, Nov 27, 2019 at 8:20 AM Michał Walenia < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> can you please post the command you used in the terminal? It seems >>>>>>>> you used a wrong combination of quotes, but I'd need to see it to be >>>>>>>> sure. >>>>>>>> Cheers, >>>>>>>> Michal >>>>>>>> >>>>>>>> On Wed, Nov 27, 2019 at 5:11 PM Rehman Murad Ali < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Hi Community, >>>>>>>>> >>>>>>>>> I have been recently trying to test data flow jobs with the beam. >>>>>>>>> I have set up Gcloud account and tried to copy a file from the local >>>>>>>>> system >>>>>>>>> to the Gcloud storage (which works fine). >>>>>>>>> >>>>>>>>> Now I am trying to run preCommitLegacyWorker task on local and I >>>>>>>>> am getting the following error: >>>>>>>>> >>>>>>>>> Unable to instantiate test options from system property >>>>>>>>> beamTestPipelineOptions >>>>>>>>> >>>>>>>>> >>>>>>>>> Caused by: com.fasterxml.jackson.core.JsonParseException: >>>>>>>>> Unexpected character ('-' (code 45)) in numeric value: expected digit >>>>>>>>> (0-9) >>>>>>>>> to follow minus sign, for valid numeric value >>>>>>>>> at [Source: >>>>>>>>> (String)"[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]"; >>>>>>>>> line: 1, column: 4] >>>>>>>>> at >>>>>>>>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) >>>>>>>>> at >>>>>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693) >>>>>>>>> at >>>>>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase.reportUnexpectedNumberChar(ParserMinimalBase.java:541) >>>>>>>>> at >>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleInvalidNumberStart(ReaderBasedJsonParser.java:1637) >>>>>>>>> at >>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseNegNumber(ReaderBasedJsonParser.java:1391) >>>>>>>>> at >>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:742) >>>>>>>>> at >>>>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextTextValue(ReaderBasedJsonParser.java:1160) >>>>>>>>> at >>>>>>>>> com.fasterxml.jackson.databind.deser.std.StringArrayDeserializer.deserialize(StringArrayDeserializer.java:145) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Any help would be appreciable. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Thanks & Regards* >>>>>>>>> >>>>>>>>> >>>>>>>>> *Rehman Murad Ali* >>>>>>>>> >>>>>>>>> Software Engineer >>>>>>>>> Mobile: +92 3452076766 <+92%20345%202076766> >>>>>>>>> Skype: rehman,muradali >>>>>>>>> >>>>>>>>> <http://venturedive.com/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Michał Walenia >>>>>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>>>>> >>>>>>>> M: +48 791 432 002 <+48791432002> >>>>>>>> E: [email protected] >>>>>>>> >>>>>>>> Unique Tech >>>>>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> *Muhammad Zohaib Baig* >>>>>> Senior Software Engineer >>>>>> Mobile: +92 3443060266 >>>>>> Skype: mzobii.baig >>>>>> >>>>>> <http://venturedive.com/> >>>>>> >>>>> >>>> >>>> -- >>>> >>>> Michał Walenia >>>> Polidea <https://www.polidea.com/> | Software Engineer >>>> >>>> M: +48 791 432 002 <+48791432002> >>>> E: [email protected] >>>> >>>> Unique Tech >>>> Check out our projects! <https://www.polidea.com/our-work> >>>> >>> >> >> -- >> >> Michał Walenia >> Polidea <https://www.polidea.com/> | Software Engineer >> >> M: +48 791 432 002 <+48791432002> >> E: [email protected] >> >> Unique Tech >> Check out our projects! <https://www.polidea.com/our-work> >> > -- Michał Walenia Polidea <https://www.polidea.com/> | Software Engineer M: +48 791 432 002 <+48791432002> E: [email protected] Unique Tech Check out our projects! <https://www.polidea.com/our-work>
