What kind of a shell are you using? Screenshot suggests it's not CMD, but a bash shell on Windows(?) BTW, do you have permissions on the apache-beam-testing project that allow you to start dataflow jobs directly? If you don't, change the project to your org project, get the permissions and try again :)
On Thu, Nov 28, 2019 at 11:19 AM Rehman Murad Ali < [email protected]> wrote: > I am using Windows CLI for this command. Moreover, I have tried setting up > docker and run this command which results in "Build Successful" but I > cannot find any jobs running on the cloud console. Here are the output > logs. Is there another way to run Dataflow test case on Windows? > > root@cfbdbfdf72ea:/beam# ./gradlew > :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker > -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing > -PgcpProject=apache-beam-testing > -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java > -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java > > Task :buildSrc:compileJava NO-SOURCE > > Task :buildSrc:compileGroovy UP-TO-DATE > > Task :buildSrc:pluginDescriptors UP-TO-DATE > > Task :buildSrc:processResources UP-TO-DATE > > Task :buildSrc:classes UP-TO-DATE > > Task :buildSrc:jar UP-TO-DATE > > Task :buildSrc:assemble UP-TO-DATE > > Task :buildSrc:spotlessGroovy UP-TO-DATE > > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE > > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > > Task :buildSrc:spotlessCheck UP-TO-DATE > > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE > > Task :buildSrc:compileTestJava NO-SOURCE > > Task :buildSrc:compileTestGroovy NO-SOURCE > > Task :buildSrc:processTestResources NO-SOURCE > > Task :buildSrc:testClasses UP-TO-DATE > > Task :buildSrc:test NO-SOURCE > > Task :buildSrc:validateTaskProperties UP-TO-DATE > > Task :buildSrc:check UP-TO-DATE > > Task :buildSrc:build UP-TO-DATE > Configuration on demand is an incubating feature. > > Task :model:job-management:extractProto UP-TO-DATE > > Task :model:job-management:processResources UP-TO-DATE > > Task :sdks:java:core:generateAvroProtocol NO-SOURCE > > Task :sdks:java:core:generateAvroJava NO-SOURCE > > Task :sdks:java:core:generateGrammarSource UP-TO-DATE > > Task :sdks:java:core:processResources UP-TO-DATE > > Task :sdks:java:extensions:google-cloud-platform-core:processResources > NO-SOURCE > > Task :sdks:java:extensions:protobuf:extractProto UP-TO-DATE > > Task :sdks:java:extensions:protobuf:processResources NO-SOURCE > > Task :model:pipeline:extractIncludeProto UP-TO-DATE > > Task :model:pipeline:extractProto UP-TO-DATE > > Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE > > Task :examples:java:processResources NO-SOURCE > > Task :examples:java:processTestResources UP-TO-DATE > > Task :runners:core-construction-java:processResources NO-SOURCE > > Task :model:fn-execution:extractProto UP-TO-DATE > > Task :model:pipeline:generateProto UP-TO-DATE > > Task :model:fn-execution:processResources UP-TO-DATE > > Task :sdks:java:fn-execution:processResources NO-SOURCE > > Task :runners:core-java:processResources NO-SOURCE > > Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE > > Task :sdks:java:harness:processResources NO-SOURCE > > Task :runners:java-fn-execution:processResources NO-SOURCE > > Task :runners:local-java:processResources NO-SOURCE > > Task :runners:direct-java:processResources NO-SOURCE > > Task :runners:google-cloud-dataflow-java:processResources UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:examples:compileJava NO-SOURCE > > Task :runners:google-cloud-dataflow-java:examples:processResources > NO-SOURCE > > Task :runners:google-cloud-dataflow-java:examples:classes UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:examples:compileTestJava > NO-SOURCE > > Task :model:pipeline:compileJava UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:examples:processTestResources > NO-SOURCE > > Task :runners:google-cloud-dataflow-java:examples:testClasses UP-TO-DATE > > Task :model:pipeline:processResources UP-TO-DATE > > Task :model:pipeline:classes UP-TO-DATE > > Task :model:pipeline:jar UP-TO-DATE > > Task > :runners:google-cloud-dataflow-java:worker:windmill:extractIncludeProto > UP-TO-DATE > > Task :model:pipeline:shadowJar UP-TO-DATE > > Task :model:job-management:extractIncludeProto UP-TO-DATE > > Task :model:job-management:generateProto UP-TO-DATE > > Task :model:fn-execution:extractIncludeProto UP-TO-DATE > > Task :model:fn-execution:generateProto UP-TO-DATE > > Task :model:job-management:compileJava UP-TO-DATE > > Task :model:job-management:classes UP-TO-DATE > > Task :model:job-management:shadowJar UP-TO-DATE > > Task :model:fn-execution:compileJava UP-TO-DATE > > Task :model:fn-execution:classes UP-TO-DATE > > Task :model:fn-execution:shadowJar UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:worker:windmill:extractProto > UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:worker:windmill:generateProto > UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:worker:windmill:compileJava > UP-TO-DATE > > Task > :runners:google-cloud-dataflow-java:worker:windmill:processResources > UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:worker:windmill:classes > UP-TO-DATE > > Task :sdks:java:core:compileJava UP-TO-DATE > > Task :sdks:java:core:classes UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:worker:windmill:shadowJar > UP-TO-DATE > > Task > :runners:google-cloud-dataflow-java:worker:legacy-worker:processResources > NO-SOURCE > > Task :sdks:java:core:shadowJar UP-TO-DATE > > Task :sdks:java:extensions:google-cloud-platform-core:compileJava > UP-TO-DATE > > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > > Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE > > Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE > > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE > > Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE > > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE > > Task :sdks:java:extensions:protobuf:jar UP-TO-DATE > > Task :runners:core-construction-java:compileJava UP-TO-DATE > > Task :runners:core-construction-java:classes UP-TO-DATE > > Task :runners:core-construction-java:jar UP-TO-DATE > > Task :sdks:java:fn-execution:compileJava UP-TO-DATE > > Task :sdks:java:fn-execution:classes UP-TO-DATE > > Task :sdks:java:fn-execution:jar UP-TO-DATE > > Task :runners:core-java:compileJava UP-TO-DATE > > Task :runners:core-java:classes UP-TO-DATE > > Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE > > Task :runners:core-java:jar UP-TO-DATE > > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > > Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE > > Task :vendor:sdks-java-extensions-protobuf:compileJava UP-TO-DATE > > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE > > Task :vendor:sdks-java-extensions-protobuf:shadowJar UP-TO-DATE > > Task :sdks:java:harness:compileJava UP-TO-DATE > > Task :sdks:java:harness:classes UP-TO-DATE > > Task :sdks:java:harness:jar UP-TO-DATE > > Task :examples:java:compileJava UP-TO-DATE > > Task :examples:java:classes UP-TO-DATE > > Task :examples:java:jar UP-TO-DATE > > Task :sdks:java:harness:shadowJar UP-TO-DATE > > Task :examples:java:compileTestJava UP-TO-DATE > > Task :examples:java:testClasses UP-TO-DATE > > Task :examples:java:testJar UP-TO-DATE > > Task :runners:local-java:compileJava UP-TO-DATE > > Task :runners:local-java:classes UP-TO-DATE > > Task :runners:java-fn-execution:compileJava UP-TO-DATE > > Task :runners:java-fn-execution:classes UP-TO-DATE > > Task :runners:java-fn-execution:jar UP-TO-DATE > > Task :runners:local-java:jar UP-TO-DATE > > Task :runners:direct-java:compileJava UP-TO-DATE > > Task :runners:direct-java:classes UP-TO-DATE > > Task :runners:direct-java:shadowJar UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:compileJava UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:classes UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:jar UP-TO-DATE > > Task > :runners:google-cloud-dataflow-java:worker:legacy-worker:compileJava > UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:classes > UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar > UP-TO-DATE > > Task :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker > FROM-CACHE > > Deprecated Gradle features were used in this build, making it incompatible > with Gradle 6.0. > Use '--warning-mode all' to show the individual deprecation warnings. > See > https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings > > BUILD SUCCESSFUL in 2s > 65 actionable tasks: 1 from cache, 64 up-to-date > root@cfbdbfdf72ea:/beam# > > > > *Thanks & Regards* > > > *Rehman Murad Ali* > > Software Engineer > Mobile: +92 3452076766 > Skype: rehman,muradali > > <http://venturedive.com/> > > > On Thu, Nov 28, 2019 at 12:37 PM Michał Walenia < > [email protected]> wrote: > >> That's interesting, the command works for me (it crashes on communication >> with GCP, but that was expected). From the log output it seems you're using >> Windows. >> Did you use WSL to run this? Do you have an option to use Linux to check >> this command? The JSON created in the Gradle script may be treated >> differently in CMD (due to quoting differences between it and Bash/ZSH/ >> other shells). >> >> I'd try to run this somewhere else other than CMD. >> Good luck! >> >> @Brian: -DbeamPipelineOptions parameter is constructed by Gradle from an >> object via JsonOutput, so it should be a valid object after reading back. >> >> On Thu, Nov 28, 2019 at 7:39 AM Rehman Murad Ali < >> [email protected]> wrote: >> >>> Thank you, Brian and Michal, for replying. Here is the full command: >>> >>> ./gradlew >>> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >>> -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing >>> -PgcpProject=apache-beam-testing >>> -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>> -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >>> >>> >>> Source: Confluence Java Tips >>> <https://cwiki.apache.org/confluence/display/BEAM/Java+Tips> >>> >>> >>> >>> *Thanks & Regards* >>> >>> >>> *Rehman Murad Ali* >>> >>> Software Engineer >>> Mobile: +92 3452076766 >>> Skype: rehman,muradali >>> >>> <http://venturedive.com/> >>> >>> >>> On Thu, Nov 28, 2019 at 10:42 AM Zohaib Baig < >>> [email protected]> wrote: >>> >>>> +Rehman Murad Ali <[email protected]> >>>> >>>> On Thu, Nov 28, 2019 at 2:58 AM Brian Hulette <[email protected]> >>>> wrote: >>>> >>>>> It looks like you passed an argument like >>>>> -DbeamTestPipelineOptions >>>>> "[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]", >>>>> but the string inside the quotes needs to be a valid JSON array of >>>>> strings. >>>>> If you change it to something like -DbeamTestPipelineOptions >>>>> '["--project=apache-beam-testing",...]' you should get past that error. >>>>> >>>>> Agree with Michał though that we could help best if you share your >>>>> full command line. >>>>> >>>>> Brian >>>>> >>>>> On Wed, Nov 27, 2019 at 8:20 AM Michał Walenia < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> can you please post the command you used in the terminal? It seems >>>>>> you used a wrong combination of quotes, but I'd need to see it to be >>>>>> sure. >>>>>> Cheers, >>>>>> Michal >>>>>> >>>>>> On Wed, Nov 27, 2019 at 5:11 PM Rehman Murad Ali < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Community, >>>>>>> >>>>>>> I have been recently trying to test data flow jobs with the beam. I >>>>>>> have set up Gcloud account and tried to copy a file from the local >>>>>>> system >>>>>>> to the Gcloud storage (which works fine). >>>>>>> >>>>>>> Now I am trying to run preCommitLegacyWorker task on local and I am >>>>>>> getting the following error: >>>>>>> >>>>>>> Unable to instantiate test options from system property >>>>>>> beamTestPipelineOptions >>>>>>> >>>>>>> >>>>>>> Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected >>>>>>> character ('-' (code 45)) in numeric value: expected digit (0-9) to >>>>>>> follow >>>>>>> minus sign, for valid numeric value >>>>>>> at [Source: >>>>>>> (String)"[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]"; >>>>>>> line: 1, column: 4] >>>>>>> at >>>>>>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) >>>>>>> at >>>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693) >>>>>>> at >>>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase.reportUnexpectedNumberChar(ParserMinimalBase.java:541) >>>>>>> at >>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleInvalidNumberStart(ReaderBasedJsonParser.java:1637) >>>>>>> at >>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseNegNumber(ReaderBasedJsonParser.java:1391) >>>>>>> at >>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:742) >>>>>>> at >>>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextTextValue(ReaderBasedJsonParser.java:1160) >>>>>>> at >>>>>>> com.fasterxml.jackson.databind.deser.std.StringArrayDeserializer.deserialize(StringArrayDeserializer.java:145) >>>>>>> >>>>>>> >>>>>>> >>>>>>> Any help would be appreciable. >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Thanks & Regards* >>>>>>> >>>>>>> >>>>>>> *Rehman Murad Ali* >>>>>>> >>>>>>> Software Engineer >>>>>>> Mobile: +92 3452076766 <+92%20345%202076766> >>>>>>> Skype: rehman,muradali >>>>>>> >>>>>>> <http://venturedive.com/> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Michał Walenia >>>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>>> >>>>>> M: +48 791 432 002 <+48791432002> >>>>>> E: [email protected] >>>>>> >>>>>> Unique Tech >>>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>>> >>>>> >>>> >>>> -- >>>> >>>> *Muhammad Zohaib Baig* >>>> Senior Software Engineer >>>> Mobile: +92 3443060266 >>>> Skype: mzobii.baig >>>> >>>> <http://venturedive.com/> >>>> >>> >> >> -- >> >> Michał Walenia >> Polidea <https://www.polidea.com/> | Software Engineer >> >> M: +48 791 432 002 <+48791432002> >> E: [email protected] >> >> Unique Tech >> Check out our projects! <https://www.polidea.com/our-work> >> > -- Michał Walenia Polidea <https://www.polidea.com/> | Software Engineer M: +48 791 432 002 <+48791432002> E: [email protected] Unique Tech Check out our projects! <https://www.polidea.com/our-work>
