I am using Windows CLI for this command. Moreover, I have tried setting up docker and run this command which results in "Build Successful" but I cannot find any jobs running on the cloud console. Here are the output logs. Is there another way to run Dataflow test case on Windows?
root@cfbdbfdf72ea:/beam# ./gradlew :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing -PgcpProject=apache-beam-testing -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java > Task :buildSrc:compileJava NO-SOURCE > Task :buildSrc:compileGroovy UP-TO-DATE > Task :buildSrc:pluginDescriptors UP-TO-DATE > Task :buildSrc:processResources UP-TO-DATE > Task :buildSrc:classes UP-TO-DATE > Task :buildSrc:jar UP-TO-DATE > Task :buildSrc:assemble UP-TO-DATE > Task :buildSrc:spotlessGroovy UP-TO-DATE > Task :buildSrc:spotlessGroovyCheck UP-TO-DATE > Task :buildSrc:spotlessGroovyGradle UP-TO-DATE > Task :buildSrc:spotlessGroovyGradleCheck UP-TO-DATE > Task :buildSrc:spotlessCheck UP-TO-DATE > Task :buildSrc:pluginUnderTestMetadata UP-TO-DATE > Task :buildSrc:compileTestJava NO-SOURCE > Task :buildSrc:compileTestGroovy NO-SOURCE > Task :buildSrc:processTestResources NO-SOURCE > Task :buildSrc:testClasses UP-TO-DATE > Task :buildSrc:test NO-SOURCE > Task :buildSrc:validateTaskProperties UP-TO-DATE > Task :buildSrc:check UP-TO-DATE > Task :buildSrc:build UP-TO-DATE Configuration on demand is an incubating feature. > Task :model:job-management:extractProto UP-TO-DATE > Task :model:job-management:processResources UP-TO-DATE > Task :sdks:java:core:generateAvroProtocol NO-SOURCE > Task :sdks:java:core:generateAvroJava NO-SOURCE > Task :sdks:java:core:generateGrammarSource UP-TO-DATE > Task :sdks:java:core:processResources UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:processResources NO-SOURCE > Task :sdks:java:extensions:protobuf:extractProto UP-TO-DATE > Task :sdks:java:extensions:protobuf:processResources NO-SOURCE > Task :model:pipeline:extractIncludeProto UP-TO-DATE > Task :model:pipeline:extractProto UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:processResources NO-SOURCE > Task :examples:java:processResources NO-SOURCE > Task :examples:java:processTestResources UP-TO-DATE > Task :runners:core-construction-java:processResources NO-SOURCE > Task :model:fn-execution:extractProto UP-TO-DATE > Task :model:pipeline:generateProto UP-TO-DATE > Task :model:fn-execution:processResources UP-TO-DATE > Task :sdks:java:fn-execution:processResources NO-SOURCE > Task :runners:core-java:processResources NO-SOURCE > Task :vendor:sdks-java-extensions-protobuf:processResources NO-SOURCE > Task :sdks:java:harness:processResources NO-SOURCE > Task :runners:java-fn-execution:processResources NO-SOURCE > Task :runners:local-java:processResources NO-SOURCE > Task :runners:direct-java:processResources NO-SOURCE > Task :runners:google-cloud-dataflow-java:processResources UP-TO-DATE > Task :runners:google-cloud-dataflow-java:examples:compileJava NO-SOURCE > Task :runners:google-cloud-dataflow-java:examples:processResources NO-SOURCE > Task :runners:google-cloud-dataflow-java:examples:classes UP-TO-DATE > Task :runners:google-cloud-dataflow-java:examples:compileTestJava NO-SOURCE > Task :model:pipeline:compileJava UP-TO-DATE > Task :runners:google-cloud-dataflow-java:examples:processTestResources NO-SOURCE > Task :runners:google-cloud-dataflow-java:examples:testClasses UP-TO-DATE > Task :model:pipeline:processResources UP-TO-DATE > Task :model:pipeline:classes UP-TO-DATE > Task :model:pipeline:jar UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:windmill:extractIncludeProto UP-TO-DATE > Task :model:pipeline:shadowJar UP-TO-DATE > Task :model:job-management:extractIncludeProto UP-TO-DATE > Task :model:job-management:generateProto UP-TO-DATE > Task :model:fn-execution:extractIncludeProto UP-TO-DATE > Task :model:fn-execution:generateProto UP-TO-DATE > Task :model:job-management:compileJava UP-TO-DATE > Task :model:job-management:classes UP-TO-DATE > Task :model:job-management:shadowJar UP-TO-DATE > Task :model:fn-execution:compileJava UP-TO-DATE > Task :model:fn-execution:classes UP-TO-DATE > Task :model:fn-execution:shadowJar UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:windmill:extractProto UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:windmill:generateProto UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:windmill:compileJava UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:windmill:processResources UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:windmill:classes UP-TO-DATE > Task :sdks:java:core:compileJava UP-TO-DATE > Task :sdks:java:core:classes UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:windmill:shadowJar UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:processResources NO-SOURCE > Task :sdks:java:core:shadowJar UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:compileJava UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:classes UP-TO-DATE > Task :sdks:java:extensions:google-cloud-platform-core:jar UP-TO-DATE > Task :sdks:java:extensions:protobuf:extractIncludeProto UP-TO-DATE > Task :sdks:java:extensions:protobuf:generateProto NO-SOURCE > Task :sdks:java:extensions:protobuf:compileJava UP-TO-DATE > Task :sdks:java:extensions:protobuf:classes UP-TO-DATE > Task :sdks:java:extensions:protobuf:jar UP-TO-DATE > Task :runners:core-construction-java:compileJava UP-TO-DATE > Task :runners:core-construction-java:classes UP-TO-DATE > Task :runners:core-construction-java:jar UP-TO-DATE > Task :sdks:java:fn-execution:compileJava UP-TO-DATE > Task :sdks:java:fn-execution:classes UP-TO-DATE > Task :sdks:java:fn-execution:jar UP-TO-DATE > Task :runners:core-java:compileJava UP-TO-DATE > Task :runners:core-java:classes UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:compileJava UP-TO-DATE > Task :runners:core-java:jar UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:classes UP-TO-DATE > Task :sdks:java:io:google-cloud-platform:jar UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:compileJava UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:classes UP-TO-DATE > Task :vendor:sdks-java-extensions-protobuf:shadowJar UP-TO-DATE > Task :sdks:java:harness:compileJava UP-TO-DATE > Task :sdks:java:harness:classes UP-TO-DATE > Task :sdks:java:harness:jar UP-TO-DATE > Task :examples:java:compileJava UP-TO-DATE > Task :examples:java:classes UP-TO-DATE > Task :examples:java:jar UP-TO-DATE > Task :sdks:java:harness:shadowJar UP-TO-DATE > Task :examples:java:compileTestJava UP-TO-DATE > Task :examples:java:testClasses UP-TO-DATE > Task :examples:java:testJar UP-TO-DATE > Task :runners:local-java:compileJava UP-TO-DATE > Task :runners:local-java:classes UP-TO-DATE > Task :runners:java-fn-execution:compileJava UP-TO-DATE > Task :runners:java-fn-execution:classes UP-TO-DATE > Task :runners:java-fn-execution:jar UP-TO-DATE > Task :runners:local-java:jar UP-TO-DATE > Task :runners:direct-java:compileJava UP-TO-DATE > Task :runners:direct-java:classes UP-TO-DATE > Task :runners:direct-java:shadowJar UP-TO-DATE > Task :runners:google-cloud-dataflow-java:compileJava UP-TO-DATE > Task :runners:google-cloud-dataflow-java:classes UP-TO-DATE > Task :runners:google-cloud-dataflow-java:jar UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:compileJava UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:classes UP-TO-DATE > Task :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar UP-TO-DATE > Task :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker FROM-CACHE Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 2s 65 actionable tasks: 1 from cache, 64 up-to-date root@cfbdbfdf72ea:/beam# *Thanks & Regards* *Rehman Murad Ali* Software Engineer Mobile: +92 3452076766 Skype: rehman,muradali <http://venturedive.com/> On Thu, Nov 28, 2019 at 12:37 PM Michał Walenia <[email protected]> wrote: > That's interesting, the command works for me (it crashes on communication > with GCP, but that was expected). From the log output it seems you're using > Windows. > Did you use WSL to run this? Do you have an option to use Linux to check > this command? The JSON created in the Gradle script may be treated > differently in CMD (due to quoting differences between it and Bash/ZSH/ > other shells). > > I'd try to run this somewhere else other than CMD. > Good luck! > > @Brian: -DbeamPipelineOptions parameter is constructed by Gradle from an > object via JsonOutput, so it should be a valid object after reading back. > > On Thu, Nov 28, 2019 at 7:39 AM Rehman Murad Ali < > [email protected]> wrote: > >> Thank you, Brian and Michal, for replying. Here is the full command: >> >> ./gradlew >> :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker >> -PdataflowProject=apache-beam-testing -Pproject=apache-beam-testing >> -PgcpProject=apache-beam-testing >> -PgcsTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >> -PdataflowTempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java >> >> >> Source: Confluence Java Tips >> <https://cwiki.apache.org/confluence/display/BEAM/Java+Tips> >> >> >> >> *Thanks & Regards* >> >> >> *Rehman Murad Ali* >> >> Software Engineer >> Mobile: +92 3452076766 >> Skype: rehman,muradali >> >> <http://venturedive.com/> >> >> >> On Thu, Nov 28, 2019 at 10:42 AM Zohaib Baig <[email protected]> >> wrote: >> >>> +Rehman Murad Ali <[email protected]> >>> >>> On Thu, Nov 28, 2019 at 2:58 AM Brian Hulette <[email protected]> >>> wrote: >>> >>>> It looks like you passed an argument like >>>> -DbeamTestPipelineOptions >>>> "[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]", >>>> but the string inside the quotes needs to be a valid JSON array of strings. >>>> If you change it to something like -DbeamTestPipelineOptions >>>> '["--project=apache-beam-testing",...]' you should get past that error. >>>> >>>> Agree with Michał though that we could help best if you share your full >>>> command line. >>>> >>>> Brian >>>> >>>> On Wed, Nov 27, 2019 at 8:20 AM Michał Walenia < >>>> [email protected]> wrote: >>>> >>>>> Hi, >>>>> can you please post the command you used in the terminal? It seems you >>>>> used a wrong combination of quotes, but I'd need to see it to be sure. >>>>> Cheers, >>>>> Michal >>>>> >>>>> On Wed, Nov 27, 2019 at 5:11 PM Rehman Murad Ali < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Community, >>>>>> >>>>>> I have been recently trying to test data flow jobs with the beam. I >>>>>> have set up Gcloud account and tried to copy a file from the local system >>>>>> to the Gcloud storage (which works fine). >>>>>> >>>>>> Now I am trying to run preCommitLegacyWorker task on local and I am >>>>>> getting the following error: >>>>>> >>>>>> Unable to instantiate test options from system property >>>>>> beamTestPipelineOptions >>>>>> >>>>>> >>>>>> Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected >>>>>> character ('-' (code 45)) in numeric value: expected digit (0-9) to >>>>>> follow >>>>>> minus sign, for valid numeric value >>>>>> at [Source: >>>>>> (String)"[--project=apache-beam-testing,--tempRoot=gs://venturedive-beamers-shoaib-mszb/rehman-java,--runner=TestDataflowRunner,--dataflowWorkerJar=D:\\Workspace\\apache\\beam\\runners\\google-cloud-dataflow-java\\worker\\legacy-worker\\build\\libs\\beam-runners-google-cloud-dataflow-java-legacy-worker-2.18.0-SNAPSHOT.jar,]"; >>>>>> line: 1, column: 4] >>>>>> at >>>>>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804) >>>>>> at >>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:693) >>>>>> at >>>>>> com.fasterxml.jackson.core.base.ParserMinimalBase.reportUnexpectedNumberChar(ParserMinimalBase.java:541) >>>>>> at >>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleInvalidNumberStart(ReaderBasedJsonParser.java:1637) >>>>>> at >>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser._parseNegNumber(ReaderBasedJsonParser.java:1391) >>>>>> at >>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:742) >>>>>> at >>>>>> com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextTextValue(ReaderBasedJsonParser.java:1160) >>>>>> at >>>>>> com.fasterxml.jackson.databind.deser.std.StringArrayDeserializer.deserialize(StringArrayDeserializer.java:145) >>>>>> >>>>>> >>>>>> >>>>>> Any help would be appreciable. >>>>>> >>>>>> >>>>>> >>>>>> *Thanks & Regards* >>>>>> >>>>>> >>>>>> *Rehman Murad Ali* >>>>>> >>>>>> Software Engineer >>>>>> Mobile: +92 3452076766 <+92%20345%202076766> >>>>>> Skype: rehman,muradali >>>>>> >>>>>> <http://venturedive.com/> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Michał Walenia >>>>> Polidea <https://www.polidea.com/> | Software Engineer >>>>> >>>>> M: +48 791 432 002 <+48791432002> >>>>> E: [email protected] >>>>> >>>>> Unique Tech >>>>> Check out our projects! <https://www.polidea.com/our-work> >>>>> >>>> >>> >>> -- >>> >>> *Muhammad Zohaib Baig* >>> Senior Software Engineer >>> Mobile: +92 3443060266 >>> Skype: mzobii.baig >>> >>> <http://venturedive.com/> >>> >> > > -- > > Michał Walenia > Polidea <https://www.polidea.com/> | Software Engineer > > M: +48 791 432 002 <+48791432002> > E: [email protected] > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work> >
