[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-08 Thread Pei He (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648972#comment-15648972
 ] 

Pei He commented on BEAM-898:
-

One idea to support test arguments is to add "String getTestArg()" in 
TestPipelineOptions.
TestArg could be in the format of "--project=ABC --kind=DEF".
And, each individual IT tests could parse their own test arguemnts.

Any thoughts?

> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645582#comment-15645582
 ] 

ASF GitHub Bot commented on BEAM-898:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1301


> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645525#comment-15645525
 ] 

ASF GitHub Bot commented on BEAM-898:
-

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1301

[BEAM-898] Fix Jenkins Build Failure - IT Options Conflicts

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - Remove checksum variable in IT Options since it causes conflict between 
different ITs.
 - only register TestPipelineOptions.class to avoid incompatibility of 
options defined in different pipeline.

Test is done by running full set of IT tests on Dataflow service.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam fix-IT-option-issue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1301.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1301


commit 0e87f6c2a8a5997c26806283a64d8b812e32b203
Author: Mark Liu 
Date:   2016-11-07T19:27:15Z

Fix IT Options Conflict




> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644614#comment-15644614
 ] 

ASF GitHub Bot commented on BEAM-898:
-

Github user markflyhigh closed the pull request at:

https://github.com/apache/incubator-beam/pull/1288


> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-04 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638200#comment-15638200
 ] 

ASF GitHub Bot commented on BEAM-898:
-

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1288

[BEAM-898] Fix Jenkins IT Options Conflict Error

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

 - change the IT pipeline option to non-global to avoid @Default check 
conflict.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam fix-IT-option-issue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1288.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1288






> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-04 Thread Pei He (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15638117#comment-15638117
 ] 

Pei He commented on BEAM-898:
-

I think if IT tests only register TestPipelineOptions.class, and hard code 
checksum, the problem will go away.

> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-04 Thread Luke Cwik (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637972#comment-15637972
 ] 

Luke Cwik commented on BEAM-898:


Still failing:
https://builds.apache.org/job/beam_PostCommit_MavenVerify/org.apache.beam$beam-examples-java/1734/testReport/

Error Message

Expected getter for property [output] to be marked with @Default on all 
[org.apache.beam.examples.WordCount$WordCountOptions, 
org.apache.beam.examples.cookbook.BigQueryTornadoes$Options], found only on 
[org.apache.beam.examples.WordCount$WordCountOptions]
Stacktrace

java.lang.IllegalArgumentException: Expected getter for property [output] to be 
marked with @Default on all 
[org.apache.beam.examples.WordCount$WordCountOptions, 
org.apache.beam.examples.cookbook.BigQueryTornadoes$Options], found only on 
[org.apache.beam.examples.WordCount$WordCountOptions]
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.throwForGettersWithInconsistentAnnotation(PipelineOptionsFactory.java:1309)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateGettersHaveConsistentAnnotation(PipelineOptionsFactory.java:1150)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateMethodAnnotations(PipelineOptionsFactory.java:1065)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateClass(PipelineOptionsFactory.java:995)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.validateWellFormed(PipelineOptionsFactory.java:627)
at 
org.apache.beam.sdk.options.PipelineOptionsFactory.register(PipelineOptionsFactory.java:561)
at 
org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2EBigQueryTornadoes(BigQueryTornadoesIT.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at 
org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-04 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15637727#comment-15637727
 ] 

ASF GitHub Bot commented on BEAM-898:
-

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-beam/pull/1279


> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-03 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634754#comment-15634754
 ] 

ASF GitHub Bot commented on BEAM-898:
-

GitHub user markflyhigh opened a pull request:

https://github.com/apache/incubator-beam/pull/1279

[BEAM-898] Remove @Default from IT options

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

Fix integration tests failure. It's related to 
[PR-1159](https://github.com/apache/incubator-beam/pull/1159)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markflyhigh/incubator-beam 
fix-IT-PipelineOptions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-beam/pull/1279.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1279


commit 749f9de01b0f38a02b693d67b2457d8e55dee7f7
Author: Mark Liu 
Date:   2016-11-03T23:45:43Z

Remove @Default from IT options




> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Mark Liu
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (BEAM-898) BigQueryTornadoes IT has invalid PipelineOptions

2016-11-03 Thread Pei He (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634438#comment-15634438
 ] 

Pei He commented on BEAM-898:
-

Chatted with Luke and Mark, and there are several ways to fix them:
1. move the default to test scrip. 2. hard code checksum and input. 3. rename 
checksum option and keep @Default. 4. "fork JVM per test"

I am re-assign jira back to Mark to make the call.

> BigQueryTornadoes IT has invalid PipelineOptions
> 
>
> Key: BEAM-898
> URL: https://issues.apache.org/jira/browse/BEAM-898
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp, testing
>Reporter: Daniel Halperin
>Assignee: Pei He
>
> https://builds.apache.org/job/beam_PostCommit_MavenVerify/1718/
> This PR: https://github.com/apache/incubator-beam/pull/1159
> checks that pipeline options cannot have multiple incompatible defaults.
> BigQueryTornadoes ITs have a problem with how they register pipeline options. 
> Luke can give more details on fix.
> cc [~pei...@gmail.com] [~lcwik] [~jasonkuster]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)