[jira] [Commented] (BEAM-1964) Upgrade pylint to 1.7.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968694#comment-15968694
 ] 

ASF GitHub Bot commented on BEAM-1964:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2539

[BEAM-1964] Fix lint issues for linter upgrade -2

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1964-upgrade-pylint-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2539


commit e026ace6d2aae71ec212ff126f2c2f334cabe458
Author: Sourabh Bajaj 
Date:   2017-04-14T00:19:56Z

[BEAM-1964] Fix lint issues for linter upgrade -2




> Upgrade pylint to 1.7.0
> ---
>
> Key: BEAM-1964
> URL: https://issues.apache.org/jira/browse/BEAM-1964
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Aviem Zur
>Assignee: Ahmet Altay
>
> Pre-commit tests seem to all be failing on pylint
> For example: 
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/9493/consoleFull



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2539: [BEAM-1964] Fix lint issues for linter upgrade -2

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2539

[BEAM-1964] Fix lint issues for linter upgrade -2

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1964-upgrade-pylint-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2539.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2539


commit e026ace6d2aae71ec212ff126f2c2f334cabe458
Author: Sourabh Bajaj 
Date:   2017-04-14T00:19:56Z

[BEAM-1964] Fix lint issues for linter upgrade -2




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1757) Improve log configuration when using multiple logging frameworks in Pipeline

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1757:
---

Assignee: (was: Aviem Zur)

> Improve log configuration when using multiple logging frameworks in Pipeline
> 
>
> Key: BEAM-1757
> URL: https://issues.apache.org/jira/browse/BEAM-1757
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Priority: Minor
>
> We need to improve / document how to configure the different logging 
> frameworks to support the different Beam modules (SDK/Runners/IOs).
> I have seen this reported already in slack, but it also bit me recently when 
> trying to log things with a module whose dependencies only supported java 
> based logging over slf4j (no log4j). I tried to provide the classical 
> log4j.properties but I could not get it to log anything, nothing from the 
> sdk, and logically nothing from the java logging framework.
> [Discussion on dev 
> list|https://lists.apache.org/thread.html/502fc4a0575534ddd5f6abffd0743de8d2ccc7fee078b5922c2e1300@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #306

[Apache Jenkins Server]

See 


Changes:

[altay] [BEAM-1964] Fix lint issues for linter upgrade

[altay] fix the simplest typo

[altay] Update java8 examples pom files to include maven-shade-plugin.

--
[...truncated 248.73 KB...]
Resolving deltas:   0% (0/1484)   Resolving deltas:   1% (16/1484)   Resolving 
deltas:   2% (30/1484)   Resolving deltas:   3% (47/1484)   Resolving deltas:   
4% (67/1484)   Resolving deltas:   5% (75/1484)   Resolving deltas:   6% 
(90/1484)   Resolving deltas:   7% (108/1484)   Resolving deltas:   9% 
(135/1484)   Resolving deltas:  13% (203/1484)   Resolving deltas:  16% 
(248/1484)   Resolving deltas:  17% (261/1484)   Resolving deltas:  19% 
(285/1484)   Resolving deltas:  20% (297/1484)   Resolving deltas:  21% 
(312/1484)   Resolving deltas:  22% (328/1484)   Resolving deltas:  23% 
(343/1484)   Resolving deltas:  24% (357/1484)   Resolving deltas:  25% 
(371/1484)   Resolving deltas:  26% (390/1484)   Resolving deltas:  27% 
(401/1484)   Resolving deltas:  28% (426/1484)   Resolving deltas:  29% 
(444/1484)   Resolving deltas:  30% (446/1484)   Resolving deltas:  31% 
(461/1484)   Resolving deltas:  32% (480/1484)   Resolving deltas:  33% 
(501/1484)   Resolving deltas:  34% (506/1484)   Resolving deltas:  35% 
(520/1484)   Resolving deltas:  36% (537/1484)   Resolving deltas:  37% 
(551/1484)   Resolving deltas:  38% (565/1484)   Resolving deltas:  39% 
(579/1484)   Resolving deltas:  40% (595/1484)   Resolving deltas:  41% 
(612/1484)   Resolving deltas:  42% (624/1484)   Resolving deltas:  43% 
(639/1484)   Resolving deltas:  44% (664/1484)   Resolving deltas:  45% 
(677/1484)   Resolving deltas:  46% (694/1484)   Resolving deltas:  47% 
(702/1484)   Resolving deltas:  48% (717/1484)   Resolving deltas:  49% 
(731/1484)   Resolving deltas:  50% (742/1484)   Resolving deltas:  51% 
(757/1484)   Resolving deltas:  52% (780/1484)   Resolving deltas:  53% 
(792/1484)   Resolving deltas:  54% (810/1484)   Resolving deltas:  55% 
(817/1484)   Resolving deltas:  56% (832/1484)   Resolving deltas:  57% 
(846/1484)   Resolving deltas:  58% (861/1484)   Resolving deltas:  59% 
(876/1484)   Resolving deltas:  60% (892/1484)   Resolving deltas:  61% 
(907/1484)   Resolving deltas:  62% (923/1484)   Resolving deltas:  63% 
(935/1484)   Resolving deltas:  64% (950/1484)   Resolving deltas:  65% 
(965/1484)   Resolving deltas:  66% (981/1484)   Resolving deltas:  67% 
(995/1484)   Resolving deltas:  68% (1010/1484)   Resolving deltas:  69% 
(1024/1484)   Resolving deltas:  70% (1039/1484)   Resolving deltas:  71% 
(1054/1484)   Resolving deltas:  72% (1069/1484)   Resolving deltas:  73% 
(1090/1484)   Resolving deltas:  74% (1100/1484)   Resolving deltas:  75% 
(1113/1484)   Resolving deltas:  76% (1128/1484)   Resolving deltas:  77% 
(1143/1484)   Resolving deltas:  78% (1159/1484)   Resolving deltas:  79% 
(1175/1484)   Resolving deltas:  80% (1189/1484)   Resolving deltas:  81% 
(1206/1484)   Resolving deltas:  82% (1224/1484)   Resolving deltas:  83% 
(1234/1484)   Resolving deltas:  84% (1248/1484)   Resolving deltas:  85% 
(1263/1484)   Resolving deltas:  86% (1277/1484)   Resolving deltas:  87% 
(1292/1484)   Resolving deltas:  88% (1306/1484)   Resolving deltas:  89% 
(1322/1484)   Resolving deltas:  90% (1336/1484)   Resolving deltas:  91% 
(1351/1484)   Resolving deltas:  92% (1367/1484)   Resolving deltas:  93% 
(1381/1484)   Resolving deltas:  94% (1395/1484)   Resolving deltas:  95% 
(1410/1484)   Resolving deltas:  96% (1425/1484)   Resolving deltas:  97% 
(1440/1484)   Resolving deltas:  98% (1455/1484)   Resolving deltas:  99% 
(1470/1484)   Resolving deltas: 100% (1484/1484)   Resolving deltas: 100% 
(1484/1484), completed with 522 local objects.
   3fb75d3..4daac66  master -> origin/master
error: unable to resolve reference refs/remotes/origin/pr/1687/merge: No such 
file or directory
 ! 6ee392f...2015abe refs/pull/1687/merge -> origin/pr/1687/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/1725/merge: No such 
file or directory
 ! f3dc640...41265ef refs/pull/1725/merge -> origin/pr/1725/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/1729/merge: No such 
file or directory
 ! 8afd986...3335bab refs/pull/1729/merge -> origin/pr/1729/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/1851/merge: No such 
file or directory
 ! e3df4cd...e0ed99c refs/pull/1851/merge -> origin/pr/1851/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/1898/merge: No such 
file or directory
 ! cb4974c...ce64239 refs/pull/1898/merge -> origin/pr/1898/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2057/merge: No such 
file or directory
 ! 4f67125...1

[jira] [Assigned] (BEAM-1757) Improve log configuration when using multiple logging frameworks in Pipeline

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1757:
---

Assignee: Aviem Zur

> Improve log configuration when using multiple logging frameworks in Pipeline
> 
>
> Key: BEAM-1757
> URL: https://issues.apache.org/jira/browse/BEAM-1757
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Aviem Zur
>Priority: Minor
>
> We need to improve / document how to configure the different logging 
> frameworks to support the different Beam modules (SDK/Runners/IOs).
> I have seen this reported already in slack, but it also bit me recently when 
> trying to log things with a module whose dependencies only supported java 
> based logging over slf4j (no log4j). I tried to provide the classical 
> log4j.properties but I could not get it to log anything, nothing from the 
> sdk, and logically nothing from the java logging framework.
> [Discussion on dev 
> list|https://lists.apache.org/thread.html/502fc4a0575534ddd5f6abffd0743de8d2ccc7fee078b5922c2e1300@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1975) Documentation around logging in different runners

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1975:

Description: 
Add documentation on how to configure logging in different runners, relate to 
SLF4J and bindings, and which binding is used in which runner.
Add helpful links to the different logging configuration guides for the 
bindings used in each runner.

> Documentation around logging in different runners
> -
>
> Key: BEAM-1975
> URL: https://issues.apache.org/jira/browse/BEAM-1975
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Aviem Zur
>Assignee: Davor Bonaci
>
> Add documentation on how to configure logging in different runners, relate to 
> SLF4J and bindings, and which binding is used in which runner.
> Add helpful links to the different logging configuration guides for the 
> bindings used in each runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1757) Improve log configuration when using multiple logging frameworks in Pipeline

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur reassigned BEAM-1757:
---

Assignee: (was: Aviem Zur)

> Improve log configuration when using multiple logging frameworks in Pipeline
> 
>
> Key: BEAM-1757
> URL: https://issues.apache.org/jira/browse/BEAM-1757
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Priority: Minor
>
> We need to improve / document how to configure the different logging 
> frameworks to support the different Beam modules (SDK/Runners/IOs).
> I have seen this reported already in slack, but it also bit me recently when 
> trying to log things with a module whose dependencies only supported java 
> based logging over slf4j (no log4j). I tried to provide the classical 
> log4j.properties but I could not get it to log anything, nothing from the 
> sdk, and logically nothing from the java logging framework.
> [Discussion on dev 
> list|https://lists.apache.org/thread.html/502fc4a0575534ddd5f6abffd0743de8d2ccc7fee078b5922c2e1300@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1976) Allow only one runner profile active at once in examples and archetypes

[Aviem Zur (JIRA)]

Aviem Zur created BEAM-1976:
---

 Summary: Allow only one runner profile active at once in examples 
and archetypes
 Key: BEAM-1976
 URL: https://issues.apache.org/jira/browse/BEAM-1976
 Project: Beam
  Issue Type: Sub-task
  Components: examples-java
Reporter: Aviem Zur
Assignee: Frances Perry


Since only one SLF4J logger binding is allowed in the classpath, we shouldn't 
allow more than one runner profile to be active at once in our 
examples/archetype modules since different runners use different bindings.
Also, remove slf4j-jdk14 dependency from root and place it instead in 
direct-runner and dataflow-runner profiles, for the same reason.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1975) Documentation around logging in different runners

[Aviem Zur (JIRA)]

Aviem Zur created BEAM-1975:
---

 Summary: Documentation around logging in different runners
 Key: BEAM-1975
 URL: https://issues.apache.org/jira/browse/BEAM-1975
 Project: Beam
  Issue Type: Sub-task
  Components: website
Reporter: Aviem Zur
Assignee: Davor Bonaci






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1757) Improve log configuration when using multiple logging frameworks in Pipeline

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1757:

Description: 
We need to improve / document how to configure the different logging frameworks 
to support the different Beam modules (SDK/Runners/IOs).

I have seen this reported already in slack, but it also bit me recently when 
trying to log things with a module whose dependencies only supported java based 
logging over slf4j (no log4j). I tried to provide the classical 
log4j.properties but I could not get it to log anything, nothing from the sdk, 
and logically nothing from the java logging framework.

[Discussion on dev 
list|https://lists.apache.org/thread.html/502fc4a0575534ddd5f6abffd0743de8d2ccc7fee078b5922c2e1300@%3Cdev.beam.apache.org%3E]

  was:
We need to improve / document how to configure the different logging frameworks 
to support the different Beam modules (SDK/Runners/IOs).

I have seen this reported already in slack, but it also bit me recently when 
trying to log things with a module whose dependencies only supported java based 
logging over slf4j (no log4j). I tried to provide the classical 
log4j.properties but I could not get it to log anything, nothing from the sdk, 
and logically nothing from the java logging framework.


> Improve log configuration when using multiple logging frameworks in Pipeline
> 
>
> Key: BEAM-1757
> URL: https://issues.apache.org/jira/browse/BEAM-1757
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Ismaël Mejía
>Assignee: Aviem Zur
>Priority: Minor
>
> We need to improve / document how to configure the different logging 
> frameworks to support the different Beam modules (SDK/Runners/IOs).
> I have seen this reported already in slack, but it also bit me recently when 
> trying to log things with a module whose dependencies only supported java 
> based logging over slf4j (no log4j). I tried to provide the classical 
> log4j.properties but I could not get it to log anything, nothing from the 
> sdk, and logically nothing from the java logging framework.
> [Discussion on dev 
> list|https://lists.apache.org/thread.html/502fc4a0575534ddd5f6abffd0743de8d2ccc7fee078b5922c2e1300@%3Cdev.beam.apache.org%3E]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-301) Add a Beam SQL DSL

[Tyler Akidau (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968651#comment-15968651
 ] 

Tyler Akidau commented on BEAM-301:
---

[~julianhyde], thanks! Make sense. A quick question on that query to make sure 
I'm understanding it correct. It's basically a stream of orders whose unit 
numbers are greater than the average number of units per order for that same 
product over the last year?

> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1974) Metrics documentation

[Aviem Zur (JIRA)]

Aviem Zur created BEAM-1974:
---

 Summary: Metrics documentation
 Key: BEAM-1974
 URL: https://issues.apache.org/jira/browse/BEAM-1974
 Project: Beam
  Issue Type: Improvement
  Components: website
Reporter: Aviem Zur
Assignee: Davor Bonaci


Document metrics API and uses (make sure to remark that it is still 
experimental).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1958) Standard IO Metrics in Java SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968646#comment-15968646
 ] 

ASF GitHub Bot commented on BEAM-1958:
--

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/2538

[BEAM-1958] Standard IO Metrics in Java SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam standard-io-metrics-java

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2538


commit e1b4db663931c363551b262f21c3f20ddb9a211a
Author: Aviem Zur 
Date:   2017-04-13T17:27:33Z

[BEAM-1958] Standard IO Metrics in Java SDK




> Standard IO Metrics in Java SDK
> ---
>
> Key: BEAM-1958
> URL: https://issues.apache.org/jira/browse/BEAM-1958
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Create class {{IOMetrics}} with factories to create standard IO metrics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2538: [BEAM-1958] Standard IO Metrics in Java SDK

[aviemzur]

GitHub user aviemzur opened a pull request:

https://github.com/apache/beam/pull/2538

[BEAM-1958] Standard IO Metrics in Java SDK

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aviemzur/beam standard-io-metrics-java

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2538


commit e1b4db663931c363551b262f21c3f20ddb9a211a
Author: Aviem Zur 
Date:   2017-04-13T17:27:33Z

[BEAM-1958] Standard IO Metrics in Java SDK




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2832

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968552#comment-15968552
 ] 

ASF GitHub Bot commented on BEAM-1886:
--

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/2537

[BEAM-1886] Remove TextIO override in Flink runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1886

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2537


commit d2a037cabcde65dd3aced6ad1e09dbfa6909dc35
Author: JingsongLi 
Date:   2017-04-14T03:22:43Z

[BEAM-1886] Remove TextIO override in Flink runner




> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2537: [BEAM-1886] Remove TextIO override in Flink runner

[JingsongLi]

GitHub user JingsongLi opened a pull request:

https://github.com/apache/beam/pull/2537

[BEAM-1886] Remove TextIO override in Flink runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/JingsongLi/beam BEAM-1886

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2537


commit d2a037cabcde65dd3aced6ad1e09dbfa6909dc35
Author: JingsongLi 
Date:   2017-04-14T03:22:43Z

[BEAM-1886] Remove TextIO override in Flink runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1886) Remove TextIO override in Flink runner

[Jingsong Lee (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingsong Lee reassigned BEAM-1886:
--

Assignee: Jingsong Lee

> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Jingsong Lee
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1886) Remove TextIO override in Flink runner

[Jingsong Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968548#comment-15968548
 ] 

Jingsong Lee commented on BEAM-1886:


Flink WriteAsText and Beam native TextIO There are two main differences:
1.TextIO's localFile is not prefixed, and writeAsText needs to be URI format.
2. The file name of TextIO default is in accordance with the format of 
ShardNameTemplate, and writeAsText is the original file name.

> Remove TextIO override in Flink runner
> --
>
> Key: BEAM-1886
> URL: https://issues.apache.org/jira/browse/BEAM-1886
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
> Fix For: First stable release
>
>
> Today, the Flink runner replaces TextIO with a customized version. I believe 
> this is related to adequate support for files HDFS.
> However, the capabilities are less, in particular the recent support for 
> window-and-pane sharded writes of unbounded collections.
> Concretely, we have had to remove WindowedWordCountIT from the precommit 
> Jenkins run.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2831

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1179) Update assertions of source_test_utils from camelcase to underscore-separated

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968501#comment-15968501
 ] 

ASF GitHub Bot commented on BEAM-1179:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/2536

[BEAM-1179] Renames assertions of source_test_utils

Renames assertions of source_test_utils from camelcase to 
underscore-separated.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam rename_sourcetestutil_asserts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2536


commit 82ba164b6f0ca69abbc707163232fa5b5791dc9a
Author: chamik...@google.com 
Date:   2017-04-14T01:57:04Z

Update assertions of source_test_utils from camelcase to 
underscore-separated.




> Update assertions of source_test_utils from camelcase to underscore-separated
> -
>
> Key: BEAM-1179
> URL: https://issues.apache.org/jira/browse/BEAM-1179
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: First stable release
>
>
> Assertions in source_test_utils module [1] currently have method names in 
> camel case. This was original done so that it's similar to assertions in 
> Python unittest module. But it's better to deprecate these and add 
> underscore-separated variations so that code in Python SDK is consistent.
> [1] 
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/source_test_utils.py



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2536: [BEAM-1179] Renames assertions of source_test_utils

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/2536

[BEAM-1179] Renames assertions of source_test_utils

Renames assertions of source_test_utils from camelcase to 
underscore-separated.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam rename_sourcetestutil_asserts

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2536


commit 82ba164b6f0ca69abbc707163232fa5b5791dc9a
Author: chamik...@google.com 
Date:   2017-04-14T01:57:04Z

Update assertions of source_test_utils from camelcase to 
underscore-separated.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #3299

[Apache Jenkins Server]

See 

[1/2] beam git commit: Update java8 examples pom files to include maven-shade-plugin.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 930196ac4 -> 4daac6644


Update java8 examples pom files to include maven-shade-plugin.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/710b4800
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/710b4800
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/710b4800

Branch: refs/heads/master
Commit: 710b480044b00c314ded7654d6c7e1fe1cff90f7
Parents: 930196a
Author: Ahmet Altay 
Authored: Thu Apr 13 17:28:22 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 18:27:32 2017 -0700

--
 examples/java8/pom.xml | 34 ++
 1 file changed, 34 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/710b4800/examples/java8/pom.xml
--
diff --git a/examples/java8/pom.xml b/examples/java8/pom.xml
index 5ea6ca5..912c341 100644
--- a/examples/java8/pom.xml
+++ b/examples/java8/pom.xml
@@ -155,6 +155,40 @@
 
   
 org.apache.maven.plugins
+maven-shade-plugin
+
+  
+package
+
+  shade
+
+
+  
${project.artifactId}-bundled-${project.version}
+  
+
+  *:*
+
+  
+  
+
+  *:*
+  
+META-INF/*.SF
+META-INF/*.DSA
+META-INF/*.RSA
+  
+
+  
+  
+
+  
+
+  
+
+  
+
+  
+org.apache.maven.plugins
 maven-jar-plugin
   
 

[GitHub] beam pull request #2535: Update java8 examples pom files to include maven-sh...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2535


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2535

[altay]

This closes #2535


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4daac664
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4daac664
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4daac664

Branch: refs/heads/master
Commit: 4daac664426da06268fa84a0a07793f818ce3913
Parents: 930196a 710b480
Author: Ahmet Altay 
Authored: Thu Apr 13 18:29:01 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 18:29:01 2017 -0700

--
 examples/java8/pom.xml | 34 ++
 1 file changed, 34 insertions(+)
--

[jira] [Assigned] (BEAM-1973) Improve error message for missing required Pipeline options in Datalflow runner for Java SDK

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-1973:
-

Assignee: Valentyn Tymofieiev  (was: Davor Bonaci)

> Improve error message for missing required Pipeline options in Datalflow 
> runner for Java SDK
> 
>
> Key: BEAM-1973
> URL: https://issues.apache.org/jira/browse/BEAM-1973
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Minor
>
> Here is an example error message when we fail to set GCP project ID:
> Caused by: java.lang.IllegalArgumentException: Missing required value for 
> [public abstract java.lang.String 
> org.apache.beam.runners.dataflow.options.DataflowPipelineOptions.getProject(),
>  "Project id. Required when running a Dataflow in the cloud. See 
> https://cloud.google.com/storage/docs/projects for further details."].
> We should make the error message more actionable by suggesting a command line 
> parameter that sets the pipeline option or pointing to relevant programmatic 
> API that sets it.
> Full error message:
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Failed to construct instance from 
> factory method DataflowRunner#fromOptions(interface 
> org.apache.beam.sdk.options.PipelineOptions)
> at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
> at 
> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
> at 
> org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:56)
> at org.apache.beam.sdk.Pipeline.create(Pipeline.java:135)
> at org.apache.beam.examples.WordCount.main(WordCount.java:175)
> ... 6 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
> ... 10 more
> Caused by: java.lang.IllegalArgumentException: Missing required value for 
> [public abstract java.lang.String 
> org.apache.beam.runners.dataflow.options.DataflowPipelineOptions.getProject(),
>  "Project id. Required when running a Dataflow in the cloud. See 
> https://cloud.google.com/storage/docs/projects for further details."].
> at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
> at 
> org.apache.beam.sdk.options.PipelineOptionsValidator.validate(PipelineOptionsValidator.java:70)
> at 
> org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:196)
> ... 15 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #3298

[Apache Jenkins Server]

See 

[jira] [Updated] (BEAM-1973) Improve error message for missing required Pipeline options in Datalflow runner for Java SDK

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-1973:
--
Summary: Improve error message for missing required Pipeline options in 
Datalflow runner for Java SDK  (was: Improve error message for missing required 
Pipeline options in Java API)

> Improve error message for missing required Pipeline options in Datalflow 
> runner for Java SDK
> 
>
> Key: BEAM-1973
> URL: https://issues.apache.org/jira/browse/BEAM-1973
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Valentyn Tymofieiev
>Assignee: Davor Bonaci
>Priority: Minor
>
> Here is an example error message when we fail to set GCP project ID:
> Caused by: java.lang.IllegalArgumentException: Missing required value for 
> [public abstract java.lang.String 
> org.apache.beam.runners.dataflow.options.DataflowPipelineOptions.getProject(),
>  "Project id. Required when running a Dataflow in the cloud. See 
> https://cloud.google.com/storage/docs/projects for further details."].
> We should make the error message more actionable by suggesting a command line 
> parameter that sets the pipeline option or pointing to relevant programmatic 
> API that sets it.
> Full error message:
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Failed to construct instance from 
> factory method DataflowRunner#fromOptions(interface 
> org.apache.beam.sdk.options.PipelineOptions)
> at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
> at 
> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
> at 
> org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:56)
> at org.apache.beam.sdk.Pipeline.create(Pipeline.java:135)
> at org.apache.beam.examples.WordCount.main(WordCount.java:175)
> ... 6 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
> ... 10 more
> Caused by: java.lang.IllegalArgumentException: Missing required value for 
> [public abstract java.lang.String 
> org.apache.beam.runners.dataflow.options.DataflowPipelineOptions.getProject(),
>  "Project id. Required when running a Dataflow in the cloud. See 
> https://cloud.google.com/storage/docs/projects for further details."].
> at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
> at 
> org.apache.beam.sdk.options.PipelineOptionsValidator.validate(PipelineOptionsValidator.java:70)
> at 
> org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:196)
> ... 15 more



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1973) Improve error message for missing required Pipeline options in Java API

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-1973:
-

 Summary: Improve error message for missing required Pipeline 
options in Java API
 Key: BEAM-1973
 URL: https://issues.apache.org/jira/browse/BEAM-1973
 Project: Beam
  Issue Type: Improvement
  Components: runner-dataflow
Reporter: Valentyn Tymofieiev
Assignee: Davor Bonaci
Priority: Minor


Here is an example error message when we fail to set GCP project ID:

Caused by: java.lang.IllegalArgumentException: Missing required value for 
[public abstract java.lang.String 
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions.getProject(), 
"Project id. Required when running a Dataflow in the cloud. See 
https://cloud.google.com/storage/docs/projects for further details."].

We should make the error message more actionable by suggesting a command line 
parameter that sets the pipeline option or pointing to relevant programmatic 
API that sets it.

Full error message:

java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Failed to construct instance from 
factory method DataflowRunner#fromOptions(interface 
org.apache.beam.sdk.options.PipelineOptions)
at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at 
org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at 
org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:56)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:135)
at org.apache.beam.examples.WordCount.main(WordCount.java:175)
... 6 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 10 more
Caused by: java.lang.IllegalArgumentException: Missing required value for 
[public abstract java.lang.String 
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions.getProject(), 
"Project id. Required when running a Dataflow in the cloud. See 
https://cloud.google.com/storage/docs/projects for further details."].
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:383)
at 
org.apache.beam.sdk.options.PipelineOptionsValidator.validate(PipelineOptionsValidator.java:70)
at 
org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:196)
... 15 more





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-375) HadoopIO and runners-spark conflict with hadoop.version

[liyuntian (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968410#comment-15968410
 ] 

liyuntian edited comment on BEAM-375 at 4/14/17 1:15 AM:
-

yes,I get state by "pipelineResult.getState().isTerminal()" ,But ,IT IS running 
all the time，and the spark executors Disk Used is 0.0B, but when I change 
hadoop version to 2.7.0,I can get the result correctly.so I want to know  I 
must use Hadoop2.7.0 in beam0.5?perhaps I lost some jars?


was (Author: yuntian):
yes,I get state by "pipelineResult.getState().isTerminal()" ,But ,IT IS running 
all the time，and the spark executors Disk Used is 0.0B, but when I change 
hadoop version to 2.7.0,I can get the result correctly.so I want to know  I 
must use Hadoop2.7.0 in beam0.5?

> HadoopIO and runners-spark conflict with hadoop.version
> ---
>
> Key: BEAM-375
> URL: https://issues.apache.org/jira/browse/BEAM-375
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Pei He
>Assignee: Pei He
>
> HadoopIO currently uses 2.7.0 and runners-spark uses 2.2.0 for hadoop-client, 
> hadoop-common.
> From [~amitsela]
> "Spark can be built against different hadoop versions, but the release in 
> maven central is a 2.2.0 build (latest). ''
> For HadoopIO, I don't know why 2.7.0 is picked at the beginning. I can check 
> if it will work with 2.2.0.
> I am creating this issue, since I think it there is a general question.
> In principle, HadoopIO and other sdks Sources should work with any runners. 
> But, when one set of runners require version A, but the other set of runners 
> require version B, we will need a general solution for it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2830

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #3297

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-1101) Remove inconsistencies in Python PipelineOptions

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1101.
---
Resolution: Fixed

> Remove inconsistencies in Python PipelineOptions
> 
>
> Key: BEAM-1101
> URL: https://issues.apache.org/jira/browse/BEAM-1101
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> Some options have been removed from Java, and some have different default 
> behavior in Java. Gotta make this consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-301) Add a Beam SQL DSL

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968460#comment-15968460
 ] 

Julian Hyde commented on BEAM-301:
--

[~takidau], +1 to that approach.

In the short term I think we will have multiple variants of streaming SQL. 
Calcite will support STREAM and using the same identifier for streams/tables, 
and will provide a 
[switch|https://calcite.apache.org/apidocs/org/apache/calcite/sql/validate/SqlConformance.html]
 so that Beam can disable them. Over the longer term I will try make the case 
that these features are useful (or I will fail, and these features will wither 
away). I can't really make the case until we have features like self-join of a 
stream to its own history.

The crucial query which illustrates this is this one:

{code}select stream *
from Orders as o
where units > (
  select avg(units)
  from Orders as h
  where h.productId = o.productId
  and h.rowtime > o.rowtime - interval ‘1’ year){code}

It combines the {{Orders}} stream with its own history. But after the query has 
been running for a while, the records that passed through the stream will have 
entered the history. The history relation {{h}} is neither bounded, nor 
unbounded (in Beam's terms), but time-varying.

> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2534

[altay]

This closes #2534


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/930196ac
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/930196ac
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/930196ac

Branch: refs/heads/master
Commit: 930196ac46063d455c2e3ecf4a67bf0775c93a4b
Parents: 08da94d f7af9a2
Author: Ahmet Altay 
Authored: Thu Apr 13 17:37:37 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 17:37:37 2017 -0700

--
 sdks/python/apache_beam/transforms/trigger.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] beam pull request #2534: fix the simplest typo

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2534


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: fix the simplest typo

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 08da94df4 -> 930196ac4


fix the simplest typo


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f7af9a27
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f7af9a27
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f7af9a27

Branch: refs/heads/master
Commit: f7af9a2718f70398d855482e1de16df8f51f3950
Parents: 08da94d
Author: Ahmet Altay 
Authored: Thu Apr 13 16:59:01 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 17:37:30 2017 -0700

--
 sdks/python/apache_beam/transforms/trigger.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f7af9a27/sdks/python/apache_beam/transforms/trigger.py
--
diff --git a/sdks/python/apache_beam/transforms/trigger.py 
b/sdks/python/apache_beam/transforms/trigger.py
index a527c82..6a4cf24 100644
--- a/sdks/python/apache_beam/transforms/trigger.py
+++ b/sdks/python/apache_beam/transforms/trigger.py
@@ -839,7 +839,7 @@ class TriggerDriver(object):
 
 
 class _UnwindowedValues(observable.ObservableMixin):
-  """Exposes iterable of windowed values as interable of unwindowed values."""
+  """Exposes iterable of windowed values as iterable of unwindowed values."""
 
   def __init__(self, windowed_values):
 super(_UnwindowedValues, self).__init__()

[GitHub] beam pull request #2535: Update java8 examples pom files to include maven-sh...

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2535

Update java8 examples pom files to include maven-shade-plugin

R: @davorbonaci 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam pom

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2535


commit 590e4aef30ac3bbffa83aaa23f457aa1bb6e9c81
Author: Ahmet Altay 
Date:   2017-04-14T00:28:22Z

Update java8 examples pom files to include maven-shade-plugin.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1964) Upgrade pylint to 1.7.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968434#comment-15968434
 ] 

ASF GitHub Bot commented on BEAM-1964:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2532


> Upgrade pylint to 1.7.0
> ---
>
> Key: BEAM-1964
> URL: https://issues.apache.org/jira/browse/BEAM-1964
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Aviem Zur
>Assignee: Ahmet Altay
>
> Pre-commit tests seem to all be failing on pylint
> For example: 
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/9493/consoleFull



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2532: [BEAM-1964] Fix lint issues for linter upgrade - pa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2532


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1964] Fix lint issues for linter upgrade

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 3fb75d3c2 -> 08da94df4


[BEAM-1964] Fix lint issues for linter upgrade


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d2a7d1e8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d2a7d1e8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d2a7d1e8

Branch: refs/heads/master
Commit: d2a7d1e8af904d432f0a5613ed1ff0b87a79d3ec
Parents: 3fb75d3
Author: Sourabh Bajaj 
Authored: Thu Apr 13 16:18:32 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 17:17:24 2017 -0700

--
 sdks/python/apache_beam/coders/coder_impl.py|  50 +
 sdks/python/apache_beam/coders/coders.py|   6 +-
 .../examples/complete/game/user_score.py|   3 -
 .../examples/complete/top_wikipedia_sessions.py |   8 --
 .../examples/snippets/snippets_test.py  |   3 +-
 .../apache_beam/examples/wordcount_debugging.py |   4 -
 sdks/python/apache_beam/pvalue.py   |  16 +--
 sdks/python/apache_beam/transforms/core.py  | 102 ---
 sdks/python/apache_beam/transforms/display.py   |   3 +-
 .../python/apache_beam/transforms/ptransform.py |  36 +++
 .../apache_beam/transforms/ptransform_test.py   |   3 +-
 .../python/apache_beam/transforms/sideinputs.py |  11 +-
 sdks/python/apache_beam/transforms/trigger.py   |  24 ++---
 .../apache_beam/transforms/trigger_test.py  |   6 +-
 sdks/python/apache_beam/typehints/decorators.py |  23 ++---
 .../apache_beam/typehints/trivial_inference.py  |  26 ++---
 .../typehints/trivial_inference_test.py |   3 +-
 sdks/python/apache_beam/typehints/typecheck.py  |   7 +-
 sdks/python/apache_beam/typehints/typehints.py  |  63 +---
 .../apache_beam/typehints/typehints_test.py |   5 +-
 .../apache_beam/utils/annotations_test.py   |   2 +-
 sdks/python/apache_beam/utils/path.py   |   3 +-
 sdks/python/apache_beam/utils/proto_utils.py|  15 ++-
 sdks/python/apache_beam/utils/retry.py  |  12 +--
 sdks/python/apache_beam/utils/timestamp.py  |   6 +-
 sdks/python/apache_beam/utils/windowed_value.py |  17 ++--
 sdks/python/run_pylint.sh   |   2 +-
 27 files changed, 177 insertions(+), 282 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d2a7d1e8/sdks/python/apache_beam/coders/coder_impl.py
--
diff --git a/sdks/python/apache_beam/coders/coder_impl.py 
b/sdks/python/apache_beam/coders/coder_impl.py
index 137d1be..d56606d 100644
--- a/sdks/python/apache_beam/coders/coder_impl.py
+++ b/sdks/python/apache_beam/coders/coder_impl.py
@@ -161,8 +161,8 @@ class CallbackCoderImpl(CoderImpl):
 if isinstance(value, observable.ObservableMixin):
   # CallbackCoderImpl can presumably encode the elements too.
   return 1, [(value, self)]
-else:
-  return self.estimate_size(value, nested), []
+
+return self.estimate_size(value, nested), []
 
 
 class DeterministicFastPrimitivesCoderImpl(CoderImpl):
@@ -243,10 +243,10 @@ class FastPrimitivesCoderImpl(StreamCoderImpl):
 if isinstance(value, observable.ObservableMixin):
   # FastPrimitivesCoderImpl can presumably encode the elements too.
   return 1, [(value, self)]
-else:
-  out = ByteCountingOutputStream()
-  self.encode_to_stream(value, out, nested)
-  return out.get_count(), []
+
+out = ByteCountingOutputStream()
+self.encode_to_stream(value, out, nested)
+return out.get_count(), []
 
   def encode_to_stream(self, value, stream, nested):
 t = type(value)
@@ -304,8 +304,7 @@ class FastPrimitivesCoderImpl(StreamCoderImpl):
 return vlist
   elif t == TUPLE_TYPE:
 return tuple(vlist)
-  else:
-return set(vlist)
+  return set(vlist)
 elif t == DICT_TYPE:
   vlen = stream.read_var_int64()
   v = {}
@@ -315,8 +314,8 @@ class FastPrimitivesCoderImpl(StreamCoderImpl):
   return v
 elif t == BOOL_TYPE:
   return not not stream.read_byte()
-else:
-  return self.fallback_coder_impl.decode_from_stream(stream, nested)
+
+return self.fallback_coder_impl.decode_from_stream(stream, nested)
 
 
 class BytesCoderImpl(CoderImpl):
@@ -408,8 +407,7 @@ class VarIntCoderImpl(StreamCoderImpl):
 ivalue = value  # type cast
 if 0 <= ivalue < len(small_ints):
   return small_ints[ivalue]
-else:
-  return StreamCoderImpl.encode(self, value)
+return StreamCoderImpl.encode(self, value)
 
   def decode(self, encoded):
 if len(encoded) == 1:
@@ -596,20 +594,20 @@ class SequenceCoderImpl(StreamCoderImpl):
 estimated_size += 4
 if isinstance(value, observable.ObservableMixin):
   return estimated_size, [(value, self._elem_coder)]
-else:
-  observables = []
-  f

[2/2] beam git commit: This closes #2532

[altay]

This closes #2532


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/08da94df
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/08da94df
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/08da94df

Branch: refs/heads/master
Commit: 08da94df404b00f62a6b250caf6d597c0430ae27
Parents: 3fb75d3 d2a7d1e
Author: Ahmet Altay 
Authored: Thu Apr 13 17:17:29 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 17:17:29 2017 -0700

--
 sdks/python/apache_beam/coders/coder_impl.py|  50 +
 sdks/python/apache_beam/coders/coders.py|   6 +-
 .../examples/complete/game/user_score.py|   3 -
 .../examples/complete/top_wikipedia_sessions.py |   8 --
 .../examples/snippets/snippets_test.py  |   3 +-
 .../apache_beam/examples/wordcount_debugging.py |   4 -
 sdks/python/apache_beam/pvalue.py   |  16 +--
 sdks/python/apache_beam/transforms/core.py  | 102 ---
 sdks/python/apache_beam/transforms/display.py   |   3 +-
 .../python/apache_beam/transforms/ptransform.py |  36 +++
 .../apache_beam/transforms/ptransform_test.py   |   3 +-
 .../python/apache_beam/transforms/sideinputs.py |  11 +-
 sdks/python/apache_beam/transforms/trigger.py   |  24 ++---
 .../apache_beam/transforms/trigger_test.py  |   6 +-
 sdks/python/apache_beam/typehints/decorators.py |  23 ++---
 .../apache_beam/typehints/trivial_inference.py  |  26 ++---
 .../typehints/trivial_inference_test.py |   3 +-
 sdks/python/apache_beam/typehints/typecheck.py  |   7 +-
 sdks/python/apache_beam/typehints/typehints.py  |  63 +---
 .../apache_beam/typehints/typehints_test.py |   5 +-
 .../apache_beam/utils/annotations_test.py   |   2 +-
 sdks/python/apache_beam/utils/path.py   |   3 +-
 sdks/python/apache_beam/utils/proto_utils.py|  15 ++-
 sdks/python/apache_beam/utils/retry.py  |  12 +--
 sdks/python/apache_beam/utils/timestamp.py  |   6 +-
 sdks/python/apache_beam/utils/windowed_value.py |  17 ++--
 sdks/python/run_pylint.sh   |   2 +-
 27 files changed, 177 insertions(+), 282 deletions(-)
--

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink #2316

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1968) Tests don't pass in the Java8 Examples Archetype under the flink profile

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968424#comment-15968424
 ] 

Thomas Groh commented on BEAM-1968:
---

This may only apply to the 0.6.0 archetype, as it's not the case in the 
examples/java8/ module

> Tests don't pass in the Java8 Examples Archetype under the flink profile
> 
>
> Key: BEAM-1968
> URL: https://issues.apache.org/jira/browse/BEAM-1968
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: Thomas Groh
>
> Likely in all profiles. It's because the unit tests either aren't configured 
> to not run in those profiles, or because the profiles aren't configured to 
> set up the TestPipelineOptions. Either is sufficient to stop teh build from 
> failing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PerformanceTests_Dataflow #305

[Apache Jenkins Server]

See 


Changes:

[altay] Pin versions in tox script

[altay] Update user_score.py

[sourabhbajaj] [BEAM-1101, BEAM-1068] Remove service account name credential 
pipeline

[altay] Improve gcloud logging message

[altay] [BEAM-1708] Improve error message when GCP not installed

[dhalperi] Remove Triggers.java from SDK entirely

[dhalperi] Merge PR#2423: Add Kubernetes scripts for clusters for Performance 
and

--
[...truncated 257.47 KB...]
error: unable to resolve reference refs/remotes/origin/pr/2289/merge: No such 
file or directory
 ! f2cd85f...65b38a8 refs/pull/2289/merge -> origin/pr/2289/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2331/merge: No such 
file or directory
 ! 08461a1...ea1a1c1 refs/pull/2331/merge -> origin/pr/2331/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2339/merge: No such 
file or directory
 ! 00175d7...a15b8bc refs/pull/2339/merge -> origin/pr/2339/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2341/merge: No such 
file or directory
 ! 6829e8e...2885312 refs/pull/2341/merge -> origin/pr/2341/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2344/merge: No such 
file or directory
 ! a88c6b1...fd7b79c refs/pull/2344/merge -> origin/pr/2344/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2350/merge: No such 
file or directory
 ! c7799e0...4c4970e refs/pull/2350/merge -> origin/pr/2350/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2358/merge: No such 
file or directory
 ! 2f534a5...e659631 refs/pull/2358/merge -> origin/pr/2358/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2367/merge: No such 
file or directory
 ! 9bedccd...27660d8 refs/pull/2367/merge -> origin/pr/2367/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2382/merge: No such 
file or directory
 ! 3b11d5e...0b9e094 refs/pull/2382/merge -> origin/pr/2382/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2394/merge: No such 
file or directory
 ! 241ec60...0da4bb9 refs/pull/2394/merge -> origin/pr/2394/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2406/merge: No such 
file or directory
 ! 83f1610...01efd4d refs/pull/2406/merge -> origin/pr/2406/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2407/merge: No such 
file or directory
 ! 62ddf5b...774b88a refs/pull/2407/merge -> origin/pr/2407/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2410/merge: No such 
file or directory
 ! f06b0c8...57fedf9 refs/pull/2410/merge -> origin/pr/2410/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2411/merge: No such 
file or directory
 ! a96af72...c155a44 refs/pull/2411/merge -> origin/pr/2411/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2412/merge: No such 
file or directory
 ! 8d36b9c...4549102 refs/pull/2412/merge -> origin/pr/2412/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2414/merge: No such 
file or directory
 ! 49fd3ee...6f97586 refs/pull/2414/merge -> origin/pr/2414/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2417/merge: No such 
file or directory
 ! fad0cc2...9be1fc4 refs/pull/2417/merge -> origin/pr/2417/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2421/merge: No such 
file or directory
 ! 1f4bba2...af0d020 refs/pull/2421/merge -> origin/pr/2421/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2423/merge: No such 
file or directory
 ! 55ee39c...7da0ba1 refs/pull/2423/merge -> origin/pr/2423/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2426/merge: No such 
file or directory
 ! af76f0b...08315f8 refs/pull/2426/merge -> origin/pr/2426/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2428/merge: No such 
file or directory
 ! ccb8287...a471275 refs/pull/2428/merge -> origin/pr/2428/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2441/merge: No such 
file or directory
 ! d341a1b...ce97533 refs/pull/2441/merge -> origin/pr/2441/merge  (unable to 
update local ref)
error: unable to resolve reference refs/remotes/origin/pr/2442/merge: No such 
file or directory
 ! 8b033b5...7c181c3 refs/pull/2442/merge -> origin/pr/

[GitHub] beam pull request #2534: fix the simplest typo

[aaltay]

GitHub user aaltay opened a pull request:

https://github.com/apache/beam/pull/2534

fix the simplest typo

@tvalentyn 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aaltay/beam typo

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2534


commit cf77b54c49d5f57b93026204067fcf9555d2d542
Author: Ahmet Altay 
Date:   2017-04-13T23:59:01Z

fix the simplest typo




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1949) Rename DoFn.Context#sideOutput to #output

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968415#comment-15968415
 ] 

ASF GitHub Bot commented on BEAM-1949:
--

GitHub user melap opened a pull request:

https://github.com/apache/beam-site/pull/212

[BEAM-1949] Associated doc changes for renaming sideOutput to output

Doc changes associated with apache/beam#2512
This does not change any references to side inputs in Python related 
comments and past blog posts. I can change the past blog posts if desired.
I also split out places that had a single "Side Inputs and Side Outputs" 
section to separate sections, "Side Inputs" and "Additional Outputs" so they 
aren't lumped together anymore.
R: @tgroh 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/beam-site outputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/212.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #212


commit a425683c964895dabfb1a4c13ab50ffc73b15b44
Author: melissa 
Date:   2017-04-13T22:46:31Z

[BEAM-1949] Associated doc changes for renaming sideOutput to output




> Rename DoFn.Context#sideOutput to #output
> -
>
> Key: BEAM-1949
> URL: https://issues.apache.org/jira/browse/BEAM-1949
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>
> Having two methods, both named output, one which takes the "main output type" 
> and one that takes a tag to specify the type more clearly communicates the 
> actual behavior - sideOutput isn't a "special" way to output, it's the same 
> as output(T), just to a specified PCollection. This will help pipeline 
> authors understand the actual behavior of outputting to a tag, and detangle 
> it from "sideInput", which is a special way to receive input. Giving them the 
> same name means that it's not even strange to call output and provide the 
> main output type, which is what we want - it's a more specific way to output, 
> but does not have different restrictions or capabilities.
> This is also a pretty small change within the SDK - it touches about 20 
> files, and the changes are pretty automatic.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1972) HIFIO jdk module fails enforcer when only java 7 is installed on machine

[Stephen Sisk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968414#comment-15968414
 ] 

Stephen Sisk commented on BEAM-1972:


cc [~jasonkuster] I thought we had a CI test covering this scenario. If not, we 
should add one.

Is that in the travis tests that everyone is ignoring right now?

> HIFIO jdk module fails enforcer when only java 7 is installed on machine
> 
>
> Key: BEAM-1972
> URL: https://issues.apache.org/jira/browse/BEAM-1972
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: First stable release
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> When running on a machine with only java 7, you cannot do a full compile of 
> the project. 
> The error message is:
> [WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed 
> with message:
> Detected JDK Version: 1.7.0-121 is not in the allowed range [1.8,).
> The failed module is:
> [INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop :: jdk1.8-tests FAILURE [ 
> 37.549 s]
> This is likely due to the enforcer rules at 
> https://github.com/apache/beam/blob/master/sdks/java/io/hadoop/jdk1.8-tests/pom.xml#L39
> without this exception to the enforcer rules, it throws errors since the 
> cassandra dependency in that directory is java 8, so the intent is that 
> enforcer should not error out since this module. 
> Below are the errors (well, warnings) when run with [1.8,) 
> is commented out: 
> WARNING] 
> /home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java:
>  
> /home/sisk/.m2/repository/org/apache/cassandra/cassandra-all/3.9/cassandra-all-3.9.jar(org/apache/cassandra/service/EmbeddedCassandraService.class):
>  major version 52 is newer than 51, the highest major version supported by 
> this compiler.
>   It is recommended that the compiler be upgraded.
> [WARNING] 
> /home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithElasticTest.java:
>  
> /home/sisk/.m2/repository/org/elasticsearch/elasticsearch/5.0.0/elasticsearch-5.0.0.jar(org/elasticsearch/action/admin/indices/create/CreateIndexRequest.class):
>  major version 52 is newer than 51, the highest major version supported by 
> this compiler.
>   It is recommended that the compiler be upgraded.
> [WARNING] 
> /home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithElasticTest.java:
>  
> /home/sisk/.m2/repository/org/elasticsearch/elasticsearch/5.0.0/elasticsearch-5.0.0.jar(org/elasticsearch/action/admin/indices/delete/DeleteIndexRequest.class):
>  major version 52 is newer than 51, the highest major version supported by 
> this compiler.
>   It is recommended that the compiler be upgraded.
> ...and a couple more of these errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #212: [BEAM-1949] Associated doc changes for renaming...

[melap]

GitHub user melap opened a pull request:

https://github.com/apache/beam-site/pull/212

[BEAM-1949] Associated doc changes for renaming sideOutput to output

Doc changes associated with apache/beam#2512
This does not change any references to side inputs in Python related 
comments and past blog posts. I can change the past blog posts if desired.
I also split out places that had a single "Side Inputs and Side Outputs" 
section to separate sections, "Side Inputs" and "Additional Outputs" so they 
aren't lumped together anymore.
R: @tgroh 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/beam-site outputs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/212.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #212


commit a425683c964895dabfb1a4c13ab50ffc73b15b44
Author: melissa 
Date:   2017-04-13T22:46:31Z

[BEAM-1949] Associated doc changes for renaming sideOutput to output




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (BEAM-1972) HIFIO jdk module fails enforcer when only java 7 is installed on machine

[Stephen Sisk (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Sisk reassigned BEAM-1972:
--

Assignee: Stephen Sisk  (was: Davor Bonaci)

> HIFIO jdk module fails enforcer when only java 7 is installed on machine
> 
>
> Key: BEAM-1972
> URL: https://issues.apache.org/jira/browse/BEAM-1972
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Affects Versions: First stable release
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>
> When running on a machine with only java 7, you cannot do a full compile of 
> the project. 
> The error message is:
> [WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed 
> with message:
> Detected JDK Version: 1.7.0-121 is not in the allowed range [1.8,).
> The failed module is:
> [INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop :: jdk1.8-tests FAILURE [ 
> 37.549 s]
> This is likely due to the enforcer rules at 
> https://github.com/apache/beam/blob/master/sdks/java/io/hadoop/jdk1.8-tests/pom.xml#L39
> without this exception to the enforcer rules, it throws errors since the 
> cassandra dependency in that directory is java 8, so the intent is that 
> enforcer should not error out since this module. 
> Below are the errors (well, warnings) when run with [1.8,) 
> is commented out: 
> WARNING] 
> /home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java:
>  
> /home/sisk/.m2/repository/org/apache/cassandra/cassandra-all/3.9/cassandra-all-3.9.jar(org/apache/cassandra/service/EmbeddedCassandraService.class):
>  major version 52 is newer than 51, the highest major version supported by 
> this compiler.
>   It is recommended that the compiler be upgraded.
> [WARNING] 
> /home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithElasticTest.java:
>  
> /home/sisk/.m2/repository/org/elasticsearch/elasticsearch/5.0.0/elasticsearch-5.0.0.jar(org/elasticsearch/action/admin/indices/create/CreateIndexRequest.class):
>  major version 52 is newer than 51, the highest major version supported by 
> this compiler.
>   It is recommended that the compiler be upgraded.
> [WARNING] 
> /home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithElasticTest.java:
>  
> /home/sisk/.m2/repository/org/elasticsearch/elasticsearch/5.0.0/elasticsearch-5.0.0.jar(org/elasticsearch/action/admin/indices/delete/DeleteIndexRequest.class):
>  major version 52 is newer than 51, the highest major version supported by 
> this compiler.
>   It is recommended that the compiler be upgraded.
> ...and a couple more of these errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2533: Javadoc fixups after style guide changes

[jkff]

GitHub user jkff opened a pull request:

https://github.com/apache/beam/pull/2533

Javadoc fixups after style guide changes

R: @tgroh   

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jkff/incubator-beam style-javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2533


commit aa5f8441c7e510a2b023a2f5c06db5b24326a0c2
Author: Eugene Kirpichov 
Date:   2017-04-13T23:55:12Z

Javadoc fixups after style guide changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-375) HadoopIO and runners-spark conflict with hadoop.version

[liyuntian (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968410#comment-15968410
 ] 

liyuntian commented on BEAM-375:


yes,I get state by "pipelineResult.getState().isTerminal()" ,But ,IT IS running 
all the time，and the spark executors Disk Used is 0.0B, but when I change 
hadoop version to 2.7.0,I can get the result correctly.so I want to know  I 
must use Hadoop2.7.0 in beam0.5?

> HadoopIO and runners-spark conflict with hadoop.version
> ---
>
> Key: BEAM-375
> URL: https://issues.apache.org/jira/browse/BEAM-375
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Pei He
>Assignee: Pei He
>
> HadoopIO currently uses 2.7.0 and runners-spark uses 2.2.0 for hadoop-client, 
> hadoop-common.
> From [~amitsela]
> "Spark can be built against different hadoop versions, but the release in 
> maven central is a 2.2.0 build (latest). ''
> For HadoopIO, I don't know why 2.7.0 is picked at the beginning. I can check 
> if it will work with 2.2.0.
> I am creating this issue, since I think it there is a general question.
> In principle, HadoopIO and other sdks Sources should work with any runners. 
> But, when one set of runners require version A, but the other set of runners 
> require version B, we will need a general solution for it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2829

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #2315

[Apache Jenkins Server]

See 


--
[...truncated 790.21 KB...]
 x [deleted] (none) -> origin/pr/913/merge
 x [deleted] (none) -> origin/pr/914/head
 x [deleted] (none) -> origin/pr/914/merge
 x [deleted] (none) -> origin/pr/915/head
 x [deleted] (none) -> origin/pr/915/merge
 x [deleted] (none) -> origin/pr/916/head
 x [deleted] (none) -> origin/pr/916/merge
 x [deleted] (none) -> origin/pr/917/head
 x [deleted] (none) -> origin/pr/917/merge
 x [deleted] (none) -> origin/pr/918/head
 x [deleted] (none) -> origin/pr/918/merge
 x [deleted] (none) -> origin/pr/919/head
 x [deleted] (none) -> origin/pr/919/merge
 x [deleted] (none) -> origin/pr/92/head
 x [deleted] (none) -> origin/pr/92/merge
 x [deleted] (none) -> origin/pr/920/head
 x [deleted] (none) -> origin/pr/920/merge
 x [deleted] (none) -> origin/pr/921/head
 x [deleted] (none) -> origin/pr/921/merge
 x [deleted] (none) -> origin/pr/922/head
 x [deleted] (none) -> origin/pr/922/merge
 x [deleted] (none) -> origin/pr/923/head
 x [deleted] (none) -> origin/pr/924/head
 x [deleted] (none) -> origin/pr/925/head
 x [deleted] (none) -> origin/pr/925/merge
 x [deleted] (none) -> origin/pr/926/head
 x [deleted] (none) -> origin/pr/926/merge
 x [deleted] (none) -> origin/pr/927/head
 x [deleted] (none) -> origin/pr/927/merge
 x [deleted] (none) -> origin/pr/928/head
 x [deleted] (none) -> origin/pr/929/head
 x [deleted] (none) -> origin/pr/93/head
 x [deleted] (none) -> origin/pr/930/head
 x [deleted] (none) -> origin/pr/930/merge
 x [deleted] (none) -> origin/pr/931/head
 x [deleted] (none) -> origin/pr/931/merge
 x [deleted] (none) -> origin/pr/932/head
 x [deleted] (none) -> origin/pr/932/merge
 x [deleted] (none) -> origin/pr/933/head
 x [deleted] (none) -> origin/pr/933/merge
 x [deleted] (none) -> origin/pr/934/head
 x [deleted] (none) -> origin/pr/934/merge
 x [deleted] (none) -> origin/pr/935/head
 x [deleted] (none) -> origin/pr/936/head
 x [deleted] (none) -> origin/pr/936/merge
 x [deleted] (none) -> origin/pr/937/head
 x [deleted] (none) -> origin/pr/937/merge
 x [deleted] (none) -> origin/pr/938/head
 x [deleted] (none) -> origin/pr/939/head
 x [deleted] (none) -> origin/pr/94/head
 x [deleted] (none) -> origin/pr/940/head
 x [deleted] (none) -> origin/pr/940/merge
 x [deleted] (none) -> origin/pr/941/head
 x [deleted] (none) -> origin/pr/941/merge
 x [deleted] (none) -> origin/pr/942/head
 x [deleted] (none) -> origin/pr/942/merge
 x [deleted] (none) -> origin/pr/943/head
 x [deleted] (none) -> origin/pr/943/merge
 x [deleted] (none) -> origin/pr/944/head
 x [deleted] (none) -> origin/pr/945/head
 x [deleted] (none) -> origin/pr/945/merge
 x [deleted] (none) -> origin/pr/946/head
 x [deleted] (none) -> origin/pr/946/merge
 x [deleted] (none) -> origin/pr/947/head
 x [deleted] (none) -> origin/pr/947/merge
 x [deleted] (none) -> origin/pr/948/head
 x [deleted] (none) -> origin/pr/948/merge
 x [deleted] (none) -> origin/pr/949/head
 x [deleted] (none) -> origin/pr/949/merge
 x [deleted] (none) -> origin/pr/95/head
 x [deleted] (none) -> origin/pr/95/merge
 x [deleted] (none) -> origin/pr/950/head
 x [deleted] (none) -> origin/pr/951/head
 x [deleted] (none) -> origin/pr/951/merge
 x [deleted] (none) -> origin/pr/952/head
 x [deleted] (none) -> origin/pr/952/merge
 x [deleted] (none) -> origin/pr/953/head
 x [deleted] (none) -> origin/pr/954/head
 x [deleted] (none) -> origin/pr/954/merge
 x [deleted] (none) -> origin/pr/955/head
 x [deleted] (none) -> origin/pr/955/merge
 x [deleted] (none) -> origin/pr/956/head
 x [deleted] (none) -> origin/pr/957/head
 x [deleted] (none) -> origin/pr/958/head
 x [deleted] (none) -> origin/pr/959/head
 x [deleted] (none) -> origin/pr/959/merge
 x [deleted] (none) -> origin/pr/96/head
 x [deleted] (none) -> origin/pr/96/merge
 x [deleted] (none) -> origin/pr

[1/3] beam-site git commit: Update quickstart-java.md

[altay]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site ce7149962 -> e6d68ba89


Update quickstart-java.md

Add --project pipeline option in command line for WordCount example on Dataflow 
Runner.

Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/e2964720
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/e2964720
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/e2964720

Branch: refs/heads/asf-site
Commit: e2964720852184f2bdce1469a4c9f10f23c03b74
Parents: ce71499
Author: tvalentyn 
Authored: Thu Apr 13 16:22:38 2017 -0700
Committer: GitHub 
Committed: Thu Apr 13 16:22:38 2017 -0700

--
 src/get-started/quickstart-java.md | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/e2964720/src/get-started/quickstart-java.md
--
diff --git a/src/get-started/quickstart-java.md 
b/src/get-started/quickstart-java.md
index a97a61f..37f6d1d 100644
--- a/src/get-started/quickstart-java.md
+++ b/src/get-started/quickstart-java.md
@@ -105,8 +105,9 @@ $ mvn compile exec:java 
-Dexec.mainClass=org.apache.beam.examples.WordCount \
 {:.runner-dataflow}
 ```
 $ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
--Dexec.args="--runner=DataflowRunner 
--gcpTempLocation=gs:///tmp \
- --inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts" \
+ -Dexec.args="--runner=DataflowRunner --project= \
+  --gcpTempLocation=gs:///tmp \
+  --inputFile=gs://apache-beam-samples/shakespeare/* 
--output=gs:///counts" \
  -Pdataflow-runner
 ```
 

[2/3] beam-site git commit: Regenerate website

[altay]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/67180c86
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/67180c86
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/67180c86

Branch: refs/heads/asf-site
Commit: 67180c866a193e3d16a486f88e13e7eeacec3de4
Parents: e296472
Author: Ahmet Altay 
Authored: Thu Apr 13 16:34:06 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 16:34:06 2017 -0700

--
 content/get-started/quickstart-java/index.html | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/67180c86/content/get-started/quickstart-java/index.html
--
diff --git a/content/get-started/quickstart-java/index.html 
b/content/get-started/quickstart-java/index.html
index 51f5e4c..6994a1d 100644
--- a/content/get-started/quickstart-java/index.html
+++ b/content/get-started/quickstart-java/index.html
@@ -253,8 +253,9 @@ You can monitor the running job by visiting the Flink 
dashboard at http://

[3/3] beam-site git commit: This closes #211

[altay]

This closes #211


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/e6d68ba8
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/e6d68ba8
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/e6d68ba8

Branch: refs/heads/asf-site
Commit: e6d68ba899c91e402368dd8d6a771583e0db23e4
Parents: ce71499 67180c8
Author: Ahmet Altay 
Authored: Thu Apr 13 16:34:07 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 16:34:07 2017 -0700

--
 content/get-started/quickstart-java/index.html | 5 +++--
 src/get-started/quickstart-java.md | 5 +++--
 2 files changed, 6 insertions(+), 4 deletions(-)
--

[GitHub] beam-site pull request #211: Update quickstart-java.md

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/211


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1493) runners/core-java should be a pre-execution and an execution-time module

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-1493.
---
   Resolution: Fixed
 Assignee: Thomas Groh
Fix Version/s: Not applicable

runners/core-construction-java is now a module.

> runners/core-java should be a pre-execution and an execution-time module
> 
>
> Key: BEAM-1493
> URL: https://issues.apache.org/jira/browse/BEAM-1493
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
> Fix For: Not applicable
>
>
> This permits a runner to use an internal version of runners-core, but have 
> utilities that interact with the Pipeline within the runner shim.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #3296

[Apache Jenkins Server]

See 


--
[...truncated 234.94 KB...]
 x [deleted] (none) -> origin/pr/942/head
 x [deleted] (none) -> origin/pr/942/merge
 x [deleted] (none) -> origin/pr/943/head
 x [deleted] (none) -> origin/pr/943/merge
 x [deleted] (none) -> origin/pr/944/head
 x [deleted] (none) -> origin/pr/945/head
 x [deleted] (none) -> origin/pr/945/merge
 x [deleted] (none) -> origin/pr/946/head
 x [deleted] (none) -> origin/pr/946/merge
 x [deleted] (none) -> origin/pr/947/head
 x [deleted] (none) -> origin/pr/947/merge
 x [deleted] (none) -> origin/pr/948/head
 x [deleted] (none) -> origin/pr/948/merge
 x [deleted] (none) -> origin/pr/949/head
 x [deleted] (none) -> origin/pr/949/merge
 x [deleted] (none) -> origin/pr/95/head
 x [deleted] (none) -> origin/pr/95/merge
 x [deleted] (none) -> origin/pr/950/head
 x [deleted] (none) -> origin/pr/951/head
 x [deleted] (none) -> origin/pr/951/merge
 x [deleted] (none) -> origin/pr/952/head
 x [deleted] (none) -> origin/pr/952/merge
 x [deleted] (none) -> origin/pr/953/head
 x [deleted] (none) -> origin/pr/954/head
 x [deleted] (none) -> origin/pr/954/merge
 x [deleted] (none) -> origin/pr/955/head
 x [deleted] (none) -> origin/pr/955/merge
 x [deleted] (none) -> origin/pr/956/head
 x [deleted] (none) -> origin/pr/957/head
 x [deleted] (none) -> origin/pr/958/head
 x [deleted] (none) -> origin/pr/959/head
 x [deleted] (none) -> origin/pr/959/merge
 x [deleted] (none) -> origin/pr/96/head
 x [deleted] (none) -> origin/pr/96/merge
 x [deleted] (none) -> origin/pr/960/head
 x [deleted] (none) -> origin/pr/960/merge
 x [deleted] (none) -> origin/pr/961/head
 x [deleted] (none) -> origin/pr/962/head
 x [deleted] (none) -> origin/pr/962/merge
 x [deleted] (none) -> origin/pr/963/head
 x [deleted] (none) -> origin/pr/963/merge
 x [deleted] (none) -> origin/pr/964/head
 x [deleted] (none) -> origin/pr/965/head
 x [deleted] (none) -> origin/pr/965/merge
 x [deleted] (none) -> origin/pr/966/head
 x [deleted] (none) -> origin/pr/967/head
 x [deleted] (none) -> origin/pr/967/merge
 x [deleted] (none) -> origin/pr/968/head
 x [deleted] (none) -> origin/pr/968/merge
 x [deleted] (none) -> origin/pr/969/head
 x [deleted] (none) -> origin/pr/969/merge
 x [deleted] (none) -> origin/pr/97/head
 x [deleted] (none) -> origin/pr/97/merge
 x [deleted] (none) -> origin/pr/970/head
 x [deleted] (none) -> origin/pr/970/merge
 x [deleted] (none) -> origin/pr/971/head
 x [deleted] (none) -> origin/pr/971/merge
 x [deleted] (none) -> origin/pr/972/head
 x [deleted] (none) -> origin/pr/973/head
 x [deleted] (none) -> origin/pr/974/head
 x [deleted] (none) -> origin/pr/974/merge
 x [deleted] (none) -> origin/pr/975/head
 x [deleted] (none) -> origin/pr/975/merge
 x [deleted] (none) -> origin/pr/976/head
 x [deleted] (none) -> origin/pr/976/merge
 x [deleted] (none) -> origin/pr/977/head
 x [deleted] (none) -> origin/pr/977/merge
 x [deleted] (none) -> origin/pr/978/head
 x [deleted] (none) -> origin/pr/978/merge
 x [deleted] (none) -> origin/pr/979/head
 x [deleted] (none) -> origin/pr/979/merge
 x [deleted] (none) -> origin/pr/98/head
 x [deleted] (none) -> origin/pr/980/head
 x [deleted] (none) -> origin/pr/980/merge
 x [deleted] (none) -> origin/pr/981/head
 x [deleted] (none) -> origin/pr/982/head
 x [deleted] (none) -> origin/pr/982/merge
 x [deleted] (none) -> origin/pr/983/head
 x [deleted] (none) -> origin/pr/983/merge
 x [deleted] (none) -> origin/pr/984/head
 x [deleted] (none) -> origin/pr/984/merge
 x [deleted] (none) -> origin/pr/985/head
 x [deleted] (none) -> origin/pr/985/merge
 x [deleted] (none) -> origin/pr/986/head
 x [deleted] (none) -> origin/pr/986/merge
 x [deleted] (none) -> origin/pr/987/head
 x [deleted] (none) -> origin/pr/988/head
 x [deleted] (none) -> origin/pr/988/merge
 x [deleted] (none) -> origin/pr/989/hea

[jira] [Commented] (BEAM-1960) Hadoop InputFormat - Add Kubernetes large and small cluster Scripts for Cassandra and Elasticsearch tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968389#comment-15968389
 ] 

ASF GitHub Bot commented on BEAM-1960:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2423


> Hadoop InputFormat - Add Kubernetes large and small cluster Scripts for 
> Cassandra and Elasticsearch tests
> -
>
> Key: BEAM-1960
> URL: https://issues.apache.org/jira/browse/BEAM-1960
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Dipti Kulkarni
>Assignee: Dipti Kulkarni
>
> Add Kubernetes cluster scripts for Elasticsearch and Cassandra for both large 
> and small clusters to run Elasticsearch and Cassandra ITs and Performance 
> tests.
> Create separate clusters for small - integration test and large- performance 
> test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2423: [BEAM-1960] Add Kubernetes scripts for clusters for...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2423


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Merge PR#2423: Add Kubernetes scripts for clusters for Performance and Integration tests of Cassandra and ES for Hadoop Input Format IO

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 8761d8642 -> 3fb75d3c2


Merge PR#2423: Add Kubernetes scripts for clusters for Performance and 
Integration tests of Cassandra and ES for Hadoop Input Format IO

Large IT Cluster and files added, show_health added, start-up.sh changed

ES change for Virtual memory addition to env ES small cluster, Addressed 
Stephens comments on PR dated 7th March

ES cluster changes to introduce the load balancer service in both small and 
large clusters

Removed resource limits from statefulset yaml from Cassandra large cluster for 
the memory issue fix, changed data load script to add threads param to the ycsb 
command, comments changed

Fixes in Cassandra clusters

Removed allow filtering from test with query in Cassandra IT

Added "Create index" in data load script on field on which query happens

index creation command rectified

show_health cassandra small cluster was giving error pods not found, rectified

Small Cassandra cluster changes-  Added external service

Added delete service for cassandra-service-for-local-dev.xml in teardown.sh file

Added create service for cassandra-service-for-local-dev.xml in start-up.sh file

Moved kubernetes clusters to test-infra/kubernetes

Changed hashes in the ITs as per latest HashingFn used

Added nodeport, hashcode comment changes, mem config changes

data-load-setup script added back


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/59d91fa2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/59d91fa2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/59d91fa2

Branch: refs/heads/master
Commit: 59d91fa20ff6946e17d70a0751885552fed4d436
Parents: 8761d86
Author: Dipti Kulkarni 
Authored: Tue Apr 11 11:37:02 2017 +0530
Committer: Dan Halperin 
Committed: Thu Apr 13 16:29:38 2017 -0700

--
 .../cassandra-service-for-local-dev.yaml|  28 ++
 .../cassandra-svc-statefulset.yaml  | 114 
 .../LargeITCluster/cassandra-svc-temp.yaml  |  74 +
 .../cassandra/LargeITCluster/data-load.sh   | 122 
 .../cassandra/LargeITCluster/show_health.sh |  47 
 .../cassandra/LargeITCluster/start-up.sh|  22 ++
 .../cassandra/LargeITCluster/teardown.sh|  25 ++
 .../cassandra-service-for-local-dev.yaml|  30 ++
 .../SmallITCluster/cassandra-svc-rc.yaml|  16 +-
 .../cassandra/SmallITCluster/data-load.sh   |  86 ++
 .../cassandra/SmallITCluster/show_health.sh |  47 
 .../cassandra/SmallITCluster/start-up.sh|   2 +
 .../cassandra/SmallITCluster/teardown.sh|   1 +
 .test-infra/kubernetes/cassandra/data-load.sh   |  67 -
 .../elasticsearch-service-for-local-dev.yaml|  33 +++
 .../es-services-deployments.yaml| 258 +
 .../LargeProductionCluster/es-services.yaml | 277 ---
 .../LargeProductionCluster/start-up.sh  |   3 +-
 .../LargeProductionCluster/teardown.sh  |   3 +-
 .../elasticsearch-service-for-local-dev.yaml|  34 +++
 .../SmallITCluster/elasticsearch-svc-rc.yaml|  16 +-
 .../elasticsearch/SmallITCluster/start-up.sh|   1 +
 .../elasticsearch/SmallITCluster/teardown.sh|   1 +
 .../kubernetes/elasticsearch/data-load.sh   |   6 +-
 .../kubernetes/elasticsearch/es_test_data.py|   6 +-
 .../kubernetes/elasticsearch/show-health.sh |  12 +-
 .../integration/tests/HIFIOCassandraIT.java |   6 +-
 .../integration/tests/HIFIOElasticIT.java   |   4 +-
 28 files changed, 965 insertions(+), 376 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/59d91fa2/.test-infra/kubernetes/cassandra/LargeITCluster/cassandra-service-for-local-dev.yaml
--
diff --git 
a/.test-infra/kubernetes/cassandra/LargeITCluster/cassandra-service-for-local-dev.yaml
 
b/.test-infra/kubernetes/cassandra/LargeITCluster/cassandra-service-for-local-dev.yaml
new file mode 100644
index 000..dd0da93
--- /dev/null
+++ 
b/.test-infra/kubernetes/cassandra/LargeITCluster/cassandra-service-for-local-dev.yaml
@@ -0,0 +1,28 @@
+#Licensed to the Apache Software Foundation (ASF) under one or more
+#contributor license agreements.  See the NOTICE file distributed with
+#this work for additional information regarding copyright ownership.
+#The ASF licenses this file to You under the Apache License, Version 2.0
+#(the "License"); you may not use this file except in compliance with
+#the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+#Unless required by applicable law or agreed to in writing, software
+#distributed under the License is distributed on an "AS IS" BASIS,
+#WITHOUT WARRANTIES OR CONDITIO

[2/2] beam git commit: This closes #2423

[dhalperi]

This closes #2423


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3fb75d3c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3fb75d3c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3fb75d3c

Branch: refs/heads/master
Commit: 3fb75d3c22a7988ba9ab05af986aa3594bc7b7f8
Parents: 8761d86 59d91fa
Author: Dan Halperin 
Authored: Thu Apr 13 16:30:10 2017 -0700
Committer: Dan Halperin 
Committed: Thu Apr 13 16:30:10 2017 -0700

--
 .../cassandra-service-for-local-dev.yaml|  28 ++
 .../cassandra-svc-statefulset.yaml  | 114 
 .../LargeITCluster/cassandra-svc-temp.yaml  |  74 +
 .../cassandra/LargeITCluster/data-load.sh   | 122 
 .../cassandra/LargeITCluster/show_health.sh |  47 
 .../cassandra/LargeITCluster/start-up.sh|  22 ++
 .../cassandra/LargeITCluster/teardown.sh|  25 ++
 .../cassandra-service-for-local-dev.yaml|  30 ++
 .../SmallITCluster/cassandra-svc-rc.yaml|  16 +-
 .../cassandra/SmallITCluster/data-load.sh   |  86 ++
 .../cassandra/SmallITCluster/show_health.sh |  47 
 .../cassandra/SmallITCluster/start-up.sh|   2 +
 .../cassandra/SmallITCluster/teardown.sh|   1 +
 .test-infra/kubernetes/cassandra/data-load.sh   |  67 -
 .../elasticsearch-service-for-local-dev.yaml|  33 +++
 .../es-services-deployments.yaml| 258 +
 .../LargeProductionCluster/es-services.yaml | 277 ---
 .../LargeProductionCluster/start-up.sh  |   3 +-
 .../LargeProductionCluster/teardown.sh  |   3 +-
 .../elasticsearch-service-for-local-dev.yaml|  34 +++
 .../SmallITCluster/elasticsearch-svc-rc.yaml|  16 +-
 .../elasticsearch/SmallITCluster/start-up.sh|   1 +
 .../elasticsearch/SmallITCluster/teardown.sh|   1 +
 .../kubernetes/elasticsearch/data-load.sh   |   6 +-
 .../kubernetes/elasticsearch/es_test_data.py|   6 +-
 .../kubernetes/elasticsearch/show-health.sh |  12 +-
 .../integration/tests/HIFIOCassandraIT.java |   6 +-
 .../integration/tests/HIFIOElasticIT.java   |   4 +-
 28 files changed, 965 insertions(+), 376 deletions(-)
--

[GitHub] beam-site pull request #211: Update quickstart-java.md

[tvalentyn]

GitHub user tvalentyn opened a pull request:

https://github.com/apache/beam-site/pull/211

Update quickstart-java.md

Add --project pipeline option in command line for WordCount example on 
Dataflow Runner.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tvalentyn/beam-site patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/211.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #211


commit e2964720852184f2bdce1469a4c9f10f23c03b74
Author: tvalentyn 
Date:   2017-04-13T23:22:38Z

Update quickstart-java.md

Add --project pipeline option in command line for WordCount example on 
Dataflow Runner.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2532: [BEAM-1964] Fix lint issues for linter upgrade - pa...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2532

[BEAM-1964] Fix lint issues for linter upgrade - part 1/?

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1964-upgrade-pylint-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2532.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2532


commit 78a2af166349460e2e3f1cf484a8c865e6c2e2ce
Author: Sourabh Bajaj 
Date:   2017-04-13T23:18:32Z

[BEAM-1964] Fix lint issues for linter upgrade




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1964) Upgrade pylint to 1.7.0

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968382#comment-15968382
 ] 

ASF GitHub Bot commented on BEAM-1964:
--

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2532

[BEAM-1964] Fix lint issues for linter upgrade - part 1/?

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.pdf).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-1964-upgrade-pylint-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2532.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2532


commit 78a2af166349460e2e3f1cf484a8c865e6c2e2ce
Author: Sourabh Bajaj 
Date:   2017-04-13T23:18:32Z

[BEAM-1964] Fix lint issues for linter upgrade




> Upgrade pylint to 1.7.0
> ---
>
> Key: BEAM-1964
> URL: https://issues.apache.org/jira/browse/BEAM-1964
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Aviem Zur
>Assignee: Ahmet Altay
>
> Pre-commit tests seem to all be failing on pylint
> For example: 
> https://builds.apache.org/view/Beam/job/beam_PreCommit_Java_MavenInstall/9493/consoleFull



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1870) ByteKey / ByteKeyRangeTracker should not use ByteString on public API surface

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968363#comment-15968363
 ] 

ASF GitHub Bot commented on BEAM-1870:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2531

[BEAM-1870] ByteKey: remove ByteString from public API, replace with 
ByteBuffer

* BigtableIO: use ByteBuffer not ByteString
* HBaseIO: replace ByteString with byte[]

R: @iemejia @lukecwik 

Ismaël for general thoughts and HBase
Luke to sanity check use of ByteBuffers.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam bytekey-bigtable-hbase

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2531.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2531


commit 9922e38621b4ca21e760edc7d143a7114509a4d9
Author: Dan Halperin 
Date:   2017-04-13T22:27:57Z

ByteKey: remove ByteString from public API, replace with ByteBuffer

* BigtableIO: use ByteBuffer not ByteString
* HBaseIO: replace ByteString with byte[]




> ByteKey / ByteKeyRangeTracker should not use ByteString on public API surface
> -
>
> Key: BEAM-1870
> URL: https://issues.apache.org/jira/browse/BEAM-1870
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
>Assignee: Ismaël Mejía
> Fix For: First stable release
>
>
> We don't want these Google Protocol Buffer dependencies on the public API. We 
> should replace the use of {{ByteString}} with something in the core Java 
> libraries.
> What's the open source standard here? I guess Avro uses {{ByteBuffer}} for 
> wrapping {{byte[]}} ?
> [~iemejia] -- tentatively assigned to you as you brought this up.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2531: [BEAM-1870] ByteKey: remove ByteString from public ...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2531

[BEAM-1870] ByteKey: remove ByteString from public API, replace with 
ByteBuffer

* BigtableIO: use ByteBuffer not ByteString
* HBaseIO: replace ByteString with byte[]

R: @iemejia @lukecwik 

Ismaël for general thoughts and HBase
Luke to sanity check use of ByteBuffers.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam bytekey-bigtable-hbase

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2531.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2531


commit 9922e38621b4ca21e760edc7d143a7114509a4d9
Author: Dan Halperin 
Date:   2017-04-13T22:27:57Z

ByteKey: remove ByteString from public API, replace with ByteBuffer

* BigtableIO: use ByteBuffer not ByteString
* HBaseIO: replace ByteString with byte[]




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-1068) Service Account Credentials File Specified via Pipeline Option Ignored

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay resolved BEAM-1068.
---
Resolution: Fixed

> Service Account Credentials File Specified via Pipeline Option Ignored
> --
>
> Key: BEAM-1068
> URL: https://issues.apache.org/jira/browse/BEAM-1068
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
> Environment: CentOS Linux release 7.1.1503 (Core)
> Python 2.7.5
>Reporter: Stephen Reichling
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: First stable release
>
>
> When writing a pipeline that authenticates with Google Dataflow APIs using a 
> service account, specifying the path to that service account's credentials 
> file in the {{PipelineOptions}} object passed in to the pipeline does not 
> work, it only works when passed as a command-line flag.
> For example, if I write code like so:
> {code}
> pipelineOptions = options.PipelineOptions()
> gcOptions = pipelineOptions.view_as(options.GoogleCloudOptions)
> gcOptions.service_account_name = 'My Service Account Name'
> gcOptions.service_account_key_file = '/some/path/keyfile.p12'
> pipeline = beam.Pipeline(options=pipelineOptions)
> # ... add stages to the pipeline
> p.run()
> {code}
> and execute it like so:
> {{python ./my_pipeline.py}}
> ...the service account I specify will not be used.
> Only if I were to execute the code like so:
> {{python ./my_pipeline.py --service_account_name 'My Service Account Name' 
> --service_account_key_file /some/path/keyfile.p12}}
> ...does it actually use the service account.
> The problem appears to be rooted in `auth.py` which reconstructs the 
> {{PipelineOptions}} object directly from {{sys.argv}} rather than using the 
> instance passed in to the pipeline: 
> https://github.com/apache/incubator-beam/blob/9ded359daefc6040d61a1f33c77563474fcb09b6/sdks/python/apache_beam/internal/auth.py#L129-L130



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1972) HIFIO jdk module fails enforcer when only java 7 is installed on machine

[Stephen Sisk (JIRA)]

Stephen Sisk created BEAM-1972:
--

 Summary: HIFIO jdk module fails enforcer when only java 7 is 
installed on machine
 Key: BEAM-1972
 URL: https://issues.apache.org/jira/browse/BEAM-1972
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Affects Versions: First stable release
Reporter: Stephen Sisk
Assignee: Davor Bonaci


When running on a machine with only java 7, you cannot do a full compile of the 
project. 

The error message is:
[WARNING] Rule 1: org.apache.maven.plugins.enforcer.RequireJavaVersion failed 
with message:
Detected JDK Version: 1.7.0-121 is not in the allowed range [1.8,).

The failed module is:
[INFO] Apache Beam :: SDKs :: Java :: IO :: Hadoop :: jdk1.8-tests FAILURE [ 
37.549 s]

This is likely due to the enforcer rules at 
https://github.com/apache/beam/blob/master/sdks/java/io/hadoop/jdk1.8-tests/pom.xml#L39

without this exception to the enforcer rules, it throws errors since the 
cassandra dependency in that directory is java 8, so the intent is that 
enforcer should not error out since this module. 

Below are the errors (well, warnings) when run with [1.8,) 
is commented out: 
WARNING] 
/home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithEmbeddedCassandraTest.java:
 
/home/sisk/.m2/repository/org/apache/cassandra/cassandra-all/3.9/cassandra-all-3.9.jar(org/apache/cassandra/service/EmbeddedCassandraService.class):
 major version 52 is newer than 51, the highest major version supported by this 
compiler.
  It is recommended that the compiler be upgraded.
[WARNING] 
/home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithElasticTest.java:
 
/home/sisk/.m2/repository/org/elasticsearch/elasticsearch/5.0.0/elasticsearch-5.0.0.jar(org/elasticsearch/action/admin/indices/create/CreateIndexRequest.class):
 major version 52 is newer than 51, the highest major version supported by this 
compiler.
  It is recommended that the compiler be upgraded.
[WARNING] 
/home/sisk/beam/sdks/java/io/hadoop/jdk1.8-tests/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HIFIOWithElasticTest.java:
 
/home/sisk/.m2/repository/org/elasticsearch/elasticsearch/5.0.0/elasticsearch-5.0.0.jar(org/elasticsearch/action/admin/indices/delete/DeleteIndexRequest.class):
 major version 52 is newer than 51, the highest major version supported by this 
compiler.
  It is recommended that the compiler be upgraded.

...and a couple more of these errors.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-301) Add a Beam SQL DSL

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968351#comment-15968351
 ] 

ASF GitHub Bot commented on BEAM-301:
-

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2530

 [BEAM-301] Add a Beam SQL DSL

1. add `checkstyle` to `dsls` module, and update as indicated;
2. correct the package name from `org.beam.dsls.sql` to 
`org.apache.beam.dsls.sql`;

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam rename_package

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2530


commit 529bc9d9375ae162d70de274975c547a878053a3
Author: mingmxu 
Date:   2017-04-13T21:46:28Z

checkstyle and rename package

correct package from org.beam.dsls.sql to org.apache.beam.dsls.sql

update with checkstyle




> Add a Beam SQL DSL
> --
>
> Key: BEAM-301
> URL: https://issues.apache.org/jira/browse/BEAM-301
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-ideas
>Reporter: Jean-Baptiste Onofré
>Assignee: Xu Mingmin
>
> The SQL DSL helps developers to build a Beam pipeline from SQL statement in 
> String directly. 
> In Phase I, it starts to support INSERT/SELECT queries with FILTERs, one 
> example SQL as below:
> {code}
> INSERT INTO `SUB_USEREVENT` (`SITEID`, `PAGEID`, `PAGENAME`, `EVENTTIMESTAMP`)
> (SELECT STREAM `USEREVENT`.`SITEID`, `USEREVENT`.`PAGEID`, 
> `USEREVENT`.`PAGENAME`, `USEREVENT`.`EVENTTIMESTAMP`
> FROM `USEREVENT` AS `USEREVENT`
> WHERE `USEREVENT`.`SITEID` > 10)
> {code}
> A design doc is available at 
> https://docs.google.com/document/d/1Uc5xYTpO9qsLXtT38OfuoqSLimH_0a1Bz5BsCROMzCU/edit?usp=sharing.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2530: [BEAM-301] Add a Beam SQL DSL

[XuMingmin]

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/2530

 [BEAM-301] Add a Beam SQL DSL

1. add `checkstyle` to `dsls` module, and update as indicated;
2. correct the package name from `org.beam.dsls.sql` to 
`org.apache.beam.dsls.sql`;

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam rename_package

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2530


commit 529bc9d9375ae162d70de274975c547a878053a3
Author: mingmxu 
Date:   2017-04-13T21:46:28Z

checkstyle and rename package

correct package from org.beam.dsls.sql to org.apache.beam.dsls.sql

update with checkstyle




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968346#comment-15968346
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2529


> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2529: [BEAM-1871] Remove Triggers.java from SDK entirely

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2529


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Remove Triggers.java from SDK entirely

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 75d7b273c -> 8761d8642


Remove Triggers.java from SDK entirely

Update DataflowRunner with support for this removal


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6832668a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6832668a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6832668a

Branch: refs/heads/master
Commit: 6832668ac6888d3e8d56819a7a73ff05798c7082
Parents: 75d7b27
Author: Dan Halperin 
Authored: Thu Apr 13 14:52:25 2017 -0700
Committer: Dan Halperin 
Committed: Thu Apr 13 15:45:21 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../beam/sdk/transforms/windowing/Triggers.java | 323 ---
 2 files changed, 1 insertion(+), 324 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/6832668a/runners/google-cloud-dataflow-java/pom.xml
--
diff --git a/runners/google-cloud-dataflow-java/pom.xml 
b/runners/google-cloud-dataflow-java/pom.xml
index 96eced8..c33faf7 100644
--- a/runners/google-cloud-dataflow-java/pom.xml
+++ b/runners/google-cloud-dataflow-java/pom.xml
@@ -33,7 +33,7 @@
   jar
 
   
-
beam-master-20170411
+
beam-master-20170413
 
1
 
6
   

http://git-wip-us.apache.org/repos/asf/beam/blob/6832668a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Triggers.java
--
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Triggers.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Triggers.java
deleted file mode 100644
index 47f05e4..000
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/Triggers.java
+++ /dev/null
@@ -1,323 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.transforms.windowing;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.collect.Lists;
-import java.io.Serializable;
-import java.lang.reflect.InvocationTargetException;
-import java.lang.reflect.Method;
-import java.util.List;
-import org.apache.beam.sdk.annotations.Experimental;
-import org.apache.beam.sdk.common.runner.v1.RunnerApi;
-import 
org.apache.beam.sdk.transforms.windowing.AfterWatermark.AfterWatermarkEarlyAndLate;
-import org.apache.beam.sdk.transforms.windowing.Never.NeverTrigger;
-import org.apache.beam.sdk.transforms.windowing.Trigger.OnceTrigger;
-import org.apache.beam.sdk.util.ReshuffleTrigger;
-import org.apache.beam.sdk.util.TimeDomain;
-import org.joda.time.Duration;
-import org.joda.time.Instant;
-
-/** Utilities for working with {@link Triggers Triggers}. */
-@Experimental(Experimental.Kind.TRIGGER)
-@Deprecated
-public class Triggers implements Serializable {
-
-  @VisibleForTesting static final ProtoConverter CONVERTER = new 
ProtoConverter();
-
-  public static RunnerApi.Trigger toProto(Trigger trigger) {
-return CONVERTER.convertTrigger(trigger);
-  }
-
-  @VisibleForTesting
-  static class ProtoConverter {
-
-public RunnerApi.Trigger convertTrigger(Trigger trigger) {
-  Method evaluationMethod = getEvaluationMethod(trigger.getClass());
-  return tryConvert(evaluationMethod, trigger);
-}
-
-private RunnerApi.Trigger tryConvert(Method evaluationMethod, Trigger 
trigger) {
-  try {
-return (RunnerApi.Trigger) evaluationMethod.invoke(this, trigger);
-  } catch (InvocationTargetException exc) {
-if (exc.getCause() instanceof RuntimeException) {
-  throw (RuntimeException) exc.getCause();
-} else {
-  throw new RuntimeException(exc.getCause());
-}
-  } catch (IllegalAccessException exc) {
-throw new IllegalStateException(
-String.format("Internal error: could not invoke %s", 
evaluationMethod));
-  }
-}
-
-private Method getEvaluatio

[2/2] beam git commit: This closes #2529

[dhalperi]

This closes #2529


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8761d864
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8761d864
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8761d864

Branch: refs/heads/master
Commit: 8761d8642e20a7504504d479b8ffd70c55f1b43d
Parents: 75d7b27 6832668
Author: Dan Halperin 
Authored: Thu Apr 13 15:45:27 2017 -0700
Committer: Dan Halperin 
Committed: Thu Apr 13 15:45:27 2017 -0700

--
 runners/google-cloud-dataflow-java/pom.xml  |   2 +-
 .../beam/sdk/transforms/windowing/Triggers.java | 323 ---
 2 files changed, 1 insertion(+), 324 deletions(-)
--

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex #1084

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #2828

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-778) Make filesystem._CompressedFile seekable.

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968309#comment-15968309
 ] 

Sourabh Bajaj commented on BEAM-778:


This can be closed now.

> Make filesystem._CompressedFile seekable.
> -
>
> Key: BEAM-778
> URL: https://issues.apache.org/jira/browse/BEAM-778
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Tibor Kiss
> Fix For: Not applicable
>
>
> We have a TODO to make filesystem._CompressedFile seekable.
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L692
> Without this, compressed file objects produce for FileBasedSource 
> implementations may not be able to use libraries that utilize methods seek() 
> and tell().
> For example tarfile.open().



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1068) Service Account Credentials File Specified via Pipeline Option Ignored

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968307#comment-15968307
 ] 

Sourabh Bajaj commented on BEAM-1068:
-

This can be closed now.

> Service Account Credentials File Specified via Pipeline Option Ignored
> --
>
> Key: BEAM-1068
> URL: https://issues.apache.org/jira/browse/BEAM-1068
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
> Environment: CentOS Linux release 7.1.1503 (Core)
> Python 2.7.5
>Reporter: Stephen Reichling
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: First stable release
>
>
> When writing a pipeline that authenticates with Google Dataflow APIs using a 
> service account, specifying the path to that service account's credentials 
> file in the {{PipelineOptions}} object passed in to the pipeline does not 
> work, it only works when passed as a command-line flag.
> For example, if I write code like so:
> {code}
> pipelineOptions = options.PipelineOptions()
> gcOptions = pipelineOptions.view_as(options.GoogleCloudOptions)
> gcOptions.service_account_name = 'My Service Account Name'
> gcOptions.service_account_key_file = '/some/path/keyfile.p12'
> pipeline = beam.Pipeline(options=pipelineOptions)
> # ... add stages to the pipeline
> p.run()
> {code}
> and execute it like so:
> {{python ./my_pipeline.py}}
> ...the service account I specify will not be used.
> Only if I were to execute the code like so:
> {{python ./my_pipeline.py --service_account_name 'My Service Account Name' 
> --service_account_key_file /some/path/keyfile.p12}}
> ...does it actually use the service account.
> The problem appears to be rooted in `auth.py` which reconstructs the 
> {{PipelineOptions}} object directly from {{sys.argv}} rather than using the 
> instance passed in to the pipeline: 
> https://github.com/apache/incubator-beam/blob/9ded359daefc6040d61a1f33c77563474fcb09b6/sdks/python/apache_beam/internal/auth.py#L129-L130



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1101) Remove inconsistencies in Python PipelineOptions

[Sourabh Bajaj (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968305#comment-15968305
 ] 

Sourabh Bajaj commented on BEAM-1101:
-

This can be closed now.

> Remove inconsistencies in Python PipelineOptions
> 
>
> Key: BEAM-1101
> URL: https://issues.apache.org/jira/browse/BEAM-1101
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> Some options have been removed from Java, and some have different default 
> behavior in Java. Gotta make this consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (BEAM-1708) Better error messages when GCP features are not installed

[Sourabh Bajaj (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sourabh Bajaj resolved BEAM-1708.
-
   Resolution: Fixed
Fix Version/s: First stable release

> Better error messages when GCP features are not installed 
> --
>
> Key: BEAM-1708
> URL: https://issues.apache.org/jira/browse/BEAM-1708
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1708) Better error messages when GCP features are not installed

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968281#comment-15968281
 ] 

ASF GitHub Bot commented on BEAM-1708:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2485


> Better error messages when GCP features are not installed 
> --
>
> Key: BEAM-1708
> URL: https://issues.apache.org/jira/browse/BEAM-1708
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Sourabh Bajaj
>Assignee: Sourabh Bajaj
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2485: [BEAM-1708] Improve error message when GCP not inst...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2485


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2529: [BEAM-1871] Remove Triggers.java from SDK entirely

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2529

[BEAM-1871] Remove Triggers.java from SDK entirely

Update DataflowRunner with support for this removal

This finishes #2513

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam update-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2529


commit 0056d57612c92c0ca85516a5d8ae34db52656089
Author: Dan Halperin 
Date:   2017-04-13T21:52:25Z

Remove Triggers.java from SDK entirely

Update DataflowRunner with support for this removal




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2485

[altay]

This closes #2485


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/75d7b273
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/75d7b273
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/75d7b273

Branch: refs/heads/master
Commit: 75d7b273c7436a40b2aef66a296599c1f0c68db8
Parents: 69343a6 233a9bd
Author: Ahmet Altay 
Authored: Thu Apr 13 14:54:53 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 14:54:53 2017 -0700

--
 sdks/python/apache_beam/io/filesystems_util.py  |  7 ++-
 sdks/python/apache_beam/io/gcp/bigquery.py  | 19 +++
 .../io/gcp/datastore/v1/datastoreio.py  | 20 
 .../runners/dataflow/dataflow_runner.py |  9 +++--
 sdks/python/apache_beam/runners/runner.py   | 10 +-
 5 files changed, 61 insertions(+), 4 deletions(-)
--

[1/2] beam git commit: [BEAM-1708] Improve error message when GCP not installed

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 69343a609 -> 75d7b273c


[BEAM-1708] Improve error message when GCP not installed


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/233a9bd3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/233a9bd3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/233a9bd3

Branch: refs/heads/master
Commit: 233a9bd3c503182dee004ac52acf37ace4eeac12
Parents: 69343a6
Author: Sourabh Bajaj 
Authored: Mon Apr 10 14:45:01 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 14:54:37 2017 -0700

--
 sdks/python/apache_beam/io/filesystems_util.py  |  7 ++-
 sdks/python/apache_beam/io/gcp/bigquery.py  | 19 +++
 .../io/gcp/datastore/v1/datastoreio.py  | 20 
 .../runners/dataflow/dataflow_runner.py |  9 +++--
 sdks/python/apache_beam/runners/runner.py   | 10 +-
 5 files changed, 61 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/233a9bd3/sdks/python/apache_beam/io/filesystems_util.py
--
diff --git a/sdks/python/apache_beam/io/filesystems_util.py 
b/sdks/python/apache_beam/io/filesystems_util.py
index 47c2361..6d21298 100644
--- a/sdks/python/apache_beam/io/filesystems_util.py
+++ b/sdks/python/apache_beam/io/filesystems_util.py
@@ -25,7 +25,12 @@ def get_filesystem(path):
   provided in the input.
   """
   if path.startswith('gs://'):
-from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
+try:
+  from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
+except ImportError:
+  raise ImportError(
+  'Google Cloud Platform IO not available, '
+  'please install apache_beam[gcp]')
 return GCSFileSystem()
   else:
 return LocalFileSystem()

http://git-wip-us.apache.org/repos/asf/beam/blob/233a9bd3/sdks/python/apache_beam/io/gcp/bigquery.py
--
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py 
b/sdks/python/apache_beam/io/gcp/bigquery.py
index 788c069..9a8174a 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -367,6 +367,15 @@ class BigQuerySource(dataflow_io.NativeSource):
   (3) both a table and a query is specified.
 """
 
+# Import here to avoid adding the dependency for local running scenarios.
+try:
+  # pylint: disable=wrong-import-order, wrong-import-position
+  from apitools.base.py import *
+except ImportError:
+  raise ImportError(
+  'Google Cloud IO not available, '
+  'please install apache_beam[gcp]')
+
 if table is not None and query is not None:
   raise ValueError('Both a BigQuery table and a query were specified.'
' Please specify only one of these.')
@@ -467,6 +476,16 @@ class BigQuerySink(dataflow_io.NativeSink):
   ValueError: if the table reference as a string does not match the 
expected
   format.
 """
+
+# Import here to avoid adding the dependency for local running scenarios.
+try:
+  # pylint: disable=wrong-import-order, wrong-import-position
+  from apitools.base.py import *
+except ImportError:
+  raise ImportError(
+  'Google Cloud IO not available, '
+  'please install apache_beam[gcp]')
+
 self.table_reference = _parse_table_reference(table, dataset, project)
 # Transform the table schema into a bigquery.TableSchema instance.
 if isinstance(schema, basestring):

http://git-wip-us.apache.org/repos/asf/beam/blob/233a9bd3/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
--
diff --git a/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py 
b/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
index af0c72b..e8ca05d 100644
--- a/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
+++ b/sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
@@ -92,6 +92,16 @@ class ReadFromDatastore(PTransform):
   namespace: An optional namespace.
   num_splits: Number of splits for the query.
 """
+
+# Import here to avoid adding the dependency for local running scenarios.
+try:
+  # pylint: disable=wrong-import-order, wrong-import-position
+  from apitools.base.py import *
+except ImportError:
+  raise ImportError(
+  'Google Cloud IO not available, '
+  'please install apache_beam[gcp]')
+
 logging.warning('datastoreio read transform is experimental.')
 super(ReadFromDatastore, self).__init__()
 
@@ -368,6 +378,16 @@ class _Mutate(PTransform):
 class WriteToDatastore(_Mutate):
   """A ``PTransform

[jira] [Commented] (BEAM-1871) Thin Java SDK Core

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968278#comment-15968278
 ] 

ASF GitHub Bot commented on BEAM-1871:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/2529

[BEAM-1871] Remove Triggers.java from SDK entirely

Update DataflowRunner with support for this removal

This finishes #2513

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam update-worker

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2529.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2529


commit 0056d57612c92c0ca85516a5d8ae34db52656089
Author: Dan Halperin 
Date:   2017-04-13T21:52:25Z

Remove Triggers.java from SDK entirely

Update DataflowRunner with support for this removal




> Thin Java SDK Core
> --
>
> Key: BEAM-1871
> URL: https://issues.apache.org/jira/browse/BEAM-1871
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Daniel Halperin
> Fix For: First stable release
>
>
> Before first stable release we need to thin out {{sdk-java-core}} module. 
> Some candidates for removal, but not a non-exhaustive list:
> {{sdk/io}}
> * anything BigQuery related
> * anything PubSub related
> * everything Protobuf related
> * TFRecordIO
> * XMLSink
> {{sdk/util}}
> * Everything GCS related
> * Everything Backoff related
> * Everything Google API related: ResponseInterceptors, RetryHttpBackoff, etc.
> * Everything CloudObject-related
> * Pubsub stuff
> {{sdk/coders}}
> * JAXBCoder
> * TableRowJsoNCoder



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #1083

[Apache Jenkins Server]

See 


--
[...truncated 791.37 KB...]
 x [deleted] (none) -> origin/pr/901/merge
 x [deleted] (none) -> origin/pr/902/head
 x [deleted] (none) -> origin/pr/902/merge
 x [deleted] (none) -> origin/pr/903/head
 x [deleted] (none) -> origin/pr/903/merge
 x [deleted] (none) -> origin/pr/904/head
 x [deleted] (none) -> origin/pr/904/merge
 x [deleted] (none) -> origin/pr/905/head
 x [deleted] (none) -> origin/pr/905/merge
 x [deleted] (none) -> origin/pr/906/head
 x [deleted] (none) -> origin/pr/906/merge
 x [deleted] (none) -> origin/pr/907/head
 x [deleted] (none) -> origin/pr/907/merge
 x [deleted] (none) -> origin/pr/908/head
 x [deleted] (none) -> origin/pr/909/head
 x [deleted] (none) -> origin/pr/909/merge
 x [deleted] (none) -> origin/pr/91/head
 x [deleted] (none) -> origin/pr/91/merge
 x [deleted] (none) -> origin/pr/910/head
 x [deleted] (none) -> origin/pr/911/head
 x [deleted] (none) -> origin/pr/911/merge
 x [deleted] (none) -> origin/pr/912/head
 x [deleted] (none) -> origin/pr/912/merge
 x [deleted] (none) -> origin/pr/913/head
 x [deleted] (none) -> origin/pr/913/merge
 x [deleted] (none) -> origin/pr/914/head
 x [deleted] (none) -> origin/pr/914/merge
 x [deleted] (none) -> origin/pr/915/head
 x [deleted] (none) -> origin/pr/915/merge
 x [deleted] (none) -> origin/pr/916/head
 x [deleted] (none) -> origin/pr/916/merge
 x [deleted] (none) -> origin/pr/917/head
 x [deleted] (none) -> origin/pr/917/merge
 x [deleted] (none) -> origin/pr/918/head
 x [deleted] (none) -> origin/pr/918/merge
 x [deleted] (none) -> origin/pr/919/head
 x [deleted] (none) -> origin/pr/919/merge
 x [deleted] (none) -> origin/pr/92/head
 x [deleted] (none) -> origin/pr/92/merge
 x [deleted] (none) -> origin/pr/920/head
 x [deleted] (none) -> origin/pr/920/merge
 x [deleted] (none) -> origin/pr/921/head
 x [deleted] (none) -> origin/pr/921/merge
 x [deleted] (none) -> origin/pr/922/head
 x [deleted] (none) -> origin/pr/922/merge
 x [deleted] (none) -> origin/pr/923/head
 x [deleted] (none) -> origin/pr/924/head
 x [deleted] (none) -> origin/pr/925/head
 x [deleted] (none) -> origin/pr/925/merge
 x [deleted] (none) -> origin/pr/926/head
 x [deleted] (none) -> origin/pr/926/merge
 x [deleted] (none) -> origin/pr/927/head
 x [deleted] (none) -> origin/pr/927/merge
 x [deleted] (none) -> origin/pr/928/head
 x [deleted] (none) -> origin/pr/929/head
 x [deleted] (none) -> origin/pr/93/head
 x [deleted] (none) -> origin/pr/930/head
 x [deleted] (none) -> origin/pr/930/merge
 x [deleted] (none) -> origin/pr/931/head
 x [deleted] (none) -> origin/pr/931/merge
 x [deleted] (none) -> origin/pr/932/head
 x [deleted] (none) -> origin/pr/932/merge
 x [deleted] (none) -> origin/pr/933/head
 x [deleted] (none) -> origin/pr/933/merge
 x [deleted] (none) -> origin/pr/934/head
 x [deleted] (none) -> origin/pr/934/merge
 x [deleted] (none) -> origin/pr/935/head
 x [deleted] (none) -> origin/pr/936/head
 x [deleted] (none) -> origin/pr/936/merge
 x [deleted] (none) -> origin/pr/937/head
 x [deleted] (none) -> origin/pr/937/merge
 x [deleted] (none) -> origin/pr/938/head
 x [deleted] (none) -> origin/pr/939/head
 x [deleted] (none) -> origin/pr/94/head
 x [deleted] (none) -> origin/pr/940/head
 x [deleted] (none) -> origin/pr/940/merge
 x [deleted] (none) -> origin/pr/941/head
 x [deleted] (none) -> origin/pr/941/merge
 x [deleted] (none) -> origin/pr/942/head
 x [deleted] (none) -> origin/pr/942/merge
 x [deleted] (none) -> origin/pr/943/head
 x [deleted] (none) -> origin/pr/943/merge
 x [deleted] (none) -> origin/pr/944/head
 x [deleted] (none) -> origin/pr/945/head
 x [deleted] (none) -> origin/pr/945/merge
 x [deleted] (none) -> origin/pr/946/head
 x [deleted] (none) -> origin/pr/946/merge
 x [deleted] (none) -> origin/pr/947/head
 x [deleted] (none) -> origin/

[GitHub] beam pull request #2527: Improve gcloud logging message

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2527


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #2527

[altay]

This closes #2527


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/69343a60
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/69343a60
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/69343a60

Branch: refs/heads/master
Commit: 69343a609313544093db37080027063a2b1786a7
Parents: 2c8071e defb1ac
Author: Ahmet Altay 
Authored: Thu Apr 13 14:44:46 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 14:44:46 2017 -0700

--
 sdks/python/apache_beam/internal/gcp/auth.py | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)
--

[1/2] beam git commit: Improve gcloud logging message

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 2c8071e20 -> 69343a609


Improve gcloud logging message


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/defb1ac0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/defb1ac0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/defb1ac0

Branch: refs/heads/master
Commit: defb1ac00e047b7e06c2174f435f7f6088e74f43
Parents: 2c8071e
Author: Sourabh Bajaj 
Authored: Thu Apr 13 13:39:26 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 14:44:34 2017 -0700

--
 sdks/python/apache_beam/internal/gcp/auth.py | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/defb1ac0/sdks/python/apache_beam/internal/gcp/auth.py
--
diff --git a/sdks/python/apache_beam/internal/gcp/auth.py 
b/sdks/python/apache_beam/internal/gcp/auth.py
index 97e21f4..53c2d07 100644
--- a/sdks/python/apache_beam/internal/gcp/auth.py
+++ b/sdks/python/apache_beam/internal/gcp/auth.py
@@ -106,9 +106,11 @@ class _GCloudWrapperCredentials(OAuth2Credentials):
 try:
   gcloud_process = processes.Popen(
   ['gcloud', 'auth', 'print-access-token'], stdout=processes.PIPE)
-except OSError as exn:
-  logging.error('The gcloud tool was not found.', exc_info=True)
-  raise AuthenticationException('The gcloud tool was not found: %s' % exn)
+except OSError:
+  message = 'gcloud tool not found so falling back to using ' +\
+'application default credentials'
+  logging.warning(message)
+  raise AuthenticationException(message)
 output, _ = gcloud_process.communicate()
 self.access_token = output.strip()
 

[GitHub] beam pull request #2526: [BEAM-1101, BEAM-1068] Remove service account name ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2526


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1101) Remove inconsistencies in Python PipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968260#comment-15968260
 ] 

ASF GitHub Bot commented on BEAM-1101:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2526


> Remove inconsistencies in Python PipelineOptions
> 
>
> Key: BEAM-1101
> URL: https://issues.apache.org/jira/browse/BEAM-1101
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Reporter: Pablo Estrada
>Assignee: Sourabh Bajaj
> Fix For: First stable release
>
>
> Some options have been removed from Java, and some have different default 
> behavior in Java. Gotta make this consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2526

[altay]

This closes #2526


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2c8071e2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2c8071e2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2c8071e2

Branch: refs/heads/master
Commit: 2c8071e20c732436bc17251361aa36831f803b34
Parents: 60834a9 549b950
Author: Ahmet Altay 
Authored: Thu Apr 13 14:43:00 2017 -0700
Committer: Ahmet Altay 
Committed: Thu Apr 13 14:43:00 2017 -0700

--
 sdks/python/apache_beam/internal/gcp/auth.py| 72 ++--
 .../apache_beam/internal/gcp/auth_test.py   | 44 
 .../apache_beam/utils/pipeline_options.py   |  7 --
 3 files changed, 19 insertions(+), 104 deletions(-)
--

[1/2] beam git commit: [BEAM-1101, BEAM-1068] Remove service account name credential pipeline options

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 60834a903 -> 2c8071e20


[BEAM-1101, BEAM-1068] Remove service account name credential pipeline options


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/549b950b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/549b950b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/549b950b

Branch: refs/heads/master
Commit: 549b950b1b3d4acd07d52297ed9722db3087729c
Parents: 60834a9
Author: Sourabh Bajaj 
Authored: Thu Apr 13 11:28:47 2017 -0700
Committer: Sourabh Bajaj 
Committed: Thu Apr 13 13:21:53 2017 -0700

--
 sdks/python/apache_beam/internal/gcp/auth.py| 72 ++--
 .../apache_beam/internal/gcp/auth_test.py   | 44 
 .../apache_beam/utils/pipeline_options.py   |  7 --
 3 files changed, 19 insertions(+), 104 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/549b950b/sdks/python/apache_beam/internal/gcp/auth.py
--
diff --git a/sdks/python/apache_beam/internal/gcp/auth.py 
b/sdks/python/apache_beam/internal/gcp/auth.py
index 8304658..97e21f4 100644
--- a/sdks/python/apache_beam/internal/gcp/auth.py
+++ b/sdks/python/apache_beam/internal/gcp/auth.py
@@ -21,7 +21,6 @@ import datetime
 import json
 import logging
 import os
-import sys
 import urllib2
 
 from oauth2client.client import GoogleCredentials
@@ -29,8 +28,6 @@ from oauth2client.client import OAuth2Credentials
 
 from apache_beam.utils import processes
 from apache_beam.utils import retry
-from apache_beam.utils.pipeline_options import GoogleCloudOptions
-from apache_beam.utils.pipeline_options import PipelineOptions
 
 
 # When we are running in GCE, we can authenticate with VM credentials.
@@ -135,53 +132,22 @@ def get_service_credentials():
 'https://www.googleapis.com/auth/datastore'
 ]
 
-# TODO(BEAM-1068): Do not recreate options from sys.argv.
-# We are currently being run from the command line.
-google_cloud_options = PipelineOptions(
-sys.argv).view_as(GoogleCloudOptions)
-if google_cloud_options.service_account_name:
-  if not google_cloud_options.service_account_key_file:
-raise AuthenticationException(
-'key file not provided for service account.')
-  if not os.path.exists(google_cloud_options.service_account_key_file):
-raise AuthenticationException(
-'Specified service account key file does not exist.')
-
-  # The following code uses oauth2client >=2.0.0 functionality and if this
-  # is not available due to import errors will use 1.5.2 functionality.
-  try:
-from oauth2client.service_account import ServiceAccountCredentials
-return ServiceAccountCredentials.from_p12_keyfile(
-google_cloud_options.service_account_name,
-google_cloud_options.service_account_key_file,
-private_key_password=None,
-scopes=client_scopes)
-  except ImportError:
-with file(google_cloud_options.service_account_key_file) as f:
-  service_account_key = f.read()
-from oauth2client.client import SignedJwtAssertionCredentials
-return SignedJwtAssertionCredentials(
-google_cloud_options.service_account_name,
-service_account_key,
-client_scopes,
-user_agent=user_agent)
-else:
-  try:
-credentials = _GCloudWrapperCredentials(user_agent)
-# Check if we are able to get an access token. If not fallback to
-# application default credentials.
-credentials.get_access_token()
-return credentials
-  except AuthenticationException:
-logging.warning('Unable to find credentials from gcloud.')
-
-  # Falling back to application default credentials.
-  try:
-credentials = GoogleCredentials.get_application_default()
-credentials = credentials.create_scoped(client_scopes)
-logging.debug('Connecting using Google Application Default '
-  'Credentials.')
-return credentials
-  except Exception:
-logging.warning('Unable to find default credentials to use.')
-raise
+try:
+  credentials = _GCloudWrapperCredentials(user_agent)
+  # Check if we are able to get an access token. If not fallback to
+  # application default credentials.
+  credentials.get_access_token()
+  return credentials
+except AuthenticationException:
+  logging.warning('Unable to find credentials from gcloud.')
+
+# Falling back to application default credentials.
+try:
+  credentials = GoogleCredentials.get_application_default()
+  credentials = credentials.create_scoped(client_scopes)
+  logging.debug('Connecting

[jira] [Commented] (BEAM-1970) Cannot run UserScore on Flink runner

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15968254#comment-15968254
 ] 

Luke Cwik commented on BEAM-1970:
-

The work around is to use a different coder then Avro like SerializableCoder on 
GameActionInfo

> Cannot run UserScore on Flink runner
> 
>
> Key: BEAM-1970
> URL: https://issues.apache.org/jira/browse/BEAM-1970
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ahmet Altay
>Assignee: Aljoscha Krettek
>
> Fails with error:
> ClassCastException: 
> org.apache.beam.examples.complete.game.UserScore$GameActionInfo cannot be 
> cast to org.apache.beam.examples.complete.game.UserScore$GameActionInfo
> full stack:
> 
>  The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error.
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:545)
> at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:339)
> at 
> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:831)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:256)
> at 
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1073)
> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1120)
> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1117)
> at 
> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at 
> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1116)
> Caused by: java.lang.RuntimeException: Pipeline execution failed
> at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:119)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:265)
> at 
> org.apache.beam.examples.complete.game.UserScore.main(UserScore.java:238)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528)
> ... 13 more
> Caused by: org.apache.flink.client.program.ProgramInvocationException: The 
> program execution failed: Job execution failed.
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
> at 
> org.apache.flink.yarn.YarnClusterClient.submitJob(YarnClusterClient.java:210)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:400)
> at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:387)
> at 
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
> at 
> org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.executePipeline(FlinkPipelineExecutionEnvironment.java:111)
> at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:116)
> ... 20 more
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job 
> execution failed.
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:900)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:843)
> at 
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:843)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at 
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
> at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
> at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at 
> scala.concurrent.f

[jira] [Created] (BEAM-1971) BigQuery should refer to gcpTempLocation instead of tempLocation

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-1971:
-

 Summary: BigQuery should refer to gcpTempLocation instead of 
tempLocation
 Key: BEAM-1971
 URL: https://issues.apache.org/jira/browse/BEAM-1971
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-gcp
Reporter: Thomas Groh


This is the actual supported option for a temp location that is known to be a 
Google Cloud Storage path.

Error messages should be cleaned up to ensure that the actual name of the 
option, 'gcpTempLocation', is present in the error messages.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)