[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852625#comment-15852625
 ] 

ASF GitHub Bot commented on BEAM-886:
-

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1917

[BEAM-886] Remove the usage of OldDoFn and clean up function names

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-886-remove-oldDoFn-6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1917.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1917


commit f7208f78141fe677a306a8844c1bf1d384836fd1
Author: Sourabh Bajaj 
Date:   2017-02-03T19:47:56Z

Remove the usage of OldDoFn and clean up function names




> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1917: [BEAM-886] Remove the usage of OldDoFn and clean up...

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/1917

[BEAM-886] Remove the usage of OldDoFn and clean up function names

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam BEAM-886-remove-oldDoFn-6

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1917.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1917


commit f7208f78141fe677a306a8844c1bf1d384836fd1
Author: Sourabh Bajaj 
Date:   2017-02-03T19:47:56Z

Remove the usage of OldDoFn and clean up function names




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1367) Need a way to force a page to be set to a specific language for code blocks

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852526#comment-15852526
 ] 

ASF GitHub Bot commented on BEAM-1367:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/140


> Need a way to force a page to be set to a specific language for code blocks
> ---
>
> Key: BEAM-1367
> URL: https://issues.apache.org/jira/browse/BEAM-1367
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Davor Bonaci
>
> When you have a language specific page that only has code blocks consisting 
> of that language and you are viewing it after toggling to a different 
> language, there is no indication there are any code blocks on the page at 
> all. Need a way to force a page to always be a specific language so the code 
> blocks always show up.
> Repro:
> 1. Go to programming guide and adapt for Java
> 2. Go to a page that only has python code blocks, and there is no indication 
> there are any code blocks at all. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #140: Add page about python type safety and type hint...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/140


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #140

[altay]

This closes #140


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/67fe97a3
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/67fe97a3
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/67fe97a3

Branch: refs/heads/asf-site
Commit: 67fe97a3c132b511a711f6b35ada7f2c553ab8b3
Parents: d583b25 56642bb
Author: Ahmet Altay 
Authored: Fri Feb 3 19:04:14 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Feb 3 19:04:14 2017 -0800

--
 .../documentation/programming-guide/index.html  |  18 +-
 .../sdks/python-type-safety/index.html  | 361 +++
 content/documentation/sdks/python/index.html|  15 +-
 .../get-started/wordcount-example/index.html|  10 +-
 src/documentation/sdks/python-type-safety.md| 138 +++
 src/documentation/sdks/python.md|  15 +-
 6 files changed, 537 insertions(+), 20 deletions(-)
--

[GitHub] beam-site pull request #147: Add new Managing Python Pipeline Dependencies p...

[melap]

GitHub user melap opened a pull request:

https://github.com/apache/beam-site/pull/147

Add new Managing Python Pipeline Dependencies page

R: @aaltay 


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/melap/incubator-beam-site python2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/147.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #147


commit 0b74641a37c949f94ef86b885ab295e14b62303a
Author: melissa 
Date:   2017-02-04T02:44:19Z

Add new Managing Python Pipeline Dependencies page




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1388) Update default configuration of retry decorator so that wait times are more practical

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852463#comment-15852463
 ] 

ASF GitHub Bot commented on BEAM-1388:
--

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/1916

[BEAM-1388] Updates default values used by retry decorator.

Updates following defaults so that total wait time by default is more 
practical.

num_retries from 16 to 7.
max_delay_secs from 4 hours to 1 hour.

With this update, for maximum number of retries, system will wait for 635 
sec while wait before last retry being 320 sec.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam update_retry_defaults

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1916.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1916


commit 640f5c61a25c100df0eca79b1a4417b81dbb9a83
Author: Chamikara Jayalath 
Date:   2017-02-04T01:32:49Z

Updates default values used by retry decorator.




> Update default configuration of retry decorator so that wait times are more 
> practical
> -
>
> Key: BEAM-1388
> URL: https://issues.apache.org/jira/browse/BEAM-1388
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> Currently retry decorator [1] has following default configuration.
> initial delay: 5 sec
> number of retries: 16
> factor: 2
> max delay: 4 hours
> I propose updating these values to following so that default values are more 
> practical.
> initial delay: 5 sec
> number of retries: 7
> factor: 2
> max delay: 1 hour
> With this update, for maximum number of retries, system will wait for 635 sec 
> while wait before last retry being 320 sec.
> [1] 
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/retry.py#L117



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1916: [BEAM-1388] Updates default values used by retry de...

[chamikaramj]

GitHub user chamikaramj opened a pull request:

https://github.com/apache/beam/pull/1916

[BEAM-1388] Updates default values used by retry decorator.

Updates following defaults so that total wait time by default is more 
practical.

num_retries from 16 to 7.
max_delay_secs from 4 hours to 1 hour.

With this update, for maximum number of retries, system will wait for 635 
sec while wait before last retry being 320 sec.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chamikaramj/beam update_retry_defaults

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1916.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1916


commit 640f5c61a25c100df0eca79b1a4417b81dbb9a83
Author: Chamikara Jayalath 
Date:   2017-02-04T01:32:49Z

Updates default values used by retry decorator.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1914

[altay]

This closes #1914


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/6e220bb3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/6e220bb3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/6e220bb3

Branch: refs/heads/master
Commit: 6e220bb3719c0b39886b56182156f20136b14415
Parents: fbd69dc c2d8d71
Author: Ahmet Altay 
Authored: Fri Feb 3 17:02:21 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Feb 3 17:02:21 2017 -0800

--
 sdks/python/apache_beam/io/gcsio.py  | 71 +++
 sdks/python/apache_beam/io/gcsio_test.py | 38 ++
 2 files changed, 99 insertions(+), 10 deletions(-)
--

[jira] [Created] (BEAM-1388) Update default configuration of retry decorator so that wait times are more practical

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-1388:


 Summary: Update default configuration of retry decorator so that 
wait times are more practical
 Key: BEAM-1388
 URL: https://issues.apache.org/jira/browse/BEAM-1388
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py
Reporter: Chamikara Jayalath
Assignee: Chamikara Jayalath


Currently retry decorator [1] has following default configuration.

initial delay: 5 sec
number of retries: 16
factor: 2
max delay: 4 hours

I propose updating these values to following so that default values are more 
practical.

initial delay: 5 sec
number of retries: 7
factor: 2
max delay: 1 hour

With this update, for maximum number of retries, system will wait for 635 sec 
while wait before last retry being 320 sec.



[1] 
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/retry.py#L117



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam-site pull request #146: Update programming guide with Python snippets a...

[hadarhg]

GitHub user hadarhg opened a pull request:

https://github.com/apache/beam-site/pull/146

Update programming guide with Python snippets and data encoding section.

R: @francesperry 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hadarhg/incubator-beam-site programming-guide

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam-site/pull/146.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #146


commit 2d55990cbd6c66c1467736fd958bab83c81108dd
Author: Hadar Hod 
Date:   2017-02-04T00:35:41Z

Update programming guide with Python snippets and data encoding section.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #1892

[davor]

This closes #1892


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/fbd69dcf
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/fbd69dcf
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/fbd69dcf

Branch: refs/heads/master
Commit: fbd69dcf762dcd3633baef77f01d3828f31b9faf
Parents: 72fef99 16c8e9f
Author: Davor Bonaci 
Authored: Fri Feb 3 15:57:54 2017 -0800
Committer: Davor Bonaci 
Committed: Fri Feb 3 15:57:54 2017 -0800

--
 .jenkins/common_job_properties.groovy   | 27 ++--
 ...ostCommit_Java_RunnableOnService_Apex.groovy |  6 +
 ...ommit_Java_RunnableOnService_Dataflow.groovy |  6 +
 ...stCommit_Java_RunnableOnService_Flink.groovy |  6 +
 ...ommit_Java_RunnableOnService_Gearpump.groovy |  8 +-
 ...stCommit_Java_RunnableOnService_Spark.groovy |  6 +
 .../job_beam_PostCommit_Python_Verify.groovy|  6 +
 .../job_beam_PreCommit_Java_MavenInstall.groovy |  2 +-
 .../job_beam_PreCommit_Website_Stage.groovy |  2 +-
 .jenkins/job_beam_PreCommit_Website_Test.groovy |  2 +-
 .jenkins/job_seed.groovy|  6 +
 11 files changed, 71 insertions(+), 6 deletions(-)
--

[1/2] beam git commit: Enable oneshots of Jenkins runs in precommit.

[davor]

Repository: beam
Updated Branches:
  refs/heads/master 72fef99a6 -> fbd69dcf7


Enable oneshots of Jenkins runs in precommit.

Signed-off-by: Jason Kuster 


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/16c8e9fe
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/16c8e9fe
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/16c8e9fe

Branch: refs/heads/master
Commit: 16c8e9fe40e89097d44c0116bc14fe57c818d0a2
Parents: 72fef99
Author: Jason Kuster 
Authored: Wed Feb 1 15:32:44 2017 -0800
Committer: Davor Bonaci 
Committed: Fri Feb 3 15:57:45 2017 -0800

--
 .jenkins/common_job_properties.groovy   | 27 ++--
 ...ostCommit_Java_RunnableOnService_Apex.groovy |  6 +
 ...ommit_Java_RunnableOnService_Dataflow.groovy |  6 +
 ...stCommit_Java_RunnableOnService_Flink.groovy |  6 +
 ...ommit_Java_RunnableOnService_Gearpump.groovy |  8 +-
 ...stCommit_Java_RunnableOnService_Spark.groovy |  6 +
 .../job_beam_PostCommit_Python_Verify.groovy|  6 +
 .../job_beam_PreCommit_Java_MavenInstall.groovy |  2 +-
 .../job_beam_PreCommit_Website_Stage.groovy |  2 +-
 .jenkins/job_beam_PreCommit_Website_Test.groovy |  2 +-
 .jenkins/job_seed.groovy|  6 +
 11 files changed, 71 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/16c8e9fe/.jenkins/common_job_properties.groovy
--
diff --git a/.jenkins/common_job_properties.groovy 
b/.jenkins/common_job_properties.groovy
index eee1cda..3c77aab 100644
--- a/.jenkins/common_job_properties.groovy
+++ b/.jenkins/common_job_properties.groovy
@@ -100,7 +100,8 @@ class common_job_properties {
   // below to insulate callers from internal parameter defaults.
   private static def setPullRequestBuildTrigger(def context,
 def commitStatusContext,
-def successComment = 
'--none--') {
+def successComment = 
'--none--',
+def trigger_phrase = '') {
 context.triggers {
   githubPullRequest {
 admins(['asfbot'])
@@ -108,12 +109,21 @@ class common_job_properties {
 orgWhitelist(['apache'])
 allowMembersOfWhitelistedOrgsAsAdmin()
 permitAll()
+// trigger_phrase is the argument which gets set when we want to allow
+// post-commit builds to run against pending pull requests. This block
+// overrides the default trigger phrase with the new one. Setting this
+// will disable automatic invocation of this build; the phrase will be
+// required to start it.
+if (trigger_phrase != '') {
+  triggerPhrase(trigger_phrase)
+  onlyTriggerPhrase()
+}
 
 extensions {
   commitStatus {
 // This is the name that will show up in the GitHub pull request UI
 // for this Jenkins project.
-delegate.context(commitStatusContext)
+delegate.context("Jenkins: " + commitStatusContext)
   }
 
   /*
@@ -171,6 +181,19 @@ class common_job_properties {
 setPullRequestBuildTrigger(context, commitStatusName, successComment)
   }
 
+  // Enable triggering postcommit runs against pull requests. Users can 
comment the trigger phrase
+  // specified in the postcommit job and have the job run against their PR to 
run
+  // tests not in the presubmit suite for additional confidence.
+  static def enablePhraseTriggeringFromPullRequest(def context,
+ def commitStatusName,
+ def trigger_phrase) {
+setPullRequestBuildTrigger(
+  context,
+  commitStatusName,
+  '--none--',
+  trigger_phrase)
+  }
+
   // Sets common config for PostCommit jobs.
   static def setPostCommit(def context,
def build_schedule = '0 */6 * * *',

http://git-wip-us.apache.org/repos/asf/beam/blob/16c8e9fe/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
--
diff --git a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy 
b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
index f657596..b2bed00 100644
--- a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
+++ b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Apex.groovy
@@ -31,6 +31,12 @@ mavenJob('beam_PostCommit_Java_RunnableOnService_Apex') {
   // Sets that this is a PostCommit job.
   common_job_properties.setPostCommit(delegate)
 
+  // 

[GitHub] beam pull request #1892: [BEAM-1374] Enable oneshots of Jenkins runs in prec...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1892


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (BEAM-1386) Job hangs without warnings after reading ~20GB of gz csv

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay closed BEAM-1386.
-
   Resolution: Information Provided
Fix Version/s: Not applicable

> Job hangs without warnings after reading ~20GB of gz csv
> 
>
> Key: BEAM-1386
> URL: https://issues.apache.org/jira/browse/BEAM-1386
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 0.5.0
> Environment: Running on Google Dataflow with 'n1-standard-8' machines.
>Reporter: Johan Brodin
>Assignee: Ahmet Altay
>Priority: Minor
> Fix For: Not applicable
>
>
> When running the job it works fine up until 20GB or around 23 million rows 
> from a gzip:ed csv file (total size 43M rows). Halted the job so the 
> statistic from it seam to disappeared, but here it is the id 
> "2017-02-03_04_25_41-15296331815975218867". Is there any built in limitations 
> to file size? Should I try to break the file up into several smaller files? 
> Could the issue be related to the workers disk size?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1387) DatastoreIO: use batch-datastore.googleapis.com as the endpoint

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852278#comment-15852278
 ] 

ASF GitHub Bot commented on BEAM-1387:
--

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1915

[BEAM-1387] DatastoreV1: use batch-datastore.googleapis.com endpoint

If localhost is not set.

R: @vikkyrk 
cc: @eddavisson

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam beam-1387

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1915


commit 3a3f7e5634827d18cb39dd2ed12ceccc9d95ab24
Author: Dan Halperin 
Date:   2017-02-03T22:58:19Z

[BEAM-1387] DatastoreV1: use batch-datastore.googleapis.com endpoint

If localhost is not set.




> DatastoreIO: use batch-datastore.googleapis.com as the endpoint
> ---
>
> Key: BEAM-1387
> URL: https://issues.apache.org/jira/browse/BEAM-1387
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp, sdk-py
>Reporter: Daniel Halperin
>Assignee: Daniel Halperin
>
> Google Cloud Datastore has added a new API endpoint for big data processing 
> systems. We should use it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1915: [BEAM-1387] DatastoreV1: use batch-datastore.google...

[dhalperi]

GitHub user dhalperi opened a pull request:

https://github.com/apache/beam/pull/1915

[BEAM-1387] DatastoreV1: use batch-datastore.googleapis.com endpoint

If localhost is not set.

R: @vikkyrk 
cc: @eddavisson

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dhalperi/beam beam-1387

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/1915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1915


commit 3a3f7e5634827d18cb39dd2ed12ceccc9d95ab24
Author: Dan Halperin 
Date:   2017-02-03T22:58:19Z

[BEAM-1387] DatastoreV1: use batch-datastore.googleapis.com endpoint

If localhost is not set.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (BEAM-278) Add I/O section

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-278.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Add I/O section
> ---
>
> Key: BEAM-278
> URL: https://issues.apache.org/jira/browse/BEAM-278
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Devin Donnelly
>Assignee: Melissa Pashniak
> Fix For: Not applicable
>
>
> Document I/O Sources and Sinks APIs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Closed] (BEAM-278) Add I/O section

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin closed BEAM-278.


> Add I/O section
> ---
>
> Key: BEAM-278
> URL: https://issues.apache.org/jira/browse/BEAM-278
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Devin Donnelly
>Assignee: Melissa Pashniak
> Fix For: Not applicable
>
>
> Document I/O Sources and Sinks APIs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-278) Add I/O section

[Melissa Pashniak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852239#comment-15852239
 ] 

Melissa Pashniak commented on BEAM-278:
---

Completed: https://github.com/apache/beam-site/pull/121


> Add I/O section
> ---
>
> Key: BEAM-278
> URL: https://issues.apache.org/jira/browse/BEAM-278
> Project: Beam
>  Issue Type: Sub-task
>  Components: website
>Reporter: Devin Donnelly
>Assignee: Melissa Pashniak
>
> Document I/O Sources and Sinks APIs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1378) Late data dropping was removed from Flink Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852016#comment-15852016
 ] 

ASF GitHub Bot commented on BEAM-1378:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1901


> Late data dropping was removed from Flink Runner
> 
>
> Key: BEAM-1378
> URL: https://issues.apache.org/jira/browse/BEAM-1378
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: Drop late data in Flink runner

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 4ccbdbc38 -> 72fef99a6


Drop late data in Flink runner


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/0454a189
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/0454a189
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/0454a189

Branch: refs/heads/master
Commit: 0454a1897c645f674754bc9ef69dc7bab2b3c3ba
Parents: 7da5a2c
Author: Kenneth Knowles 
Authored: Wed Feb 1 18:25:42 2017 -0800
Committer: Kenneth Knowles 
Committed: Wed Feb 1 18:25:42 2017 -0800

--
 .../wrappers/streaming/DoFnOperator.java| 20 ++--
 1 file changed, 18 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/0454a189/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
index de0264a..c1d33f7 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java
@@ -33,6 +33,7 @@ import org.apache.beam.runners.core.AggregatorFactory;
 import org.apache.beam.runners.core.DoFnRunner;
 import org.apache.beam.runners.core.DoFnRunners;
 import org.apache.beam.runners.core.ExecutionContext;
+import org.apache.beam.runners.core.GroupAlsoByWindowViaWindowSetNewDoFn;
 import org.apache.beam.runners.core.PushbackSideInputDoFnRunner;
 import org.apache.beam.runners.core.SideInputHandler;
 import 
org.apache.beam.runners.flink.translation.utils.SerializedPipelineOptions;
@@ -234,6 +235,8 @@ public class DoFnOperator
 
 doFnInvoker.invokeSetup();
 
+ExecutionContext.StepContext stepContext = createStepContext();
+
 DoFnRunner doFnRunner = DoFnRunners.simpleRunner(
 serializedOptions.getPipelineOptions(),
 doFn,
@@ -241,13 +244,26 @@ public class DoFnOperator
 outputManager,
 mainOutputTag,
 sideOutputTags,
-createStepContext(),
+stepContext,
 aggregatorFactory,
 windowingStrategy);
 
+if (doFn instanceof GroupAlsoByWindowViaWindowSetNewDoFn) {
+  // When the doFn is this, we know it came from WindowDoFnOperator and
+  //   InputT = KeyedWorkItem
+  //   OutputT = KV
+  //
+  // for some K, V
+
+  doFnRunner = DoFnRunners.lateDataDroppingRunner(
+  (DoFnRunner) doFnRunner,
+  stepContext,
+  windowingStrategy,
+  ((GroupAlsoByWindowViaWindowSetNewDoFn) 
doFn).getDroppedDueToLatenessAggregator());
+}
+
 pushbackDoFnRunner =
 PushbackSideInputDoFnRunner.create(doFnRunner, sideInputs, 
sideInputHandler);
-
   }
 
   @Override

[jira] [Comment Edited] (BEAM-1367) Need a way to force a page to be set to a specific language for code blocks

[Melissa Pashniak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851872#comment-15851872
 ] 

Melissa Pashniak edited comment on BEAM-1367 at 2/3/17 6:27 PM:


After this is fixed, the Python Type Safety page needs to be force set to 
Python and code blocks specified as Python. They are generic code blocks 
temporarily, to make sure users who have set Adapt for: Java previously don't 
see the page with missing code blocks. 



was (Author: melap):
After this is fixed, the Python Type Safety page needs to be force set to 
Python and code blocks specified as Python. They are generic code blocks 
temporarily. 


> Need a way to force a page to be set to a specific language for code blocks
> ---
>
> Key: BEAM-1367
> URL: https://issues.apache.org/jira/browse/BEAM-1367
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Davor Bonaci
>
> When you have a language specific page that only has code blocks consisting 
> of that language and you are viewing it after toggling to a different 
> language, there is no indication there are any code blocks on the page at 
> all. Need a way to force a page to always be a specific language so the code 
> blocks always show up.
> Repro:
> 1. Go to programming guide and adapt for Java
> 2. Go to a page that only has python code blocks, and there is no indication 
> there are any code blocks at all. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1367) Need a way to force a page to be set to a specific language for code blocks

[Melissa Pashniak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851872#comment-15851872
 ] 

Melissa Pashniak commented on BEAM-1367:


After this is fixed, the Python Type Safety page needs to be force set to 
Python and code blocks specified as Python. They are generic code blocks 
temporarily. 


> Need a way to force a page to be set to a specific language for code blocks
> ---
>
> Key: BEAM-1367
> URL: https://issues.apache.org/jira/browse/BEAM-1367
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Davor Bonaci
>
> When you have a language specific page that only has code blocks consisting 
> of that language and you are viewing it after toggling to a different 
> language, there is no indication there are any code blocks on the page at 
> all. Need a way to force a page to always be a specific language so the code 
> blocks always show up.
> Repro:
> 1. Go to programming guide and adapt for Java
> 2. Go to a page that only has python code blocks, and there is no indication 
> there are any code blocks at all. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-886) Support new DoFn in Python SDK

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851841#comment-15851841
 ] 

ASF GitHub Bot commented on BEAM-886:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1913


> Support new DoFn in Python SDK
> --
>
> Key: BEAM-886
> URL: https://issues.apache.org/jira/browse/BEAM-886
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py
>Reporter: Ahmet Altay
>Assignee: Sourabh Bajaj
>  Labels: backward-incompatible, sdk-consistency
>
> Figure out what is needed for supporting new DoFns, add support and removed 
> old DoFns.
> Related Docs from Java:
> Original Proposal email:
> https://lists.apache.org/thread.html/2abf32d528dbb64b79853552c5d10c217e2194f0685af21aeb4635dd@%3Cdev.beam.apache.org%3E
> Presentation & Doc (with short Python sections):
> https://s.apache.org/presenting-a-new-dofn
> https://s.apache.org/a-new-dofn



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1913: [BEAM-886] Rename NewDoFn to DoFn

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1913


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Rename NewDoFn to DoFn

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 99f93eb07 -> 4ccbdbc38


Rename NewDoFn to DoFn


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1af093b1
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1af093b1
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1af093b1

Branch: refs/heads/master
Commit: 1af093b10dda8e4daf505257d60f46b4bc38e7eb
Parents: 99f93eb
Author: Sourabh Bajaj 
Authored: Thu Feb 2 21:29:47 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Feb 3 09:47:48 2017 -0800

--
 .../examples/complete/top_wikipedia_sessions.py | 10 +++
 .../examples/cookbook/datastore_wordcount.py|  2 +-
 .../examples/cookbook/multiple_output_pardo.py  |  2 +-
 .../apache_beam/examples/snippets/snippets.py   | 10 +++
 .../examples/snippets/snippets_test.py  | 14 -
 sdks/python/apache_beam/examples/wordcount.py   |  2 +-
 .../apache_beam/examples/wordcount_debugging.py |  2 +-
 .../apache_beam/io/datastore/v1/datastoreio.py  |  8 +++---
 sdks/python/apache_beam/io/iobase.py|  6 ++--
 sdks/python/apache_beam/pipeline_test.py| 24 
 sdks/python/apache_beam/runners/common.py   | 30 ++--
 .../consumer_tracking_pipeline_visitor_test.py  |  6 ++--
 .../runners/direct/helper_transforms.py |  6 ++--
 .../runners/direct/transform_evaluator.py   | 18 ++--
 sdks/python/apache_beam/runners/runner_test.py  |  4 +--
 sdks/python/apache_beam/transforms/core.py  | 28 +-
 .../apache_beam/transforms/display_test.py  |  8 +++---
 .../apache_beam/transforms/ptransform_test.py   | 18 ++--
 .../apache_beam/transforms/window_test.py   |  4 +--
 sdks/python/apache_beam/typehints/typecheck.py  | 22 +++---
 .../typehints/typed_pipeline_test.py|  4 +--
 21 files changed, 114 insertions(+), 114 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1af093b1/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
--
diff --git 
a/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py 
b/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
index 43a4ee2..d7fbe30 100644
--- a/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
+++ b/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions.py
@@ -59,7 +59,7 @@ THIRTY_DAYS_IN_SECONDS = 30 * 24 * ONE_HOUR_IN_SECONDS
 MAX_TIMESTAMP = 0x7fff
 
 
-class ExtractUserAndTimestampDoFn(beam.NewDoFn):
+class ExtractUserAndTimestampDoFn(beam.DoFn):
   """Extracts user and timestamp representing a Wikipedia edit."""
 
   def process(self, element):
@@ -103,17 +103,17 @@ class TopPerMonth(beam.PTransform):
 .without_defaults())
 
 
-class SessionsToStringsDoFn(beam.NewDoFn):
+class SessionsToStringsDoFn(beam.DoFn):
   """Adds the session information to be part of the key."""
 
-  def process(self, element, window=beam.NewDoFn.WindowParam):
+  def process(self, element, window=beam.DoFn.WindowParam):
 yield (element[0] + ' : ' + str(window), element[1])
 
 
-class FormatOutputDoFn(beam.NewDoFn):
+class FormatOutputDoFn(beam.DoFn):
   """Formats a string containing the user, count, and session."""
 
-  def process(self, element, window=beam.NewDoFn.WindowParam):
+  def process(self, element, window=beam.DoFn.WindowParam):
 for kv in element:
   session = kv[0]
   count = kv[1]

http://git-wip-us.apache.org/repos/asf/beam/blob/1af093b1/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
--
diff --git a/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py 
b/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
index 42cd59b..282afbf 100644
--- a/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
+++ b/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py
@@ -84,7 +84,7 @@ word_length_counter = Metrics.counter('main', 'word_lengths')
 word_counter = Metrics.counter('main', 'total_words')
 
 
-class WordExtractingDoFn(beam.NewDoFn):
+class WordExtractingDoFn(beam.DoFn):
   """Parse each line of input text into words."""
 
   def process(self, element):

http://git-wip-us.apache.org/repos/asf/beam/blob/1af093b1/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
--
diff --git a/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py 
b/sdks/python/apache_beam/examples/cookbook/multiple_output_pardo.py
index 0cbbf9a..26e97c7 100644
--- 

[2/2] beam git commit: This closes #1913

[altay]

This closes #1913


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4ccbdbc3
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4ccbdbc3
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4ccbdbc3

Branch: refs/heads/master
Commit: 4ccbdbc3841b37efeb0b1c148216fa0f63efe4e2
Parents: 99f93eb 1af093b
Author: Ahmet Altay 
Authored: Fri Feb 3 09:47:59 2017 -0800
Committer: Ahmet Altay 
Committed: Fri Feb 3 09:47:59 2017 -0800

--
 .../examples/complete/top_wikipedia_sessions.py | 10 +++
 .../examples/cookbook/datastore_wordcount.py|  2 +-
 .../examples/cookbook/multiple_output_pardo.py  |  2 +-
 .../apache_beam/examples/snippets/snippets.py   | 10 +++
 .../examples/snippets/snippets_test.py  | 14 -
 sdks/python/apache_beam/examples/wordcount.py   |  2 +-
 .../apache_beam/examples/wordcount_debugging.py |  2 +-
 .../apache_beam/io/datastore/v1/datastoreio.py  |  8 +++---
 sdks/python/apache_beam/io/iobase.py|  6 ++--
 sdks/python/apache_beam/pipeline_test.py| 24 
 sdks/python/apache_beam/runners/common.py   | 30 ++--
 .../consumer_tracking_pipeline_visitor_test.py  |  6 ++--
 .../runners/direct/helper_transforms.py |  6 ++--
 .../runners/direct/transform_evaluator.py   | 18 ++--
 sdks/python/apache_beam/runners/runner_test.py  |  4 +--
 sdks/python/apache_beam/transforms/core.py  | 28 +-
 .../apache_beam/transforms/display_test.py  |  8 +++---
 .../apache_beam/transforms/ptransform_test.py   | 18 ++--
 .../apache_beam/transforms/window_test.py   |  4 +--
 sdks/python/apache_beam/typehints/typecheck.py  | 22 +++---
 .../typehints/typed_pipeline_test.py|  4 +--
 21 files changed, 114 insertions(+), 114 deletions(-)
--

[jira] [Updated] (BEAM-1386) Job hangs without warnings after reading ~20GB of gz csv

[Johan Brodin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johan Brodin updated BEAM-1386:
---
Priority: Minor  (was: Critical)

> Job hangs without warnings after reading ~20GB of gz csv
> 
>
> Key: BEAM-1386
> URL: https://issues.apache.org/jira/browse/BEAM-1386
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 0.5.0
> Environment: Running on Google Dataflow with 'n1-standard-8' machines.
>Reporter: Johan Brodin
>Assignee: Ahmet Altay
>Priority: Minor
>
> When running the job it works fine up until 20GB or around 23 million rows 
> from a gzip:ed csv file (total size 43M rows). Halted the job so the 
> statistic from it seam to disappeared, but here it is the id 
> "2017-02-03_04_25_41-15296331815975218867". Is there any built in limitations 
> to file size? Should I try to break the file up into several smaller files? 
> Could the issue be related to the workers disk size?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1386) Job hangs without warnings after reading ~20GB of gz csv

[Johan Brodin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Johan Brodin updated BEAM-1386:
---
Priority: Critical  (was: Major)

> Job hangs without warnings after reading ~20GB of gz csv
> 
>
> Key: BEAM-1386
> URL: https://issues.apache.org/jira/browse/BEAM-1386
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
>Affects Versions: 0.5.0
> Environment: Running on Google Dataflow with 'n1-standard-8' machines.
>Reporter: Johan Brodin
>Assignee: Ahmet Altay
>Priority: Critical
>
> When running the job it works fine up until 20GB or around 23 million rows 
> from a gzip:ed csv file (total size 43M rows). Halted the job so the 
> statistic from it seam to disappeared, but here it is the id 
> "2017-02-03_04_25_41-15296331815975218867". Is there any built in limitations 
> to file size? Should I try to break the file up into several smaller files? 
> Could the issue be related to the workers disk size?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (BEAM-1386) Job hangs without warnings after reading ~20GB of gz csv

[Johan Brodin (JIRA)]

Johan Brodin created BEAM-1386:
--

 Summary: Job hangs without warnings after reading ~20GB of gz csv
 Key: BEAM-1386
 URL: https://issues.apache.org/jira/browse/BEAM-1386
 Project: Beam
  Issue Type: New Feature
  Components: sdk-py
Affects Versions: 0.5.0
 Environment: Running on Google Dataflow with 'n1-standard-8' machines.
Reporter: Johan Brodin
Assignee: Ahmet Altay


When running the job it works fine up until 20GB or around 23 million rows from 
a gzip:ed csv file (total size 43M rows). Halted the job so the statistic from 
it seam to disappeared, but here it is the id 
"2017-02-03_04_25_41-15296331815975218867". Is there any built in limitations 
to file size? Should I try to break the file up into several smaller files? 
Could the issue be related to the workers disk size?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-842) dependency.py: package not found when running on Windows

[Kai Jiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851475#comment-15851475
 ] 

Kai Jiang commented on BEAM-842:


[~matthiasa4] Could you paste more crash error here?
Since I built Juliaset example on Windows and ran it on Dataflow, it seems all 
good to me. setuptools version is 34.1.0

> dependency.py: package not found when running on Windows
> 
>
> Key: BEAM-842
> URL: https://issues.apache.org/jira/browse/BEAM-842
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py
> Environment: Windows 10, Python 2.7.11
>Reporter: Matthias Baetens
>Assignee: Ahmet Altay
>Priority: Minor
>  Labels: newbie
>
> When having splitting your pipeline into multiple files and configuring your 
> project according to the Juliaset example 
> (https://cloud.google.com/dataflow/pipelines/dependencies-python#multiple-file-dependencies),
>  the Pipeline still crashes when using Windows.
> This is caused by setuptools defaulting to a .zip on Windows, and the current 
> Beam code looks for a .tar.gz (dependency.py, line 400). When changing this 
> line to: output_files = glob.glob(os.path.join(temp_dir, '*.zip')), it works. 
> Suggestion: checking the OS would probably solve this issue. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] beam git commit: [BEAM-1036] Support for new State API in FlinkRunner (streaming)

[aljoscha]

Repository: beam
Updated Branches:
  refs/heads/master cef31093f -> 99f93eb07


[BEAM-1036] Support for new State API in FlinkRunner (streaming)


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b56f4609
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b56f4609
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b56f4609

Branch: refs/heads/master
Commit: b56f460974a3119804040b08fe1c7b190ff3f356
Parents: cef3109
Author: JingsongLi 
Authored: Tue Jan 31 12:32:26 2017 +0800
Committer: Aljoscha Krettek 
Committed: Fri Feb 3 13:36:11 2017 +0100

--
 runners/flink/runner/pom.xml|   1 -
 .../FlinkStreamingTransformTranslators.java | 137 ++-
 .../wrappers/streaming/DoFnOperator.java|  15 +-
 .../streaming/KvToByteBufferKeySelector.java|  56 
 .../wrappers/streaming/WindowDoFnOperator.java  |  13 +-
 .../beam/runners/flink/PipelineOptionsTest.java |   4 +-
 .../flink/streaming/DoFnOperatorTest.java   |   9 +-
 7 files changed, 182 insertions(+), 53 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b56f4609/runners/flink/runner/pom.xml
--
diff --git a/runners/flink/runner/pom.xml b/runners/flink/runner/pom.xml
index fe058b5..207740a 100644
--- a/runners/flink/runner/pom.xml
+++ b/runners/flink/runner/pom.xml
@@ -88,7 +88,6 @@
   
org.apache.beam.sdk.testing.RunnableOnService
   
 org.apache.beam.sdk.testing.FlattenWithHeterogeneousCoders,
-org.apache.beam.sdk.testing.UsesStatefulParDo,
 org.apache.beam.sdk.testing.UsesTimersInParDo,
 org.apache.beam.sdk.testing.UsesSplittableParDo,
 org.apache.beam.sdk.testing.UsesAttemptedMetrics,

http://git-wip-us.apache.org/repos/asf/beam/blob/b56f4609/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
--
diff --git 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
index 24ef987..1195c82 100644
--- 
a/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
+++ 
b/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/translation/FlinkStreamingTransformTranslators.java
@@ -35,6 +35,7 @@ import 
org.apache.beam.runners.flink.translation.functions.FlinkAssignWindows;
 import org.apache.beam.runners.flink.translation.types.CoderTypeInformation;
 import org.apache.beam.runners.flink.translation.types.FlinkCoder;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator;
+import 
org.apache.beam.runners.flink.translation.wrappers.streaming.KvToByteBufferKeySelector;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKeyedWorkItem;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.SingletonKeyedWorkItemCoder;
 import 
org.apache.beam.runners.flink.translation.wrappers.streaming.WindowDoFnOperator;
@@ -307,19 +308,9 @@ public class FlinkStreamingTransformTranslators {
 }
   }
 
-  private static void rejectStateAndTimers(DoFn doFn) {
+  private static void rejectTimers(DoFn doFn) {
 DoFnSignature signature = DoFnSignatures.getSignature(doFn.getClass());
 
-if (signature.stateDeclarations().size() > 0) {
-  throw new UnsupportedOperationException(
-  String.format(
-  "Found %s annotations on %s, but %s cannot yet be used with 
state in the %s.",
-  DoFn.StateId.class.getSimpleName(),
-  doFn.getClass().getName(),
-  DoFn.class.getSimpleName(),
-  FlinkRunner.class.getSimpleName()));
-}
-
 if (signature.timerDeclarations().size() > 0) {
   throw new UnsupportedOperationException(
   String.format(
@@ -341,7 +332,7 @@ public class FlinkStreamingTransformTranslators {
 FlinkStreamingTranslationContext context) {
 
   DoFn doFn = transform.getFn();
-  rejectStateAndTimers(doFn);
+  rejectTimers(doFn);
 
   WindowingStrategy windowingStrategy =
   context.getOutput(transform).getWindowingStrategy();
@@ -357,6 +348,20 @@ public class FlinkStreamingTransformTranslators {
   TypeInformation inputTypeInfo =
   context.getTypeInfo(inputPCollection);
 
+  DataStream inputDataStream =
+  

[2/2] beam git commit: This closes #1878

[aljoscha]

This closes #1878


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/99f93eb0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/99f93eb0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/99f93eb0

Branch: refs/heads/master
Commit: 99f93eb078819882f622d953ff05ca419516c90f
Parents: cef3109 b56f460
Author: Aljoscha Krettek 
Authored: Fri Feb 3 13:54:18 2017 +0100
Committer: Aljoscha Krettek 
Committed: Fri Feb 3 13:54:18 2017 +0100

--
 runners/flink/runner/pom.xml|   1 -
 .../FlinkStreamingTransformTranslators.java | 137 ++-
 .../wrappers/streaming/DoFnOperator.java|  15 +-
 .../streaming/KvToByteBufferKeySelector.java|  56 
 .../wrappers/streaming/WindowDoFnOperator.java  |  13 +-
 .../beam/runners/flink/PipelineOptionsTest.java |   4 +-
 .../flink/streaming/DoFnOperatorTest.java   |   9 +-
 7 files changed, 182 insertions(+), 53 deletions(-)
--

[jira] [Resolved] (BEAM-1127) JmsIO should create an unique source in case of topic

[Daniel Halperin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Halperin resolved BEAM-1127.
---
   Resolution: Fixed
Fix Version/s: 0.6.0

> JmsIO should create an unique source in case of topic
> -
>
> Key: BEAM-1127
> URL: https://issues.apache.org/jira/browse/BEAM-1127
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
> Fix For: 0.6.0
>
>
> The JmsIO deals with both queue and topic.
> Currently, we create multiple sources depending of the desired number of 
> splits.
> If this behavior is correct when using a queue (we have concurrent consumers 
> of the queue), it's basically wrong when using a topic (we have multiple 
> subscribers, so potentially duplicated message).
> When using a topic, the JmsIO should use an unique source (to avoid messages 
> duplication).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1127) JmsIO should create an unique source in case of topic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851369#comment-15851369
 ] 

ASF GitHub Bot commented on BEAM-1127:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1573


> JmsIO should create an unique source in case of topic
> -
>
> Key: BEAM-1127
> URL: https://issues.apache.org/jira/browse/BEAM-1127
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>
> The JmsIO deals with both queue and topic.
> Currently, we create multiple sources depending of the desired number of 
> splits.
> If this behavior is correct when using a queue (we have concurrent consumers 
> of the queue), it's basically wrong when using a topic (we have multiple 
> subscribers, so potentially duplicated message).
> When using a topic, the JmsIO should use an unique source (to avoid messages 
> duplication).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1573: [BEAM-1127] Create an unique source when using a JM...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1573


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1127] Create an unique source when using a JMS topic to avoid elements duplication

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 8ee3572b4 -> cef31093f


[BEAM-1127] Create an unique source when using a JMS topic to avoid elements 
duplication


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/eed4efc5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/eed4efc5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/eed4efc5

Branch: refs/heads/master
Commit: eed4efc537e5214d037665561c0e901931929c29
Parents: 8ee3572
Author: Jean-Baptiste Onofré 
Authored: Thu Feb 2 16:17:25 2017 +0100
Committer: Dan Halperin 
Committed: Fri Feb 3 03:28:33 2017 -0800

--
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java  | 15 +++-
 .../org/apache/beam/sdk/io/jms/JmsIOTest.java   | 39 ++--
 2 files changed, 49 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/eed4efc5/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
--
diff --git 
a/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java 
b/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
index c1f1cb4..270fe31 100644
--- a/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
+++ b/sdks/java/io/jms/src/main/java/org/apache/beam/sdk/io/jms/JmsIO.java
@@ -325,7 +325,11 @@ public class JmsIO {
 
   private JmsIO() {}
 
-  private static class UnboundedJmsSource extends UnboundedSource {
+  /**
+   * An unbounded JMS source.
+   */
+  @VisibleForTesting
+  protected static class UnboundedJmsSource extends UnboundedSource {
 
 private final Read spec;
 
@@ -337,8 +341,15 @@ public class JmsIO {
 public List generateInitialSplits(
 int desiredNumSplits, PipelineOptions options) throws Exception {
   List sources = new ArrayList<>();
-  for (int i = 0; i < desiredNumSplits; i++) {
+  if (spec.getTopic() != null) {
+// in the case of a topic, we create a single source, so an unique 
subscriber, to avoid
+// element duplication
 sources.add(new UnboundedJmsSource(spec));
+  } else {
+// in the case of a queue, we allow concurrent consumers
+for (int i = 0; i < desiredNumSplits; i++) {
+  sources.add(new UnboundedJmsSource(spec));
+}
   }
   return sources;
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/eed4efc5/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
--
diff --git 
a/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java 
b/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
index c756cd0..a06bba3 100644
--- a/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
+++ b/sdks/java/io/jms/src/test/java/org/apache/beam/sdk/io/jms/JmsIOTest.java
@@ -17,6 +17,9 @@
  */
 package org.apache.beam.sdk.io.jms;
 
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
 import java.util.ArrayList;
 import java.util.List;
 
@@ -33,6 +36,8 @@ import org.apache.activemq.broker.BrokerService;
 import org.apache.activemq.security.AuthenticationUser;
 import org.apache.activemq.security.SimpleAuthenticationPlugin;
 import org.apache.activemq.store.memory.MemoryPersistenceAdapter;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.testing.NeedsRunner;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.testing.TestPipeline;
@@ -40,7 +45,6 @@ import org.apache.beam.sdk.transforms.Count;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.values.PCollection;
 import org.junit.After;
-import org.junit.Assert;
 import org.junit.Before;
 import org.junit.Rule;
 import org.junit.Test;
@@ -60,6 +64,7 @@ public class JmsIOTest {
   private static final String USERNAME = "test_user";
   private static final String PASSWORD = "test_password";
   private static final String QUEUE = "test_queue";
+  private static final String TOPIC = "test_topic";
 
   private BrokerService broker;
   private ConnectionFactory connectionFactory;
@@ -167,7 +172,7 @@ public class JmsIOTest {
 session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
 MessageConsumer consumer = 
session.createConsumer(session.createQueue(QUEUE));
 Message msg = consumer.receiveNoWait();
-Assert.assertNull(msg);
+assertNull(msg);
   }
 
   @Test
@@ -195,7 +200,35 @@ public class JmsIOTest {
 while (consumer.receive(1000) != null) {
   count++;
 }
-

[2/2] beam git commit: This closes #1573

[dhalperi]

This closes #1573


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/cef31093
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/cef31093
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/cef31093

Branch: refs/heads/master
Commit: cef31093f7878847eeaf5f747e17de2bb72e34e9
Parents: 8ee3572 eed4efc
Author: Dan Halperin 
Authored: Fri Feb 3 03:28:35 2017 -0800
Committer: Dan Halperin 
Committed: Fri Feb 3 03:28:35 2017 -0800

--
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java  | 15 +++-
 .../org/apache/beam/sdk/io/jms/JmsIOTest.java   | 39 ++--
 2 files changed, 49 insertions(+), 5 deletions(-)
--

Jenkins build is back to normal : beam_PostCommit_Python_Verify #1157

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-823) Improve DatastoreIO Documentation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851323#comment-15851323
 ] 

ASF GitHub Bot commented on BEAM-823:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1293


> Improve DatastoreIO Documentation
> -
>
> Key: BEAM-823
> URL: https://issues.apache.org/jira/browse/BEAM-823
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-gcp
>Affects Versions: 0.3.0-incubating
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>Priority: Minor
>
> Few things to be added to the DatastoreIO documentation,
> 1. Inequality Filter queries are not splittable.
> 2. Clarify Source is Batch only, while Sink support both Batch and Streaming. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1293: [BEAM-823, BEAM-923] Add localhost option for Datas...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1293


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: beam_PostCommit_Python_Verify #1156

[Apache Jenkins Server]

See 

Changes:

[dhalperi] [BEAM-1379] Shade Guava in beam-sdks-java-io-kafka module

--
[...truncated 3807 lines...]
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "write/Write/WriteImpl/Extract.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s13"
}, 
"serialized_fn": "", 
"user_name": "write/Write/WriteImpl/Extract"
  }
}, 
{
  "kind": "CollectionToSingleton", 
  "name": "s15", 
  "properties": {
"display_data": [], 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}
  ]
}, 
"output_name": "out", 
"user_name": 
"write/Write/WriteImpl/ViewAsIterable(write|Write|WriteImpl|Extract.None)/CreatePCollectionView.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s14"
}, 
"user_name": 
"write/Write/WriteImpl/ViewAsIterable(write|Write|WriteImpl|Extract.None)/CreatePCollectionView"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s16", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_finalize_write"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {
  "s15": {

[2/2] beam git commit: This closes #1906

[dhalperi]

This closes #1906


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/8b044945
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/8b044945
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/8b044945

Branch: refs/heads/master
Commit: 8b0449450bd9d50525c10453bc5cd29683848937
Parents: 08e58e1 a183cb9
Author: Dan Halperin 
Authored: Fri Feb 3 01:57:51 2017 -0800
Committer: Dan Halperin 
Committed: Fri Feb 3 01:57:51 2017 -0800

--
 runners/spark/pom.xml  |  4 ++--
 sdks/java/io/kafka/pom.xml | 33 +
 2 files changed, 35 insertions(+), 2 deletions(-)
--

[GitHub] beam pull request #1906: [BEAM-1379] Shade Guava in beam-sdks-java-io-kafka ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1906


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: [BEAM-1379] Shade Guava in beam-sdks-java-io-kafka module

[dhalperi]

Repository: beam
Updated Branches:
  refs/heads/master 08e58e1df -> 8b0449450


[BEAM-1379] Shade Guava in beam-sdks-java-io-kafka module


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a183cb9f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a183cb9f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a183cb9f

Branch: refs/heads/master
Commit: a183cb9fe1f135e1efb9fc1b7b7aa448ff1854b8
Parents: 08e58e1
Author: Aviem Zur 
Authored: Thu Feb 2 17:08:27 2017 +0200
Committer: Dan Halperin 
Committed: Fri Feb 3 01:57:48 2017 -0800

--
 runners/spark/pom.xml  |  4 ++--
 sdks/java/io/kafka/pom.xml | 33 +
 2 files changed, 35 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/a183cb9f/runners/spark/pom.xml
--
diff --git a/runners/spark/pom.xml b/runners/spark/pom.xml
index ffe6dfd..22ab027 100644
--- a/runners/spark/pom.xml
+++ b/runners/spark/pom.xml
@@ -383,11 +383,11 @@
 version used by Hadoop (v11) -->
   
 com.google.common
-
org.apache.beam.spark.relocated.com.google.common
+
org.apache.beam.spark.repackaged.com.google.common
   
   
 com.google.thirdparty
-
org.apache.beam.spark.relocated.com.google.thirdparty
+
org.apache.beam.spark.repackaged.com.google.thirdparty
   
 
 

http://git-wip-us.apache.org/repos/asf/beam/blob/a183cb9f/sdks/java/io/kafka/pom.xml
--
diff --git a/sdks/java/io/kafka/pom.xml b/sdks/java/io/kafka/pom.xml
index 02150b2..65a6095 100644
--- a/sdks/java/io/kafka/pom.xml
+++ b/sdks/java/io/kafka/pom.xml
@@ -40,6 +40,35 @@
 true
   
 
+
+  org.apache.maven.plugins
+  maven-shade-plugin
+  
+
+  package
+  
+shade
+  
+  
+
+  
+com.google.guava:guava
+  
+
+
+  
+com.google.common
+
org.apache.beam.io.kafka.repackaged.com.google.common
+  
+  
+com.google.thirdparty
+
org.apache.beam.io.kafka.repackaged.com.google.thirdparty
+  
+
+  
+
+  
+
   
 
 
@@ -61,6 +90,10 @@
 org.apache.maven.plugins
 maven-jar-plugin
   
+  
+org.apache.maven.plugins
+maven-shade-plugin
+