[jira] [Commented] (BEAM-2836) Apply futurize stage 1 ("safe")

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151410#comment-16151410
 ] 

ASF GitHub Bot commented on BEAM-2836:
--

GitHub user holdenk opened a pull request:

https://github.com/apache/beam/pull/3804

[BEAM-2836] Run the "safe" futurize stage, and then fix the parts which 
parts which weren't "safe"

Run the safe futurize stage, this only broke a few tests (yay), it did 
introduce a number of long-line issues with lambdas mostly fixed and 
refactoring them into regular functions.

The two main things that had to be fixed after stage 1 futurize is expected 
error message in type inference tests and type inference on iterable/tuple 
access.

To keep people from inadvertently breaking things until we have Python 3 
support as part of Jenkins this also updates run_pylint to verify that there 
are no changes suggested by futurize stage 1.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/beam BEAM-2836-futurize-stage-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3804.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3804


commit a5fe79e977d69dbcbd3ca9ba12f94dffb189bb0a
Author: Holden Karau 
Date:   2017-09-02T02:43:06Z

Futurize stage 1 run

commit b924740ceca1bac08652442e0a60a46d36a6d4a8
Author: Holden Karau 
Date:   2017-09-02T02:53:42Z

Add futurize to pylint script

commit 43bb552a54747064b148f7a4e0d6ac9946b813b1
Author: Holden Karau 
Date:   2017-09-02T02:56:11Z

re-order imports

commit a196f1568bd6fdea18f9d49480765fd4f81b47ad
Author: Holden Karau 
Date:   2017-09-02T02:57:19Z

Cleanup whitespace (autopep8 fix

commit 2804f8158c7bbf4c71c1dbb1431b322f5e8334fb
Author: Holden Karau 
Date:   2017-09-02T03:49:40Z

Change the ptransform_test to use assertEqual rather than assertTrue for 
improved debugging, fix the error message we're looking for since we now throw 
a tuple exception instead

commit 6bff4d847129a20ebe4543c223922931cb0c02cc
Author: Holden Karau 
Date:   2017-09-02T03:51:18Z

Add special work to handle indexable return types. Introduce a base 
IndexableTypeConstraint.

commit 190710f6dbe7f9acba1e4318f0b1c8c6801f9906
Author: Holden Karau 
Date:   2017-09-02T03:51:40Z

Break down futurize pylint

commit 7c7c315fc2bdb27669ad4e15e15a345496fc4455
Author: Holden Karau 
Date:   2017-09-02T04:38:47Z

Fix remaining style issues from auto conversion, switch ref in pylint to 
stage 1 (stages in futurize are apparently 1 indexed)

commit 85a27f9db6441fcc0dd552a1731bc84019fe2fd8
Author: Holden Karau 
Date:   2017-09-02T04:43:35Z

Restore license position that got screwed up

commit 2ffee69fd31a1f08c90a7224c0297a4c3c73aec6
Author: Holden Karau 
Date:   2017-09-02T04:53:51Z

Fix the pylint script checking futurize




> Apply futurize stage 1 ("safe")
> ---
>
> Key: BEAM-2836
> URL: https://issues.apache.org/jira/browse/BEAM-2836
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py
>Reporter: holdenk
>Assignee: Ahmet Altay
>
> Futurize has two stages: stage 1 & stage 2. In theory futurize stage 1 should 
> be safe, try and apply stage 1 on its own.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3804: [BEAM-2836] Run the "safe" futurize stage, and then...

[holdenk]

GitHub user holdenk opened a pull request:

https://github.com/apache/beam/pull/3804

[BEAM-2836] Run the "safe" futurize stage, and then fix the parts which 
parts which weren't "safe"

Run the safe futurize stage, this only broke a few tests (yay), it did 
introduce a number of long-line issues with lambdas mostly fixed and 
refactoring them into regular functions.

The two main things that had to be fixed after stage 1 futurize is expected 
error message in type inference tests and type inference on iterable/tuple 
access.

To keep people from inadvertently breaking things until we have Python 3 
support as part of Jenkins this also updates run_pylint to verify that there 
are no changes suggested by futurize stage 1.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/holdenk/beam BEAM-2836-futurize-stage-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3804.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3804


commit a5fe79e977d69dbcbd3ca9ba12f94dffb189bb0a
Author: Holden Karau 
Date:   2017-09-02T02:43:06Z

Futurize stage 1 run

commit b924740ceca1bac08652442e0a60a46d36a6d4a8
Author: Holden Karau 
Date:   2017-09-02T02:53:42Z

Add futurize to pylint script

commit 43bb552a54747064b148f7a4e0d6ac9946b813b1
Author: Holden Karau 
Date:   2017-09-02T02:56:11Z

re-order imports

commit a196f1568bd6fdea18f9d49480765fd4f81b47ad
Author: Holden Karau 
Date:   2017-09-02T02:57:19Z

Cleanup whitespace (autopep8 fix

commit 2804f8158c7bbf4c71c1dbb1431b322f5e8334fb
Author: Holden Karau 
Date:   2017-09-02T03:49:40Z

Change the ptransform_test to use assertEqual rather than assertTrue for 
improved debugging, fix the error message we're looking for since we now throw 
a tuple exception instead

commit 6bff4d847129a20ebe4543c223922931cb0c02cc
Author: Holden Karau 
Date:   2017-09-02T03:51:18Z

Add special work to handle indexable return types. Introduce a base 
IndexableTypeConstraint.

commit 190710f6dbe7f9acba1e4318f0b1c8c6801f9906
Author: Holden Karau 
Date:   2017-09-02T03:51:40Z

Break down futurize pylint

commit 7c7c315fc2bdb27669ad4e15e15a345496fc4455
Author: Holden Karau 
Date:   2017-09-02T04:38:47Z

Fix remaining style issues from auto conversion, switch ref in pylint to 
stage 1 (stages in futurize are apparently 1 indexed)

commit 85a27f9db6441fcc0dd552a1731bc84019fe2fd8
Author: Holden Karau 
Date:   2017-09-02T04:43:35Z

Restore license position that got screwed up

commit 2ffee69fd31a1f08c90a7224c0297a4c3c73aec6
Author: Holden Karau 
Date:   2017-09-02T04:53:51Z

Fix the pylint script checking futurize




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3889

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2836) Apply futurize stage 1 ("safe")

[holdenk (JIRA)]

holdenk created BEAM-2836:
-

 Summary: Apply futurize stage 1 ("safe")
 Key: BEAM-2836
 URL: https://issues.apache.org/jira/browse/BEAM-2836
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py
Reporter: holdenk
Assignee: Ahmet Altay


Futurize has two stages: stage 1 & stage 2. In theory futurize stage 1 should 
be safe, try and apply stage 1 on its own.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2821) isort and autopep8 the current Python code base

[holdenk (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151380#comment-16151380
 ] 

holdenk commented on BEAM-2821:
---

Would it be ok to close this and assign to me then?

> isort and autopep8 the current Python code base
> ---
>
> Key: BEAM-2821
> URL: https://issues.apache.org/jira/browse/BEAM-2821
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py
>Reporter: holdenk
>Assignee: Ahmet Altay
>
> As part of preparing for automated code conversion of the Apache BEAM code 
> base we should pre-sort the imports and apply some basic autopep8 changes. 
> This are useful since we will want to apply them again after futurize.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4702

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4703

[Apache Jenkins Server]

See 


Changes:

[altay] Improve bundle retry display.

--
[...truncated 89.15 KB...]
2017-09-02T00:31:16.341 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/wagon/wagon/1.0-beta-6/wagon-1.0-beta-6.pom
2017-09-02T00:31:16.376 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/wagon/wagon/1.0-beta-6/wagon-1.0-beta-6.pom
 (13 KB at 345.6 KB/sec)
2017-09-02T00:31:16.383 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/11/maven-parent-11.pom
2017-09-02T00:31:16.414 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-parent/11/maven-parent-11.pom
 (32 KB at 1020.9 KB/sec)
2017-09-02T00:31:16.419 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/apache/5/apache-5.pom
2017-09-02T00:31:16.447 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/apache/5/apache-5.pom (5 KB at 
142.9 KB/sec)
[WARNING] Failed to getClass for org.apache.maven.plugins.enforcer.EnforceMojo
[JENKINS] Archiving disabled
2017-09-02T00:31:18.820 [INFO]  
   
2017-09-02T00:31:18.820 [INFO] 

2017-09-02T00:31:18.820 [INFO] Skipping Apache Beam :: Parent
2017-09-02T00:31:18.820 [INFO] This project has been banned from the build due 
to previous failures.
2017-09-02T00:31:18.820 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled2017-09-02T00:31:39.067 [INFO] 

2017-09-02T00:31:39.067 [INFO] Reactor Summary:
2017-09-02T00:31:39.067 [INFO] 
2017-09-02T00:31:39.067 [INFO] Apache Beam :: Parent 
.. FAILURE [ 24.479 s]

2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Java :: Build Tools 
. SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs 
 SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Common 
.. SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Common :: Runner API 
 SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Common :: Fn API 
 SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Java 
 SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Java :: Core 
 SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: Runners 
. SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: Runners :: Core Construction Java 
... SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: Runners :: Core Java 
 SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: Runners :: Direct Java 
.. SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Java :: IO 
.. SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Java :: IO :: AMQP 
.. SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Java :: IO :: Common 
 SKIPPED
2017-09-02T00:31:39.067 [INFO] Apache Beam :: SDKs :: Java :: IO :: Cassandra 
. SKIPPED

[jira] [Commented] (BEAM-2718) Add bundle retry logic to the DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151337#comment-16151337
 ] 

ASF GitHub Bot commented on BEAM-2718:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3796


> Add bundle retry logic to the DirectRunner
> --
>
> Key: BEAM-2718
> URL: https://issues.apache.org/jira/browse/BEAM-2718
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: Not applicable
>
>
> When processing of a bundle fails, the bundle should be retried 3 times (for 
> a total of 4 attempts to process it).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[1/2] beam git commit: Improve bundle retry display.

[altay]

Repository: beam
Updated Branches:
  refs/heads/master 3aa2bef87 -> 2389d56a4


Improve bundle retry display.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/453896a4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/453896a4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/453896a4

Branch: refs/heads/master
Commit: 453896a4646c3fc9f73154b74abbe9f62dd2c33c
Parents: 3aa2bef
Author: Maria Garcia Herrero 
Authored: Wed Aug 30 17:00:13 2017 -0700
Committer: Ahmet Altay 
Committed: Fri Sep 1 17:28:40 2017 -0700

--
 sdks/python/apache_beam/runners/direct/executor.py | 16 +++-
 1 file changed, 11 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/453896a4/sdks/python/apache_beam/runners/direct/executor.py
--
diff --git a/sdks/python/apache_beam/runners/direct/executor.py 
b/sdks/python/apache_beam/runners/direct/executor.py
index 3e08b52..890aa88 100644
--- a/sdks/python/apache_beam/runners/direct/executor.py
+++ b/sdks/python/apache_beam/runners/direct/executor.py
@@ -262,6 +262,8 @@ class TransformExecutor(_ExecutorService.CallableTask):
   completion callback.
   """
 
+  _MAX_RETRY_PER_BUNDLE = 4
+
   def __init__(self, transform_evaluator_registry, evaluation_context,
input_bundle, fired_timers, applied_ptransform,
completion_callback, transform_evaluation_state):
@@ -278,12 +280,11 @@ class TransformExecutor(_ExecutorService.CallableTask):
 self._retry_count = 0
 # Switch to turn on/off the retry of bundles.
 pipeline_options = self._evaluation_context.pipeline_options
+# TODO(mariagh): Remove once "bundle retry" is no longer experimental.
 if not pipeline_options.view_as(DirectOptions).direct_runner_bundle_retry:
   self._max_retries_per_bundle = 1
 else:
-  self._max_retries_per_bundle = 4
-# TODO(mariagh): make _max_retries_per_bundle a constant
-# once "bundle retry" is no longer experimental.
+  self._max_retries_per_bundle = TransformExecutor._MAX_RETRY_PER_BUNDLE
 
   def call(self):
 self._call_count += 1
@@ -312,12 +313,17 @@ class TransformExecutor(_ExecutorService.CallableTask):
 break
   except Exception as e:
 self._retry_count += 1
-logging.info(
-'Exception at bundle %r, due to an exception: %s',
+logging.error(
+'Exception at bundle %r, due to an exception.\n %s',
 self._input_bundle, traceback.format_exc())
 if self._retry_count == self._max_retries_per_bundle:
   logging.error('Giving up after %s attempts.',
 self._max_retries_per_bundle)
+  if self._retry_count == 1:
+logging.info(
+'Use the experimental flag --direct_runner_bundle_retry'
+' to retry failed bundles (up to %d times).',
+TransformExecutor._MAX_RETRY_PER_BUNDLE)
   self._completion_callback.handle_exception(self, e)
 
 self._evaluation_context.metrics().commit_physical(

[jira] [Commented] (BEAM-2833) TestPipeline fail to get Coder

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151235#comment-16151235
 ] 

ASF GitHub Bot commented on BEAM-2833:
--

GitHub user XuMingmin opened a pull request:

https://github.com/apache/beam/pull/3803

[BEAM-2833] TestPipeline fail to get Coder

add option for toProto/fromProto translations in `DirectRunner`, _disabled_ 
by default.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/XuMingmin/beam BEAM-2833

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3803.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3803


commit 95434f6d6bba85c2941555e57766246872960bdf
Author: mingmxu 
Date:   2017-09-01T22:16:19Z

add option for toProto/fromProto translations in DirectRunner, disabled by 
default.




> TestPipeline fail to get Coder
> --
>
> Key: BEAM-2833
> URL: https://issues.apache.org/jira/browse/BEAM-2833
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, testing
>Reporter: Xu Mingmin
> Fix For: 2.2.0
>
>
> See this error when preparing merge branch DSL_SQL to master, it seems caused 
> by https://github.com/apache/beam/pull/3334, and only happen with 
> {{TestPipelin}} but work fine with {{org.apache.beam.sdk.Pipeline}}. 
> More error message can be found in 
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14092/#showFailuresLink.
>  
> {code}
> Error Message
> java.lang.IllegalArgumentException
> Stacktrace
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
>  java.lang.IllegalArgumentException
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:154)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.fromProto(TestStreamTranslation.java:68)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.getTestStream(TestStreamTranslation.java:101)
>   at 
> org.apache.beam.runners.direct.TestStreamEvaluatorFactory$DirectTestStreamFactory.getReplacementTransform(TestStreamEvaluatorFactory.java:179)
>   at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:540)
>   at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
>   at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:169)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:344)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:331)
>   at 
> org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.runSessionWindow(BeamSqlDslAggregationTest.java:367)
>   at 
> org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.testSessionWindowWithBounded(BeamSqlDslAggregationTest.java:338)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> 

[GitHub] beam pull request #3785: [BEAM-2821] Run isort and autopep8 in preparation f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3785


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/4] beam git commit: Isort the libraries

[robertwb]

Isort the libraries


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3151f26c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3151f26c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3151f26c

Branch: refs/heads/master
Commit: 3151f26c028ef7886aaa1adeb64600e07ff0a23a
Parents: 7d53878
Author: Holden Karau 
Authored: Tue Aug 29 12:56:34 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Sep 1 15:07:43 2017 -0700

--
 sdks/python/apache_beam/coders/coder_impl.py|  4 ++--
 sdks/python/apache_beam/coders/coders.py|  3 ++-
 sdks/python/apache_beam/coders/coders_test.py   |  2 +-
 .../apache_beam/coders/coders_test_common.py|  9 
 .../apache_beam/coders/observable_test.py   |  1 -
 .../coders/proto2_coder_test_messages_pb2.py|  4 +++-
 .../apache_beam/coders/standard_coders_test.py  |  6 ++---
 sdks/python/apache_beam/coders/stream_test.py   |  1 -
 sdks/python/apache_beam/coders/typecoders.py|  1 -
 .../examples/complete/estimate_pi.py|  5 ++--
 .../examples/complete/estimate_pi_test.py   |  2 +-
 .../complete/juliaset/juliaset/juliaset_test.py |  1 -
 .../examples/complete/juliaset/juliaset_main.py |  2 --
 .../examples/complete/juliaset/setup.py |  2 +-
 .../apache_beam/examples/complete/tfidf.py  |  2 +-
 .../apache_beam/examples/complete/tfidf_test.py |  1 -
 .../examples/complete/top_wikipedia_sessions.py |  5 ++--
 .../complete/top_wikipedia_sessions_test.py |  1 -
 .../examples/cookbook/bigquery_side_input.py|  5 ++--
 .../cookbook/bigquery_tornadoes_it_test.py  |  2 +-
 .../examples/cookbook/custom_ptransform.py  |  2 +-
 .../examples/cookbook/datastore_wordcount.py|  3 ++-
 .../examples/cookbook/group_with_coder.py   |  4 ++--
 .../examples/cookbook/group_with_coder_test.py  |  1 -
 .../apache_beam/examples/snippets/snippets.py   |  9 
 .../examples/snippets/snippets_test.py  |  6 ++---
 .../apache_beam/examples/streaming_wordcount.py |  3 +--
 .../apache_beam/examples/windowed_wordcount.py  |  1 -
 .../apache_beam/examples/wordcount_it_test.py   |  2 +-
 sdks/python/apache_beam/internal/gcp/auth.py|  1 -
 .../apache_beam/internal/gcp/json_value.py  |  1 -
 .../apache_beam/internal/gcp/json_value_test.py |  3 +--
 sdks/python/apache_beam/internal/util.py|  2 +-
 sdks/python/apache_beam/io/avroio.py|  4 ++--
 sdks/python/apache_beam/io/avroio_test.py   | 21 -
 .../python/apache_beam/io/concat_source_test.py |  1 -
 sdks/python/apache_beam/io/filebasedsink.py |  4 ++--
 .../python/apache_beam/io/filebasedsink_test.py |  3 +--
 sdks/python/apache_beam/io/filebasedsource.py   | 16 ++---
 .../apache_beam/io/filebasedsource_test.py  |  8 +++
 sdks/python/apache_beam/io/filesystem.py|  4 ++--
 sdks/python/apache_beam/io/filesystem_test.py   |  5 ++--
 sdks/python/apache_beam/io/filesystems.py   |  1 -
 sdks/python/apache_beam/io/filesystems_test.py  |  4 ++--
 sdks/python/apache_beam/io/gcp/bigquery.py  |  4 ++--
 sdks/python/apache_beam/io/gcp/bigquery_test.py |  4 ++--
 .../gcp/datastore/v1/adaptive_throttler_test.py |  1 +
 .../io/gcp/datastore/v1/datastoreio.py  | 23 ++-
 .../io/gcp/datastore/v1/datastoreio_test.py |  6 +++--
 .../apache_beam/io/gcp/datastore/v1/helper.py   | 10 +---
 .../io/gcp/datastore/v1/helper_test.py  |  5 ++--
 .../apache_beam/io/gcp/gcsfilesystem_test.py|  1 +
 sdks/python/apache_beam/io/gcp/gcsio.py |  1 +
 .../clients/bigquery/bigquery_v2_messages.py|  3 ---
 .../clients/storage/storage_v1_messages.py  |  3 ---
 sdks/python/apache_beam/io/gcp/pubsub.py|  5 ++--
 sdks/python/apache_beam/io/gcp/pubsub_test.py   |  5 ++--
 .../io/gcp/tests/bigquery_matcher_test.py   |  3 ++-
 .../apache_beam/io/gcp/tests/utils_test.py  |  4 +++-
 sdks/python/apache_beam/io/iobase.py|  7 +++---
 .../apache_beam/io/localfilesystem_test.py  |  4 ++--
 sdks/python/apache_beam/io/source_test_utils.py |  4 ++--
 .../apache_beam/io/source_test_utils_test.py|  2 +-
 sdks/python/apache_beam/io/sources_test.py  |  1 -
 sdks/python/apache_beam/io/textio.py|  5 ++--
 sdks/python/apache_beam/io/textio_test.py   | 18 ++-
 sdks/python/apache_beam/io/tfrecordio.py|  5 ++--
 sdks/python/apache_beam/io/tfrecordio_test.py   |  8 +++
 sdks/python/apache_beam/metrics/cells_test.py   |  2 +-
 sdks/python/apache_beam/metrics/execution.py|  5 ++--
 .../apache_beam/metrics/execution_test.py   |  4 ++--
 sdks/python/apache_beam/metrics/metric.py   |  3 ++-
 sdks/python/apache_beam/metrics/metric_test.py  |  2 +-
 .../apache_beam/options/pipeline_options.py |  5 ++--
 

[4/4] beam git commit: Closes #3785

[robertwb]

Closes #3785


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3aa2bef8
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3aa2bef8
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3aa2bef8

Branch: refs/heads/master
Commit: 3aa2bef87c93d2844dd7c8dbaf45db75ec607792
Parents: 7d53878 0f91f08
Author: Robert Bradshaw 
Authored: Fri Sep 1 15:09:12 2017 -0700
Committer: Robert Bradshaw 
Committed: Fri Sep 1 15:09:12 2017 -0700

--
 sdks/python/apache_beam/coders/coder_impl.py|  4 ++--
 sdks/python/apache_beam/coders/coders.py|  3 ++-
 sdks/python/apache_beam/coders/coders_test.py   |  2 +-
 .../apache_beam/coders/coders_test_common.py|  9 
 .../apache_beam/coders/observable_test.py   |  1 -
 .../coders/proto2_coder_test_messages_pb2.py|  6 ++---
 .../apache_beam/coders/standard_coders_test.py  |  6 ++---
 sdks/python/apache_beam/coders/stream_test.py   |  1 -
 sdks/python/apache_beam/coders/typecoders.py|  1 -
 .../examples/complete/estimate_pi.py|  5 ++--
 .../examples/complete/estimate_pi_test.py   |  2 +-
 .../complete/juliaset/juliaset/juliaset_test.py |  1 -
 .../examples/complete/juliaset/juliaset_main.py |  2 --
 .../examples/complete/juliaset/setup.py |  2 +-
 .../apache_beam/examples/complete/tfidf.py  |  2 +-
 .../apache_beam/examples/complete/tfidf_test.py |  1 -
 .../examples/complete/top_wikipedia_sessions.py |  5 ++--
 .../complete/top_wikipedia_sessions_test.py |  1 -
 .../examples/cookbook/bigquery_side_input.py|  5 ++--
 .../cookbook/bigquery_tornadoes_it_test.py  |  2 +-
 .../examples/cookbook/custom_ptransform.py  |  2 +-
 .../examples/cookbook/datastore_wordcount.py|  3 ++-
 .../examples/cookbook/group_with_coder.py   |  4 ++--
 .../examples/cookbook/group_with_coder_test.py  |  1 -
 .../apache_beam/examples/snippets/snippets.py   | 11 -
 .../examples/snippets/snippets_test.py  |  6 ++---
 .../apache_beam/examples/streaming_wordcount.py |  3 +--
 .../apache_beam/examples/windowed_wordcount.py  |  1 -
 .../apache_beam/examples/wordcount_it_test.py   |  2 +-
 sdks/python/apache_beam/internal/gcp/auth.py|  1 -
 .../apache_beam/internal/gcp/json_value.py  |  1 -
 .../apache_beam/internal/gcp/json_value_test.py |  3 +--
 sdks/python/apache_beam/internal/util.py|  2 +-
 sdks/python/apache_beam/io/avroio.py|  4 ++--
 sdks/python/apache_beam/io/avroio_test.py   | 21 -
 .../python/apache_beam/io/concat_source_test.py |  1 -
 sdks/python/apache_beam/io/filebasedsink.py |  4 ++--
 .../python/apache_beam/io/filebasedsink_test.py |  3 +--
 sdks/python/apache_beam/io/filebasedsource.py   | 16 ++---
 .../apache_beam/io/filebasedsource_test.py  |  8 +++
 sdks/python/apache_beam/io/filesystem.py|  4 ++--
 sdks/python/apache_beam/io/filesystem_test.py   |  5 ++--
 sdks/python/apache_beam/io/filesystems.py   |  1 -
 sdks/python/apache_beam/io/filesystems_test.py  |  4 ++--
 sdks/python/apache_beam/io/gcp/bigquery.py  |  4 ++--
 sdks/python/apache_beam/io/gcp/bigquery_test.py |  4 ++--
 .../gcp/datastore/v1/adaptive_throttler_test.py |  1 +
 .../io/gcp/datastore/v1/datastoreio.py  | 23 ++-
 .../io/gcp/datastore/v1/datastoreio_test.py |  6 +++--
 .../apache_beam/io/gcp/datastore/v1/helper.py   |  9 +---
 .../io/gcp/datastore/v1/helper_test.py  |  5 ++--
 .../apache_beam/io/gcp/gcsfilesystem_test.py|  1 +
 sdks/python/apache_beam/io/gcp/gcsio.py |  1 +
 .../clients/bigquery/bigquery_v2_messages.py|  3 ---
 .../clients/storage/storage_v1_messages.py  |  3 ---
 sdks/python/apache_beam/io/gcp/pubsub.py|  5 ++--
 sdks/python/apache_beam/io/gcp/pubsub_test.py   |  5 ++--
 .../io/gcp/tests/bigquery_matcher_test.py   |  3 ++-
 .../apache_beam/io/gcp/tests/utils_test.py  |  4 +++-
 sdks/python/apache_beam/io/iobase.py|  7 +++---
 .../apache_beam/io/localfilesystem_test.py  |  4 ++--
 sdks/python/apache_beam/io/source_test_utils.py |  4 ++--
 .../apache_beam/io/source_test_utils_test.py|  2 +-
 sdks/python/apache_beam/io/sources_test.py  |  1 -
 sdks/python/apache_beam/io/textio.py|  5 ++--
 sdks/python/apache_beam/io/textio_test.py   | 18 ++-
 sdks/python/apache_beam/io/tfrecordio.py|  5 ++--
 sdks/python/apache_beam/io/tfrecordio_test.py   |  8 +++
 sdks/python/apache_beam/metrics/cells_test.py   |  2 +-
 sdks/python/apache_beam/metrics/execution.py|  5 ++--
 .../apache_beam/metrics/execution_test.py   |  4 ++--
 sdks/python/apache_beam/metrics/metric.py   |  3 ++-
 sdks/python/apache_beam/metrics/metric_test.py  |  2 +-
 .../apache_beam/options/pipeline_options.py |  5 ++--
 

[jira] [Closed] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2624.
--
Resolution: Fixed

> File-based sinks should produce a PCollection of written filenames
> --
>
> Key: BEAM-2624
> URL: https://issues.apache.org/jira/browse/BEAM-2624
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #4700

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2833) TestPipeline fail to get Coder

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151195#comment-16151195
 ] 

Xu Mingmin commented on BEAM-2833:
--

@kenn I can submit a PR to make the round-trip optional, and disabled by 
default if that's a good option.

> TestPipeline fail to get Coder
> --
>
> Key: BEAM-2833
> URL: https://issues.apache.org/jira/browse/BEAM-2833
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, testing
>Reporter: Xu Mingmin
> Fix For: 2.2.0
>
>
> See this error when preparing merge branch DSL_SQL to master, it seems caused 
> by https://github.com/apache/beam/pull/3334, and only happen with 
> {{TestPipelin}} but work fine with {{org.apache.beam.sdk.Pipeline}}. 
> More error message can be found in 
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14092/#showFailuresLink.
>  
> {code}
> Error Message
> java.lang.IllegalArgumentException
> Stacktrace
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
>  java.lang.IllegalArgumentException
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:154)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.fromProto(TestStreamTranslation.java:68)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.getTestStream(TestStreamTranslation.java:101)
>   at 
> org.apache.beam.runners.direct.TestStreamEvaluatorFactory$DirectTestStreamFactory.getReplacementTransform(TestStreamEvaluatorFactory.java:179)
>   at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:540)
>   at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
>   at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:169)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:344)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:331)
>   at 
> org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.runSessionWindow(BeamSqlDslAggregationTest.java:367)
>   at 
> org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.testSessionWindowWithBounded(BeamSqlDslAggregationTest.java:338)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:320)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at 

[jira] [Commented] (BEAM-2833) TestPipeline fail to get Coder

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151171#comment-16151171
 ] 

Xu Mingmin commented on BEAM-2833:
--

Kenn:
Robert's bisect also indicated that the issue lies somewhere within the 
toProto/fromProto translations. The lines you mention are just the flag flip to 
enable those, which represent a lot of functionality. To get a root cause, I'd 
check in a debugger and/or write validation routine for the raw proto.

Orthogonal to finding and addressing the root cause, since there are also some 
performance issues that may or may not be worth solving for 2.2.0 I have 
suggested putting this proto round-trip (and other related bits, until we 
achieve acceptable performance) behind an PipelineOption as an experiment.

> TestPipeline fail to get Coder
> --
>
> Key: BEAM-2833
> URL: https://issues.apache.org/jira/browse/BEAM-2833
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, testing
>Reporter: Xu Mingmin
> Fix For: 2.2.0
>
>
> See this error when preparing merge branch DSL_SQL to master, it seems caused 
> by https://github.com/apache/beam/pull/3334, and only happen with 
> {{TestPipelin}} but work fine with {{org.apache.beam.sdk.Pipeline}}. 
> More error message can be found in 
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14092/#showFailuresLink.
>  
> {code}
> Error Message
> java.lang.IllegalArgumentException
> Stacktrace
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
>  java.lang.IllegalArgumentException
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:154)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.fromProto(TestStreamTranslation.java:68)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.getTestStream(TestStreamTranslation.java:101)
>   at 
> org.apache.beam.runners.direct.TestStreamEvaluatorFactory$DirectTestStreamFactory.getReplacementTransform(TestStreamEvaluatorFactory.java:179)
>   at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:540)
>   at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
>   at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:169)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:344)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:331)
>   at 
> org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.runSessionWindow(BeamSqlDslAggregationTest.java:367)
>   at 
> org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.testSessionWindowWithBounded(BeamSqlDslAggregationTest.java:338)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:320)
>   at 

[jira] [Commented] (BEAM-2833) TestPipeline fail to get Coder

[Xu Mingmin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151169#comment-16151169
 ] 

Xu Mingmin commented on BEAM-2833:
--

>From Robert:
I did a git bisect and this again looks like it's due to
https://github.com/apache/beam/pull/3334 , specifically commit
8ca45915693839edb14f824fa6835ebe3e67 (which enables the round trip
through protos on Pipeline.run(). Somehow a coder is not getting
properly serialized/stored. Don't know why SQL would trigger this
though, perhaps it's something with the BeamRecordCoder? (Perhaps that
we have several of them that are not actually equal?)

> TestPipeline fail to get Coder
> --
>
> Key: BEAM-2833
> URL: https://issues.apache.org/jira/browse/BEAM-2833
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core, testing
>Reporter: Xu Mingmin
> Fix For: 2.2.0
>
>
> See this error when preparing merge branch DSL_SQL to master, it seems caused 
> by https://github.com/apache/beam/pull/3334, and only happen with 
> {{TestPipelin}} but work fine with {{org.apache.beam.sdk.Pipeline}}. 
> More error message can be found in 
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14092/#showFailuresLink.
>  
> {code}
> Error Message
> java.lang.IllegalArgumentException
> Stacktrace
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
>  java.lang.IllegalArgumentException
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:154)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.fromProto(TestStreamTranslation.java:68)
>   at 
> org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.getTestStream(TestStreamTranslation.java:101)
>   at 
> org.apache.beam.runners.direct.TestStreamEvaluatorFactory$DirectTestStreamFactory.getReplacementTransform(TestStreamEvaluatorFactory.java:179)
>   at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:540)
>   at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
>   at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
>   at 
> org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:169)
>   at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
>   at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:344)
>   at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:331)
>   at 
> org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.runSessionWindow(BeamSqlDslAggregationTest.java:367)
>   at 
> org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.testSessionWindowWithBounded(BeamSqlDslAggregationTest.java:338)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
>   at 
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:320)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> 

[jira] [Commented] (BEAM-2831) Possible bug in Beam+Flink memory management, disk spillover

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151162#comment-16151162
 ] 

Kenneth Knowles commented on BEAM-2831:
---

I think for the most part there's no implications. I have no immediate 
objection based on any principles. The choice between wrapping an IOException 
into a CoderException versus not doing so is not clear in many cases. In 
particular, EOF and parse error are not really distinct errors, since malformed 
data causes the EOF.

This may violate our rigid backwards compatibility requirements, though it 
probably affects zero actual users. Especially since they have to handle all 
IOExceptions anyhow, and because of the semantic ambiguity above, basically 
have to take the same action too.

We have 100% rigidity on type errors and leaving working code working, but for 
bugs and places where the behavior is not well-defined (like what kind of 
exception a particular coder throws in particular situations) maybe there's 
flexibility...

> Possible bug in Beam+Flink memory management, disk spillover
> 
>
> Key: BEAM-2831
> URL: https://issues.apache.org/jira/browse/BEAM-2831
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
> Environment: Flink 1.2.1 and 1.3.0, Java HotSpot and OpenJDK 8, macOS 
> 10.12.6 and unknown Linux
>Reporter: Reinier Kip
>Assignee: Aljoscha Krettek
>
> I’ve been running a Beam pipeline on Flink. Depending on the dataset size and 
> the heap memory configuration of the jobmanager and taskmanager, I may run 
> into an EOFException, which causes the job to fail.
> As [discussed on Flink's 
> mailinglist|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/EOFException-related-to-memory-segments-during-run-of-Beam-pipeline-on-Flink-td15255.html]
>  (stacktrace enclosed), Flink catches these EOFExceptions and activates disk 
> spillover. Because Beam wraps these exceptions, this mechanism fails, the 
> exception travels up the stack, and the job aborts.
> Hopefully this is enough information and this is something that can be 
> adjusted for in Beam. I'd be glad to provide more information where needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (BEAM-2833) TestPipeline fail to get Coder

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2833:
-
Description: 
See this error when preparing merge branch DSL_SQL to master, it seems caused 
by https://github.com/apache/beam/pull/3334, and only happen with 
{{TestPipelin}} but work fine with {{org.apache.beam.sdk.Pipeline}}. 

More error message can be found in 
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14092/#showFailuresLink.
 

{code}
Error Message

java.lang.IllegalArgumentException

Stacktrace

org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
 java.lang.IllegalArgumentException
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.java.repackaged.com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986)
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.RehydratedComponents.getCoder(RehydratedComponents.java:154)
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.fromProto(TestStreamTranslation.java:68)
at 
org.apache.beam.runners.direct.repackaged.runners.core.construction.TestStreamTranslation.getTestStream(TestStreamTranslation.java:101)
at 
org.apache.beam.runners.direct.TestStreamEvaluatorFactory$DirectTestStreamFactory.getReplacementTransform(TestStreamEvaluatorFactory.java:179)
at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:540)
at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
at 
org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:169)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:62)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:344)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:331)
at 
org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.runSessionWindow(BeamSqlDslAggregationTest.java:367)
at 
org.apache.beam.sdk.extensions.sql.BeamSqlDslAggregationTest.testSessionWindowWithBounded(BeamSqlDslAggregationTest.java:338)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:320)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #3885

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2624) File-based sinks should produce a PCollection of written filenames

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151100#comment-16151100
 ] 

ASF GitHub Bot commented on BEAM-2624:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3573


> File-based sinks should produce a PCollection of written filenames
> --
>
> Key: BEAM-2624
> URL: https://issues.apache.org/jira/browse/BEAM-2624
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
> Fix For: 2.2.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3573: [BEAM-2624] Allow access to created filenames from ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3573


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3573: [BEAM-2624] Allow access to created filenames from WriteFiles

[jkff]

This closes #3573: [BEAM-2624] Allow access to created filenames from WriteFiles


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/7d538787
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/7d538787
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/7d538787

Branch: refs/heads/master
Commit: 7d53878768757ef2115170a5073b99956e924ff2
Parents: b844126 73535da
Author: Eugene Kirpichov 
Authored: Fri Sep 1 13:18:08 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Sep 1 13:18:08 2017 -0700

--
 .../construction/WriteFilesTranslation.java |  36 +++--
 .../construction/PTransformMatchersTest.java|  15 +-
 .../construction/WriteFilesTranslationTest.java |   8 +-
 .../direct/WriteWithShardingFactory.java|  26 ++--
 .../direct/WriteWithShardingFactoryTest.java|  11 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  17 ++-
 .../runners/dataflow/DataflowRunnerTest.java|   4 +-
 .../java/org/apache/beam/sdk/io/AvroIO.java | 145 ++
 .../org/apache/beam/sdk/io/FileBasedSink.java   |   8 +-
 .../java/org/apache/beam/sdk/io/TFRecordIO.java |   3 +-
 .../java/org/apache/beam/sdk/io/TextIO.java | 148 +++
 .../java/org/apache/beam/sdk/io/WriteFiles.java | 115 --
 .../apache/beam/sdk/io/WriteFilesResult.java|  81 ++
 .../java/org/apache/beam/sdk/io/AvroIOTest.java |   2 +-
 .../apache/beam/sdk/io/FileBasedSinkTest.java   |   2 +-
 .../org/apache/beam/sdk/io/TextIOWriteTest.java |  40 -
 .../java/org/apache/beam/sdk/io/xml/XmlIO.java  |   3 +-
 17 files changed, 438 insertions(+), 226 deletions(-)
--

[1/2] beam git commit: Support new output type on WriteFiles allowing access to all produced filenames.

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master b844126c8 -> 7d5387876


Support new output type on WriteFiles allowing access to all produced filenames.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/73535da5
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/73535da5
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/73535da5

Branch: refs/heads/master
Commit: 73535da5e99ee9099e3f28f65300b4ddf6399566
Parents: b844126
Author: Reuven Lax 
Authored: Mon Jul 17 10:25:41 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Sep 1 13:18:01 2017 -0700

--
 .../construction/WriteFilesTranslation.java |  36 +++--
 .../construction/PTransformMatchersTest.java|  15 +-
 .../construction/WriteFilesTranslationTest.java |   8 +-
 .../direct/WriteWithShardingFactory.java|  26 ++--
 .../direct/WriteWithShardingFactoryTest.java|  11 +-
 .../beam/runners/dataflow/DataflowRunner.java   |  17 ++-
 .../runners/dataflow/DataflowRunnerTest.java|   4 +-
 .../java/org/apache/beam/sdk/io/AvroIO.java | 145 ++
 .../org/apache/beam/sdk/io/FileBasedSink.java   |   8 +-
 .../java/org/apache/beam/sdk/io/TFRecordIO.java |   3 +-
 .../java/org/apache/beam/sdk/io/TextIO.java | 148 +++
 .../java/org/apache/beam/sdk/io/WriteFiles.java | 115 --
 .../apache/beam/sdk/io/WriteFilesResult.java|  81 ++
 .../java/org/apache/beam/sdk/io/AvroIOTest.java |   2 +-
 .../apache/beam/sdk/io/FileBasedSinkTest.java   |   2 +-
 .../org/apache/beam/sdk/io/TextIOWriteTest.java |  40 -
 .../java/org/apache/beam/sdk/io/xml/XmlIO.java  |   3 +-
 17 files changed, 438 insertions(+), 226 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/73535da5/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java
--
diff --git 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java
 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java
index aeefd4f..a9601bc 100644
--- 
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java
+++ 
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/WriteFilesTranslation.java
@@ -39,12 +39,12 @@ import 
org.apache.beam.sdk.common.runner.v1.RunnerApi.SideInput;
 import org.apache.beam.sdk.common.runner.v1.RunnerApi.WriteFilesPayload;
 import org.apache.beam.sdk.io.FileBasedSink;
 import org.apache.beam.sdk.io.WriteFiles;
+import org.apache.beam.sdk.io.WriteFilesResult;
 import org.apache.beam.sdk.runners.AppliedPTransform;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.util.SerializableUtils;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
-import org.apache.beam.sdk.values.PDone;
 import org.apache.beam.sdk.values.TupleTag;
 
 /**
@@ -104,19 +104,21 @@ public class WriteFilesTranslation {
   }
 
   public static  FileBasedSink getSink(
-  AppliedPTransform>
+  AppliedPTransform<
+  PCollection, WriteFilesResult,
+  ? extends PTransform>
   transform)
   throws IOException {
 return (FileBasedSink)
 sinkFromProto(getWriteFilesPayload(transform).getSink());
   }
 
-  public static 
-  List getDynamicDestinationSideInputs(
-  AppliedPTransform<
-  PCollection, PDone, ? extends 
PTransform>
-  transform)
-  throws IOException {
+  public static  List 
getDynamicDestinationSideInputs(
+  AppliedPTransform<
+  PCollection, WriteFilesResult,
+  ? extends PTransform>
+  transform)
+  throws IOException {
 SdkComponents sdkComponents = SdkComponents.create();
 RunnerApi.PTransform transformProto = 
PTransformTranslation.toProto(transform, sdkComponents);
 List views = Lists.newArrayList();
@@ -138,22 +140,28 @@ public class WriteFilesTranslation {
 return views;
   }
 
-  public static  boolean isWindowedWrites(
-  AppliedPTransform>
+  public static  boolean isWindowedWrites(
+  

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3884

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4699

[Apache Jenkins Server]

See 


Changes:

[ekirpichov] [BEAM-2802] Support multi-byte custom separator in TextIO

--
[...truncated 1.12 MB...]
2017-09-01T19:38:21.532 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/fusesource/sigar/1.6.4/sigar-1.6.4.jar
2017-09-01T19:38:21.548 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/jbellis/jamm/0.3.0/jamm-0.3.0.jar
 (21 KB at 3.7 KB/sec)
2017-09-01T19:38:21.548 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.jar
2017-09-01T19:38:21.616 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/fusesource/sigar/1.6.4/sigar-1.6.4.jar 
(419 KB at 73.8 KB/sec)
2017-09-01T19:38:21.616 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/caffinitas/ohc/ohc-core/0.4.3/ohc-core-0.4.3.jar
2017-09-01T19:38:21.676 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/caffinitas/ohc/ohc-core/0.4.3/ohc-core-0.4.3.jar
 (125 KB at 21.6 KB/sec)
2017-09-01T19:38:21.676 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/github/ben-manes/caffeine/caffeine/2.2.6/caffeine-2.2.6.jar
2017-09-01T19:38:21.884 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/eclipse/jdt/core/compiler/ecj/4.4.2/ecj-4.4.2.jar
 (2257 KB at 379.7 KB/sec)
2017-09-01T19:38:21.884 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/datastax/cassandra/cassandra-driver-core/3.1.1/cassandra-driver-core-3.1.1.jar
2017-09-01T19:38:21.921 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/ben-manes/caffeine/caffeine/2.2.6/caffeine-2.2.6.jar
 (926 KB at 154.8 KB/sec)
2017-09-01T19:38:21.934 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/io/netty/netty-all/4.0.39.Final/netty-all-4.0.39.Final.jar
 (2219 KB at 370.2 KB/sec)
2017-09-01T19:38:22.058 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/datastax/cassandra/cassandra-driver-core/3.1.1/cassandra-driver-core-3.1.1.jar
 (1029 KB at 168.1 KB/sec)
2017-09-01T19:38:22.546 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/storm/storm-core/1.0.1/storm-core-1.0.1.jar
 (19650 KB at 2975.4 KB/sec)
2017-09-01T19:38:22.597 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/it/unimi/dsi/fastutil/6.5.7/fastutil-6.5.7.jar
 (16508 KB at 2480.5 KB/sec)
2017-09-01T19:38:22.638 [INFO] Downloading: 
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
2017-09-01T19:38:22.638 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-core/2.6.3/cascading-core-2.6.3.jar
2017-09-01T19:38:22.638 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-hadoop/2.6.3/cascading-hadoop-2.6.3.jar
2017-09-01T19:38:22.638 [INFO] Downloading: 
http://conjars.org/repo/org/apache/spark/spark-sql_2.10/2.0.0/spark-sql_2.10-2.0.0.jar
2017-09-01T19:38:22.639 [INFO] Downloading: 
http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.jar
2017-09-01T19:38:22.697 [INFO] Downloading: 
http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
2017-09-01T19:38:22.753 [INFO] Downloaded: 
http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.jar (12 KB at 96.4 
KB/sec)
2017-09-01T19:38:22.753 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-local/2.6.3/cascading-local-2.6.3.jar
2017-09-01T19:38:22.809 [INFO] Downloaded: 
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
 (48 KB at 277.3 KB/sec)
2017-09-01T19:38:22.919 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-local/2.6.3/cascading-local-2.6.3.jar
 (43 KB at 151.2 KB/sec)
2017-09-01T19:38:22.997 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-hadoop/2.6.3/cascading-hadoop-2.6.3.jar
 (246 KB at 684.9 KB/sec)
2017-09-01T19:38:23.084 [INFO] Downloaded: 
http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
 (230 KB at 515.2 KB/sec)
2017-09-01T19:38:23.247 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-core/2.6.3/cascading-core-2.6.3.jar 
(680 KB at 1115.7 KB/sec)
2017-09-01T19:38:23.250 [INFO] Downloading: 
http://clojars.org/repo/org/apache/spark/spark-sql_2.10/2.0.0/spark-sql_2.10-2.0.0.jar
[JENKINS] Archiving disabled
2017-09-01T19:38:24.045 [INFO]  
   
2017-09-01T19:38:24.045 [INFO] 

2017-09-01T19:38:24.045 [INFO] Skipping Apache Beam :: Parent
2017-09-01T19:38:24.045 [INFO] This project has been banned from the build due 
to previous failures.
2017-09-01T19:38:24.045 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] 

[jira] [Closed] (BEAM-2802) TextIO should allow specifying a custom delimiter

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2802.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

> TextIO should allow specifying a custom delimiter
> -
>
> Key: BEAM-2802
> URL: https://issues.apache.org/jira/browse/BEAM-2802
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Minor
> Fix For: 2.2.0
>
>
> Currently TextIO use {{\r}} {{\n}} or {{\r\n}} or a mix of the two to split a 
> text file into PCollection elements. It might happen that a record is spread 
> across more than one line. In that case we should be able to specify a custom 
> record delimiter to be used in place of the default ones.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2802) TextIO should allow specifying a custom delimiter

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151029#comment-16151029
 ] 

ASF GitHub Bot commented on BEAM-2802:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3779


> TextIO should allow specifying a custom delimiter
> -
>
> Key: BEAM-2802
> URL: https://issues.apache.org/jira/browse/BEAM-2802
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Minor
>
> Currently TextIO use {{\r}} {{\n}} or {{\r\n}} or a mix of the two to split a 
> text file into PCollection elements. It might happen that a record is spread 
> across more than one line. In that case we should be able to specify a custom 
> record delimiter to be used in place of the default ones.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3779: [BEAM-2802] Support multi-byte custom separator in ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3779


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/2] beam git commit: This closes #3779: [BEAM-2802] Support multi-byte custom separator in TextIO

[jkff]

This closes #3779: [BEAM-2802] Support multi-byte custom separator in TextIO


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b844126c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b844126c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b844126c

Branch: refs/heads/master
Commit: b844126c83a274626c1353d0837f1dbc7b664d2d
Parents: c9653f2 1b6cde0
Author: Eugene Kirpichov 
Authored: Fri Sep 1 11:51:23 2017 -0700
Committer: Eugene Kirpichov 
Committed: Fri Sep 1 11:51:23 2017 -0700

--
 .../java/org/apache/beam/sdk/io/TextIO.java |  69 +--
 .../java/org/apache/beam/sdk/io/TextSource.java | 117 ---
 .../org/apache/beam/sdk/io/TextIOReadTest.java  |  88 +++---
 3 files changed, 207 insertions(+), 67 deletions(-)
--

[1/2] beam git commit: [BEAM-2802] Support multi-byte custom separator in TextIO

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master c9653f270 -> b844126c8


[BEAM-2802] Support multi-byte custom separator in TextIO

Supports only separators that can not self-overlap,
because self-overlapping separators cause ambiguous parsing.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1b6cde06
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1b6cde06
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1b6cde06

Branch: refs/heads/master
Commit: 1b6cde067ce78e1ce780b66e0cf1c883ce901959
Parents: c9653f2
Author: Etienne Chauchot 
Authored: Fri Aug 25 17:23:51 2017 +0200
Committer: Eugene Kirpichov 
Committed: Fri Sep 1 11:51:17 2017 -0700

--
 .../java/org/apache/beam/sdk/io/TextIO.java |  69 +--
 .../java/org/apache/beam/sdk/io/TextSource.java | 117 ---
 .../org/apache/beam/sdk/io/TextIOReadTest.java  |  88 +++---
 3 files changed, 207 insertions(+), 67 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1b6cde06/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
index 76102cb..7832168 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
@@ -27,6 +27,9 @@ import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Predicates;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Lists;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
 import java.util.List;
 import javax.annotation.Nullable;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -63,7 +66,8 @@ import org.joda.time.Duration;
  * PCollection}, apply {@link TextIO#readAll()}.
  *
  * {@link #read} returns a {@link PCollection} of {@link String Strings}, 
each corresponding to
- * one line of an input UTF-8 text file (split into lines delimited by '\n', 
'\r', or '\r\n').
+ * one line of an input UTF-8 text file (split into lines delimited by '\n', 
'\r', or '\r\n',
+ * or specified delimiter see {@link TextIO.Read#withDelimiter}).
  *
  * Filepattern expansion and watching
  *
@@ -255,7 +259,8 @@ public class TextIO {
   /** Implementation of {@link #read}. */
   @AutoValue
   public abstract static class Read extends PTransform {
-@Nullable abstract ValueProvider getFilepattern();
+@Nullable
+abstract ValueProvider getFilepattern();
 abstract Compression getCompression();
 
 @Nullable
@@ -266,6 +271,8 @@ public class TextIO {
 
 abstract boolean getHintMatchesManyFiles();
 abstract EmptyMatchTreatment getEmptyMatchTreatment();
+@Nullable
+abstract byte[] getDelimiter();
 
 abstract Builder toBuilder();
 
@@ -278,6 +285,7 @@ public class TextIO {
   TerminationCondition condition);
   abstract Builder setHintMatchesManyFiles(boolean hintManyFiles);
   abstract Builder setEmptyMatchTreatment(EmptyMatchTreatment treatment);
+  abstract Builder setDelimiter(byte[] delimiter);
 
   abstract Read build();
 }
@@ -360,6 +368,25 @@ public class TextIO {
   return toBuilder().setEmptyMatchTreatment(treatment).build();
 }
 
+/**
+ * Set the custom delimiter to be used in place of the default ones ('\r', 
'\n' or '\r\n').
+ */
+public Read withDelimiter(byte[] delimiter) {
+  checkArgument(delimiter != null, "delimiter can not be null");
+  checkArgument(!isSelfOverlapping(delimiter), "delimiter must not 
self-overlap");
+  return toBuilder().setDelimiter(delimiter).build();
+}
+
+static boolean isSelfOverlapping(byte[] s) {
+  // s self-overlaps if v exists such as s = vu = wv with u and w non empty
+  for (int i = 1; i < s.length - 1; ++i) {
+if (ByteBuffer.wrap(s, 0, i).equals(ByteBuffer.wrap(s, s.length - i, 
i))) {
+  return true;
+}
+  }
+  return false;
+}
+
 @Override
 public PCollection expand(PBegin input) {
   checkNotNull(getFilepattern(), "need to set the filepattern of a 
TextIO.Read transform");
@@ -370,7 +397,8 @@ public class TextIO {
   ReadAll readAll =
   readAll()
   .withCompression(getCompression())
-  .withEmptyMatchTreatment(getEmptyMatchTreatment());
+  .withEmptyMatchTreatment(getEmptyMatchTreatment())
+  .withDelimiter(getDelimiter());
   if (getWatchForNewFilesInterval() != null) {
 TerminationCondition readAllCondition =
 

[jira] [Updated] (BEAM-2834) NullPointerException @ BigQueryServicesImpl.java:759

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov updated BEAM-2834:
---
Fix Version/s: 2.2.0

> NullPointerException @ BigQueryServicesImpl.java:759
> 
>
> Key: BEAM-2834
> URL: https://issues.apache.org/jira/browse/BEAM-2834
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.1.0
>Reporter: Andy Barron
>Assignee: Thomas Groh
> Fix For: 2.2.0
>
>
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:759)
>   at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:809)
>   at 
> org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:126)
>   at 
> org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:96)
> {code}
> Going through the stack trace, the likely culprit is a null {{retryPolicy}} 
> in {{StreamingWriteFn}}.
> For context, this error showed up about 70 times between 1 am and 1 pm 
> Pacific time (2017-08-31) on a Dataflow streaming job.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2834) NullPointerException @ BigQueryServicesImpl.java:759

[Reuven Lax (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151017#comment-16151017
 ] 

Reuven Lax commented on BEAM-2834:
--

Thanks for the bug report! This does appear to be a bug. It won't cause any 
data loss in most runners (the input records will simply retry), however it 
will cause spurious crashes.

For now you can always call 
withFailedInsertRetryPolicy(InsertRetryPolicy.alwaysRetry()) to work around (in 
fact the intended behavior was for the default to be this instead of null).

> NullPointerException @ BigQueryServicesImpl.java:759
> 
>
> Key: BEAM-2834
> URL: https://issues.apache.org/jira/browse/BEAM-2834
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.1.0
>Reporter: Andy Barron
>Assignee: Thomas Groh
>
> {code}
> java.lang.NullPointerException
>   at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:759)
>   at 
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:809)
>   at 
> org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:126)
>   at 
> org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:96)
> {code}
> Going through the stack trace, the likely culprit is a null {{retryPolicy}} 
> in {{StreamingWriteFn}}.
> For context, this error showed up about 70 times between 1 am and 1 pm 
> Pacific time (2017-08-31) on a Dataflow streaming job.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2822) Add support for progress reporting in fn API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150870#comment-16150870
 ] 

ASF GitHub Bot commented on BEAM-2822:
--

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3802

[BEAM-2822] Add fn API progress reporting protos

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam progress_proto

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3802.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3802


commit a953356674adbfcfc97be019c8cc494ef94d2210
Author: Vikas Kedigehalli 
Date:   2017-09-01T16:45:22Z

Add fn API progress reporting protos




> Add support for progress reporting in fn API
> 
>
> Key: BEAM-2822
> URL: https://issues.apache.org/jira/browse/BEAM-2822
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Vikas Kedigehalli
>Assignee: Vikas Kedigehalli
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3802: [BEAM-2822] Add fn API progress reporting protos

[vikkyrk]

GitHub user vikkyrk opened a pull request:

https://github.com/apache/beam/pull/3802

[BEAM-2822] Add fn API progress reporting protos

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vikkyrk/incubator-beam progress_proto

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3802.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3802


commit a953356674adbfcfc97be019c8cc494ef94d2210
Author: Vikas Kedigehalli 
Date:   2017-09-01T16:45:22Z

Add fn API progress reporting protos




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #3528: [trivial] pom.xml: add license-maven-plugin and som...

[dhalperi]

Github user dhalperi closed the pull request at:

https://github.com/apache/beam/pull/3528


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-2831) Possible bug in Beam+Flink memory management, disk spillover

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150656#comment-16150656
 ] 

Aljoscha Krettek commented on BEAM-2831:


[~kenn] Do you think we can change {{SerializableCoder}} to pass through the 
exception instead of wrapping it in a {{CoderException}}? Not sure what 
implications this could have for other runners.

> Possible bug in Beam+Flink memory management, disk spillover
> 
>
> Key: BEAM-2831
> URL: https://issues.apache.org/jira/browse/BEAM-2831
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
> Environment: Flink 1.2.1 and 1.3.0, Java HotSpot and OpenJDK 8, macOS 
> 10.12.6 and unknown Linux
>Reporter: Reinier Kip
>Assignee: Aljoscha Krettek
>
> I’ve been running a Beam pipeline on Flink. Depending on the dataset size and 
> the heap memory configuration of the jobmanager and taskmanager, I may run 
> into an EOFException, which causes the job to fail.
> As [discussed on Flink's 
> mailinglist|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/EOFException-related-to-memory-segments-during-run-of-Beam-pipeline-on-Flink-td15255.html]
>  (stacktrace enclosed), Flink catches these EOFExceptions and activates disk 
> spillover. Because Beam wraps these exceptions, this mechanism fails, the 
> exception travels up the stack, and the job aborts.
> Hopefully this is enough information and this is something that can be 
> adjusted for in Beam. I'd be glad to provide more information where needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3801: ElasticsearchIO: Fix Null Pointer Exception in keys...

[echauchot]

GitHub user echauchot opened a pull request:

https://github.com/apache/beam/pull/3801

ElasticsearchIO: Fix Null Pointer Exception in keystore management

When keystorePath is provided by the user but not keystorePassword, then 
keystorePassword == null, generating a Null Pointer Exception

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [X] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [X] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [X] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [X] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
R: @jbonofre 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/echauchot/beam ESIO-null-pointer-keystore

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3801.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3801


commit 0cefc8a189c1178f4d1cb4aa90ee757a9a164a00
Author: Etienne Chauchot 
Date:   2017-09-01T14:31:32Z

ElasticsearchIO: Fix Null Pointer Exception in keystore management

When keystorePath is provided by the user but not keystorePassword, then 
keystorePassword == null, generating a Null Pointer Exception




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (BEAM-2835) Support executing MapReduce jobs in parallel.

[Pei He (JIRA)]

Pei He created BEAM-2835:


 Summary: Support executing MapReduce jobs in parallel.
 Key: BEAM-2835
 URL: https://issues.apache.org/jira/browse/BEAM-2835
 Project: Beam
  Issue Type: New Feature
  Components: runner-mapreduce
Reporter: Pei He


Currently, the runner executes MR jobs sequentially with the topological order. 
This is very inefficient, it will be good to allow jobs to be executed in 
parallel if they don't depend on either other.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (BEAM-2831) Possible bug in Beam+Flink memory management, disk spillover

[Reinier Kip (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149496#comment-16149496
 ] 

Reinier Kip edited comment on BEAM-2831 at 9/1/17 8:56 AM:
---

[~aljoscha] Yes! The pipeline consistently succeeds with this change. Thanks 
for your effort. I suspect we will intregrate this replacement coder while we 
wait for the fix to come out.


was (Author: rjkip):
[~aljoscha] Yes! The pipeline consistently succeeds with this change. Thanks 
for your effort. I suspect we will intregrate this replacement coder while we 
wait for the fix to come out. Do you foresee any consequences due to the 
considerations you wrote near the bottom?

> Possible bug in Beam+Flink memory management, disk spillover
> 
>
> Key: BEAM-2831
> URL: https://issues.apache.org/jira/browse/BEAM-2831
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 2.0.0, 2.1.0
> Environment: Flink 1.2.1 and 1.3.0, Java HotSpot and OpenJDK 8, macOS 
> 10.12.6 and unknown Linux
>Reporter: Reinier Kip
>Assignee: Aljoscha Krettek
>
> I’ve been running a Beam pipeline on Flink. Depending on the dataset size and 
> the heap memory configuration of the jobmanager and taskmanager, I may run 
> into an EOFException, which causes the job to fail.
> As [discussed on Flink's 
> mailinglist|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/EOFException-related-to-memory-segments-during-run-of-Beam-pipeline-on-Flink-td15255.html]
>  (stacktrace enclosed), Flink catches these EOFExceptions and activates disk 
> spillover. Because Beam wraps these exceptions, this mechanism fails, the 
> exception travels up the stack, and the job aborts.
> Hopefully this is enough information and this is something that can be 
> adjusted for in Beam. I'd be glad to provide more information where needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)