[jira] [Closed] (BEAM-2739) improve JavaDoc in BeamSqlEnv

2017-08-12 Thread Xu Mingmin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2739.

   Resolution: Fixed
 Assignee: James Xu
Fix Version/s: Not applicable

> improve JavaDoc in BeamSqlEnv
> -
>
> Key: BEAM-2739
> URL: https://issues.apache.org/jira/browse/BEAM-2739
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
> Fix For: Not applicable
>
>
> Refer to :
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfE
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfM



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Closed] (BEAM-2746) change visibility of BeamSqlTable

2017-08-12 Thread Xu Mingmin (JIRA) 

 [ 
https://issues.apache.org/jira/browse/BEAM-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin closed BEAM-2746.

   Resolution: Fixed
Fix Version/s: Not applicable

Covered by BEAM-2740

> change visibility of BeamSqlTable
> -
>
> Key: BEAM-2746
> URL: https://issues.apache.org/jira/browse/BEAM-2746
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
> Fix For: Not applicable
>
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhgQ



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2739) improve JavaDoc in BeamSqlEnv

2017-08-12 Thread James Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124790#comment-16124790
 ] 

James Xu commented on BEAM-2739:


BeamSqlEnv will be moved to `.impl` in https://github.com/apache/beam/pull/3708

> improve JavaDoc in BeamSqlEnv
> -
>
> Key: BEAM-2739
> URL: https://issues.apache.org/jira/browse/BEAM-2739
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>  Labels: dsl_sql_merge, dsl_sql_review
>
> Refer to :
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfE
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhfM



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2746) change visibility of BeamSqlTable

2017-08-12 Thread James Xu (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124789#comment-16124789
 ] 

James Xu commented on BEAM-2746:


Let's merge it into BEAM-2740, it will be covered by 
https://github.com/apache/beam/pull/3708

> change visibility of BeamSqlTable
> -
>
> Key: BEAM-2746
> URL: https://issues.apache.org/jira/browse/BEAM-2746
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: James Xu
>  Labels: dsl_sql_merge, dsl_sql_review
>
> refer to 
> https://docs.google.com/document/d/1VlAqKLhQua6YpR42KuziIYHTB1FfQTi16Mh88MEYBsU/edit?disco=BTmjhgQ



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3759

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #4565

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2820

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3758

2017-08-12 Thread Apache Jenkins Server 
See

[GitHub] beam pull request #3679: [BEAM-2718] Add initial bundle retry code for the D...

2017-08-12 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3679


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Add initial bundle retry code

2017-08-12 Thread altay 
Repository: beam
Updated Branches:
  refs/heads/master 84a23793c -> b0b642182


Add initial bundle retry code


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1f2a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1f2a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1f2a

Branch: refs/heads/master
Commit: 1f2abf541b88f01b17aa9a549081a8607bb9
Parents: 84a2379
Author: Maria Garcia Herrero 
Authored: Thu Aug 3 00:16:54 2017 -0700
Committer: Ahmet Altay 
Committed: Sat Aug 12 18:06:08 2017 -0700

--
 .../apache_beam/options/pipeline_options.py |   7 ++
 sdks/python/apache_beam/pipeline_test.py|  30 ++
 .../apache_beam/runners/direct/executor.py  | 100 ---
 3 files changed, 100 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/1f2a/sdks/python/apache_beam/options/pipeline_options.py
--
diff --git a/sdks/python/apache_beam/options/pipeline_options.py 
b/sdks/python/apache_beam/options/pipeline_options.py
index ea996a3..db65b3c 100644
--- a/sdks/python/apache_beam/options/pipeline_options.py
+++ b/sdks/python/apache_beam/options/pipeline_options.py
@@ -314,6 +314,13 @@ class DirectOptions(PipelineOptions):
 help='DirectRunner uses stacked WindowedValues within a Bundle for '
 'memory optimization. Set --no_direct_runner_use_stacked_bundle to '
 'avoid it.')
+parser.add_argument(
+'--direct_runner_bundle_retry',
+action='store_true',
+default=False,
+help=
+('Whether to allow bundle retries. If True the maximum'
+ 'number of attempts to process a bundle is 4. '))
 
 
 class GoogleCloudOptions(PipelineOptions):

http://git-wip-us.apache.org/repos/asf/beam/blob/1f2a/sdks/python/apache_beam/pipeline_test.py
--
diff --git a/sdks/python/apache_beam/pipeline_test.py 
b/sdks/python/apache_beam/pipeline_test.py
index aad0143..b3ac100 100644
--- a/sdks/python/apache_beam/pipeline_test.py
+++ b/sdks/python/apache_beam/pipeline_test.py
@@ -499,6 +499,36 @@ class RunnerApiTest(unittest.TestCase):
 self.assertEqual(MyPTransform.pickle_count[0], 20)
 
 
+class DirectRunnerRetryTests(unittest.TestCase):
+
+  def test_retry_fork_graph(self):
+pipeline_options = PipelineOptions(['--direct_runner_bundle_retry'])
+p = beam.Pipeline(options=pipeline_options)
+
+# TODO(mariagh): Remove the use of globals from the test.
+global count_b, count_c # pylint: disable=global-variable-undefined
+count_b, count_c = 0, 0
+
+def f_b(x):
+  global count_b  # pylint: disable=global-variable-undefined
+  count_b += 1
+  raise Exception('exception in f_b')
+
+def f_c(x):
+  global count_c  # pylint: disable=global-variable-undefined
+  count_c += 1
+  raise Exception('exception in f_c')
+
+names = p | 'CreateNodeA' >> beam.Create(['Ann', 'Joe'])
+
+fork_b = names | 'SendToB' >> beam.Map(f_b) # pylint: 
disable=unused-variable
+fork_c = names | 'SendToC' >> beam.Map(f_c) # pylint: 
disable=unused-variable
+
+with self.assertRaises(Exception):
+  p.run().wait_until_finish()
+assert count_b == count_c == 4
+
+
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.DEBUG)
   unittest.main()

http://git-wip-us.apache.org/repos/asf/beam/blob/1f2a/sdks/python/apache_beam/runners/direct/executor.py
--
diff --git a/sdks/python/apache_beam/runners/direct/executor.py 
b/sdks/python/apache_beam/runners/direct/executor.py
index e70e326..2e46978 100644
--- a/sdks/python/apache_beam/runners/direct/executor.py
+++ b/sdks/python/apache_beam/runners/direct/executor.py
@@ -25,10 +25,12 @@ import logging
 import Queue
 import sys
 import threading
+import traceback
 from weakref import WeakValueDictionary
 
 from apache_beam.metrics.execution import MetricsContainer
 from apache_beam.metrics.execution import ScopedMetricsContainer
+from apache_beam.options.pipeline_options import DirectOptions
 
 
 class _ExecutorService(object):
@@ -271,6 +273,15 @@ class TransformExecutor(_ExecutorService.CallableTask):
 self._side_input_values = {}
 self.blocked = False
 self._call_count = 0
+self._retry_count = 0
+# Switch to turn on/off the retry of bundles.
+pipeline_options = self._evaluation_context.pipeline_options
+if not pipeline_options.view_as(DirectOptions).direct_runner_bundle_retry:
+  self._max_retries_per_bundle = 1
+else:
+  self._max_retries_per_bundle = 4
+# TODO(mariagh): make _max_retries_per_bundle a

[jira] [Commented] (BEAM-2718) Add bundle retry logic to the DirectRunner

2017-08-12 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/BEAM-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124764#comment-16124764
 ] 

ASF GitHub Bot commented on BEAM-2718:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3679


> Add bundle retry logic to the DirectRunner
> --
>
> Key: BEAM-2718
> URL: https://issues.apache.org/jira/browse/BEAM-2718
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py
>Affects Versions: Not applicable
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: Not applicable
>
>
> When processing of a bundle fails, the bundle should be retried 3 times (for 
> a total of 4 attempts to process it).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/2] beam git commit: This closes #3679

2017-08-12 Thread altay 
This closes #3679


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b0b64218
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b0b64218
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b0b64218

Branch: refs/heads/master
Commit: b0b64218257576ef20601fd31e075e4a8e297a60
Parents: 84a2379 1f2
Author: Ahmet Altay 
Authored: Sat Aug 12 18:06:21 2017 -0700
Committer: Ahmet Altay 
Committed: Sat Aug 12 18:06:21 2017 -0700

--
 .../apache_beam/options/pipeline_options.py |   7 ++
 sdks/python/apache_beam/pipeline_test.py|  30 ++
 .../apache_beam/runners/direct/executor.py  | 100 ---
 3 files changed, 100 insertions(+), 37 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2819

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3757

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2818

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3756

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2817

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #3755

2017-08-12 Thread Apache Jenkins Server 
See

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Spark #2816

2017-08-12 Thread Apache Jenkins Server 
See