[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81512=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81512
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 17/Mar/18 05:50
Start Date: 17/Mar/18 05:50
Worklog Time Spent: 10m 
  Work Description: cclauss commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175249040
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
+# Pass these environment variables to the test environment.
+passenv = TRAVIS*
+# Don't warn that these commands aren't installed.
+whitelist_externals =
+  find
+  time
+
 [testenv:py27]
-# autocomplete_test depends on nose when invoked directly.
-deps =
-  nose==1.3.7
-  grpcio-tools==1.3.5
-whitelist_externals=find
 commands =
-  python --version
 
 Review comment:
   +1 on this request.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81512)
Time Spent: 6h  (was: 5h 50m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3865) Incorrect timestamp on merging window outputs.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3865?focusedWorklogId=81510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81510
 ]

ASF GitHub Bot logged work on BEAM-3865:


Author: ASF GitHub Bot
Created on: 17/Mar/18 05:28
Start Date: 17/Mar/18 05:28
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #4879: [BEAM-3865] Fix 
watermark hold handling bug.
URL: https://github.com/apache/beam/pull/4879#issuecomment-373895826
 
 
   Jenkins: retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81510)
Time Spent: 40m  (was: 0.5h)

> Incorrect timestamp on merging window outputs.
> --
>
> Key: BEAM-3865
> URL: https://issues.apache.org/jira/browse/BEAM-3865
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Robert Bradshaw
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Looks like we're setting multiple watermark holds with one arbitrarily being 
> held. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3865) Incorrect timestamp on merging window outputs.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3865?focusedWorklogId=81509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81509
 ]

ASF GitHub Bot logged work on BEAM-3865:


Author: ASF GitHub Bot
Created on: 17/Mar/18 05:27
Start Date: 17/Mar/18 05:27
Worklog Time Spent: 10m 
  Work Description: robertwb commented on a change in pull request #4879: 
[BEAM-3865] Fix watermark hold handling bug.
URL: https://github.com/apache/beam/pull/4879#discussion_r175248717
 
 

 ##
 File path: sdks/python/apache_beam/transforms/trigger_test.py
 ##
 @@ -81,16 +84,8 @@ def bundle_data(data, size):
 
 if not groupings:
   groupings = [1]
+grouping = list(groupings) + [-group_by for group_by in groupings]
 
 Review comment:
   It expands grouping by also adding negative values, which also tests adding 
elements in reverse order. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81509)
Time Spent: 0.5h  (was: 20m)

> Incorrect timestamp on merging window outputs.
> --
>
> Key: BEAM-3865
> URL: https://issues.apache.org/jira/browse/BEAM-3865
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Robert Bradshaw
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Looks like we're setting multiple watermark holds with one arbitrarily being 
> held. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3866) Move Go SDK to not use WindowedValue for PCollections

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3866?focusedWorklogId=81507=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81507
 ]

ASF GitHub Bot logged work on BEAM-3866:


Author: ASF GitHub Bot
Created on: 17/Mar/18 05:24
Start Date: 17/Mar/18 05:24
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #4885: [BEAM-3866] Remove 
WindowedValue on PCollections for Go SDK
URL: https://github.com/apache/beam/pull/4885#issuecomment-373895692
 
 
   R: @lostluck @aljoscha 
   
   This PR resolves one of the issues found with Go on Flink.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81507)
Time Spent: 20m  (was: 10m)

> Move Go SDK to not use WindowedValue for PCollections
> -
>
> Key: BEAM-3866
> URL: https://issues.apache.org/jira/browse/BEAM-3866
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The windowing information is part of the gRPC instructions. Dataflow still 
> expects the old way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81508=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81508
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 17/Mar/18 05:24
Start Date: 17/Mar/18 05:24
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #4801: [BEAM-3738] Enable 
py3 lint
URL: https://github.com/apache/beam/pull/4801#issuecomment-373895697
 
 
   +1 to using an environment variable, much better than falsely claiming
   3.0-3.4 support.
   
   
   On Fri, Mar 16, 2018 at 5:23 PM Ahmet Altay 
   wrote:
   
   > Closed #4801 .
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > , or mute the
   > thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81508)
Time Spent: 5h 50m  (was: 5h 40m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3866) Move Go SDK to not use WindowedValue for PCollections

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3866?focusedWorklogId=81506=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81506
 ]

ASF GitHub Bot logged work on BEAM-3866:


Author: ASF GitHub Bot
Created on: 17/Mar/18 05:21
Start Date: 17/Mar/18 05:21
Worklog Time Spent: 10m 
  Work Description: herohde opened a new pull request #4885: [BEAM-3866] 
Remove WindowedValue on PCollections for Go SDK
URL: https://github.com/apache/beam/pull/4885
 
 
   This is what the Model pipeline expects, although Dataflow expects the old 
way.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81506)
Time Spent: 10m
Remaining Estimate: 0h

> Move Go SDK to not use WindowedValue for PCollections
> -
>
> Key: BEAM-3866
> URL: https://issues.apache.org/jira/browse/BEAM-3866
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The windowing information is part of the gRPC instructions. Dataflow still 
> expects the old way.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2767) BigQueryIO result different for REPEATED field between DirectRunner and DataflowRunner

[Maximilian Roos (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403262#comment-16403262
 ] 

Maximilian Roos commented on BEAM-2767:
---

The same divergence in behavior exists in Python too

> BigQueryIO result different for REPEATED field between DirectRunner and 
> DataflowRunner
> --
>
> Key: BEAM-2767
> URL: https://issues.apache.org/jira/browse/BEAM-2767
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp, runner-dataflow, runner-direct
>Affects Versions: 2.0.0
>Reporter: Andre
>Assignee: Thomas Groh
>Priority: Minor
>
> When running a query against BigQueryIO with a REPEATED RECORD field the 
> behavior is different between DirectRunner and DataflowRunner. The field 
> containing the repeated record has to be cast to access the records. 
> Apparently the following implementations work for each runner but I would 
> expect them to be the same as my pipeline otherwise only runs on one.
> DirectRunner:
> {code:java}
> ArrayList orderLines = (ArrayList) 
> c.element().get("RepeatedField");
> {code}
> DataflowRunner:
> {code:java}
> ImmutableList orderLines = (ImmutableList) 
> c.element().get("RepeatedField");
> {code}
>   
> For example when using the ImmutableList implementation on DirectRunner the 
> following exception is thrown:
> {code:java}
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to 
> com.google.common.collect.ImmutableList
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81483
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 02:26
Start Date: 17/Mar/18 02:26
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4838: 
[BEAM-3818] Add support for streaming side inputs in the DirectRunner (part I: 
update _SideInputsContainer as the watermark advances)
URL: https://github.com/apache/beam/pull/4838#discussion_r175245310
 
 

 ##
 File path: sdks/python/apache_beam/testing/test_stream_test.py
 ##
 @@ -245,6 +247,84 @@ def fired_elements(elem):
 # TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
 self.assertEqual([('k', ['a'])], result)
 
+  def test_basic_execution_sideinputs_batch(self):
+
+# TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
+global result # pylint: disable=global-variable-undefined
+result = []
+
+def recorded_elements(elem):
+  result.append(elem)
+  return elem
+
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+main_stream = (p
+   | 'main TestStream' >> TestStream()
+   .advance_watermark_to(10)
+   .add_elements(['e']))
+side = (p
+| beam.Create([2, 1, 4])
+| beam.Map(lambda t: window.TimestampedValue(t, t)))
+
+class RecordFn(beam.DoFn):
+  def process(self,
+  elm=beam.DoFn.ElementParam,
+  ts=beam.DoFn.TimestampParam,
+  side=beam.DoFn.SideInputParam):
+yield (elm, ts, side)
+
+records = (main_stream # pylint: disable=unused-variable
+   | beam.ParDo(RecordFn(), beam.pvalue.AsList(side))
+   | beam.Map(recorded_elements))
+p.run()
+
+# TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
+self.assertEqual([('e', Timestamp(10), [2, 1, 4])], result)
+
+  def test_basic_execution_sideinputs(self):
+
+# TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
+global result # pylint: disable=global-variable-undefined
+result = []
+
+def recorded_elements(elem):
+  result.append(elem)
+  return elem
+
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+main_stream = (p
+   | 'main TestStream' >> TestStream()
+   .advance_watermark_to(10)
+   .add_elements(['e'])
+   .advance_processing_time(11))
+side_stream = (p
+   | 'side TestStream' >> TestStream()
+   .add_elements([window.TimestampedValue(2, 2)])
+   .add_elements([window.TimestampedValue(1, 1)])
+   .add_elements([window.TimestampedValue(4, 4)]))
+
+class RecordFn(beam.DoFn):
+  def process(self,
+  elm=beam.DoFn.ElementParam,
+  ts=beam.DoFn.TimestampParam,
+  side=beam.DoFn.SideInputParam):
+yield (elm, ts, side)
+
+records = (main_stream# pylint: disable=unused-variable
+   | beam.ParDo(RecordFn(), beam.pvalue.AsList(side_stream))
+   | beam.Map(recorded_elements))
+
+p.run()
+
+# TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
+self.assertEqual([('e', Timestamp(10), [2, 1, 4])], result)
+
 
 Review comment:
   For the follow up PR, please add some non-trivial side input tests. 
(Suggestions: A test with partial side inputs in the accessible window, another 
with more main elements.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81483)
Time Spent: 2h 20m  (was: 2h 10m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for 

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81485
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 02:26
Start Date: 17/Mar/18 02:26
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4838: 
[BEAM-3818] Add support for streaming side inputs in the DirectRunner (part I: 
update _SideInputsContainer as the watermark advances)
URL: https://github.com/apache/beam/pull/4838#discussion_r175244992
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -217,11 +242,13 @@ def handle_result(
 self._side_inputs_container.add_values(
 view,
 committed_bundle.get_elements_iterable(make_copy=True))
-  if (self.get_execution_context(result.transform)
-  .watermarks.input_watermark
-  == WatermarkManager.WATERMARK_POS_INF):
-self._pending_unblocked_tasks.extend(
-self._side_inputs_container.finalize_value_and_get_tasks(view))
+
+  # Tasks generated from unblocked side inputs as the watermark progresses.
+  tasks = self._watermark_manager.update_watermarks(
+  completed_bundle, result.transform, completed_timers,
+  committed_bundles, unprocessed_bundles, result.keyed_watermark_holds,
+  self._side_inputs_container)
+  self._pending_unblocked_tasks.extend(tasks)
 
 Review comment:
   Should this stay inside the if condition? Since only when a view is updated, 
it will be necessary to check for unblocked tasks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81485)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81484=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81484
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 02:26
Start Date: 17/Mar/18 02:26
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4838: 
[BEAM-3818] Add support for streaming side inputs in the DirectRunner (part I: 
update _SideInputsContainer as the watermark advances)
URL: https://github.com/apache/beam/pull/4838#discussion_r175245107
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/watermark_manager.py
 ##
 @@ -128,8 +129,9 @@ def _update_pending(self, input_committed_bundle, 
applied_ptransform,
 if input_committed_bundle and input_committed_bundle.has_elements():
   completed_tw.remove_pending(input_committed_bundle)
 
-  def _refresh_watermarks(self, applied_ptransform):
+  def _refresh_watermarks(self, applied_ptransform, side_inputs_container):
 
 Review comment:
   Similar to the above comment. Can you use self._side_inputs_container 
instead of adding this new argument here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81484)
Time Spent: 2.5h  (was: 2h 20m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81487=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81487
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 02:26
Start Date: 17/Mar/18 02:26
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4838: 
[BEAM-3818] Add support for streaming side inputs in the DirectRunner (part I: 
update _SideInputsContainer as the watermark advances)
URL: https://github.com/apache/beam/pull/4838#discussion_r175245092
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/watermark_manager.py
 ##
 @@ -94,14 +95,14 @@ def get_watermarks(self, applied_ptransform):
 
   def update_watermarks(self, completed_committed_bundle, applied_ptransform,
 completed_timers, outputs, unprocessed_bundles,
-keyed_earliest_holds):
+keyed_earliest_holds, side_inputs_container):
 
 Review comment:
   How is this new argument `side_inputs_container` different from the one 
passed to the constructor above?
   
   Can you use `self._side_inputs_container` instead of adding this new 
argument here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81487)
Time Spent: 2h 40m  (was: 2.5h)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81486=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81486
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 02:26
Start Date: 17/Mar/18 02:26
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4838: 
[BEAM-3818] Add support for streaming side inputs in the DirectRunner (part I: 
update _SideInputsContainer as the watermark advances)
URL: https://github.com/apache/beam/pull/4838#discussion_r175244875
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -56,6 +56,12 @@ def __init__(self, view):
 self.value = None
 self.has_result = False
 
+  def __repr__(self):
+elements_string = (
+', '.join(
+str(elm) for elm in self.elements) if self.elements else '')
 
 Review comment:
   Why do we use `` here and `[]` below to represent empty list?
   
   (It is fine if this is fixed in a follow up PR.)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81486)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81482=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81482
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:42
Start Date: 17/Mar/18 01:42
Worklog Time Spent: 10m 
  Work Description: mariapython commented on a change in pull request 
#4838: [BEAM-3818] Add support for streaming side inputs in the DirectRunner 
(part I: update _SideInputsContainer as the watermark advances)
URL: https://github.com/apache/beam/pull/4838#discussion_r175244099
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -99,6 +113,19 @@ def finalize_value_and_get_tasks(self, side_input):
   view.has_result = True
   return result
 
+  def update_watermarks_for_transform(self, ptransform, watermark):
+# Collect tasks that get unblocked as the workflow progresses
+unblocked_tasks = []
+for view in self._transform_to_views[ptransform]:
+  unblocked_tasks.extend(self._update_watermarks_for_view(view, watermark))
+return unblocked_tasks
+
+  def _update_watermarks_for_view(self, view, watermark):
+unblocked_tasks = []
+if watermark.input_watermark == WatermarkManager.WATERMARK_POS_INF:
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81482)
Time Spent: 2h 10m  (was: 2h)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3870) Jars contain duplicate class definitions

[Nigel Kilmer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nigel Kilmer updated BEAM-3870:
---
Description: 
There are a large number of classes with definitions in both 
beam-sdks-java-harness-2.4.0-SNAPSHOT.jar and either 
beam-model-pipeline-2.4.0-SNAPSHOT.jar or 
beam-model-fn-execution-2.4.0-SNAPSHOT.jar.

Some randomly chosen examples of classes with this problem (out of a much 
larger set):

org/apache/beam/model/pipeline/v1/Endpoints

org/apache/beam/model/pipeline/v1/RunnerApi

org/apache/beam/model/pipeline/v1/StandardWindowFns

org/apache/beam/model/fnexecution/v1/ProvisionApi

  was:
There are a large number of classes with definitions in both 
beam-sdks-java-harness-2.4.0-SNAPSHOT.jar and either 
beam-model-pipeline-2.4.0-SNAPSHOT.jar or 
beam-model-fn-execution-2.4.0-SNAPSHOT.jar.

Some randomly chosen examples of classes with this problem (out of a much, much 
larger set):

org/apache/beam/model/pipeline/v1/Endpoints

org/apache/beam/model/pipeline/v1/RunnerApi

org/apache/beam/model/pipeline/v1/StandardWindowFns

org/apache/beam/model/fnexecution/v1/ProvisionApi


> Jars contain duplicate class definitions
> 
>
> Key: BEAM-3870
> URL: https://issues.apache.org/jira/browse/BEAM-3870
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-harness
>Affects Versions: 2.4.0
>Reporter: Nigel Kilmer
>Assignee: Kenneth Knowles
>Priority: Minor
>
> There are a large number of classes with definitions in both 
> beam-sdks-java-harness-2.4.0-SNAPSHOT.jar and either 
> beam-model-pipeline-2.4.0-SNAPSHOT.jar or 
> beam-model-fn-execution-2.4.0-SNAPSHOT.jar.
> Some randomly chosen examples of classes with this problem (out of a much 
> larger set):
> org/apache/beam/model/pipeline/v1/Endpoints
> org/apache/beam/model/pipeline/v1/RunnerApi
> org/apache/beam/model/pipeline/v1/StandardWindowFns
> org/apache/beam/model/fnexecution/v1/ProvisionApi



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3870) Jars contain duplicate class definitions

[Nigel Kilmer (JIRA)]

Nigel Kilmer created BEAM-3870:
--

 Summary: Jars contain duplicate class definitions
 Key: BEAM-3870
 URL: https://issues.apache.org/jira/browse/BEAM-3870
 Project: Beam
  Issue Type: Bug
  Components: beam-model, sdk-java-harness
Affects Versions: 2.4.0
Reporter: Nigel Kilmer
Assignee: Kenneth Knowles


There are a large number of classes with definitions in both 
beam-sdks-java-harness-2.4.0-SNAPSHOT.jar and either 
beam-model-pipeline-2.4.0-SNAPSHOT.jar or 
beam-model-fn-execution-2.4.0-SNAPSHOT.jar.

Some randomly chosen examples of classes with this problem (out of a much, much 
larger set):

org/apache/beam/model/pipeline/v1/Endpoints

org/apache/beam/model/pipeline/v1/RunnerApi

org/apache/beam/model/pipeline/v1/StandardWindowFns

org/apache/beam/model/fnexecution/v1/ProvisionApi



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81478=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81478
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:34
Start Date: 17/Mar/18 01:34
Worklog Time Spent: 10m 
  Work Description: mariapython commented on a change in pull request 
#4838: [BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175243839
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -56,6 +56,11 @@ def __init__(self, view):
 self.value = None
 self.has_result = False
 
+  def __repr__(self):
+elements = ', '.join([
+str(elm) for elm in self.elements] if self.elements else [''])
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81478)
Time Spent: 1h 40m  (was: 1.5h)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81477=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81477
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:34
Start Date: 17/Mar/18 01:34
Worklog Time Spent: 10m 
  Work Description: mariapython commented on a change in pull request 
#4838: [BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175243837
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -67,8 +72,17 @@ class _SideInputsContainer(object):
   def __init__(self, views):
 self._lock = threading.Lock()
 self._views = {}
+self._transform_to_views = collections.defaultdict(list)
+
 for view in views:
   self._views[view] = _SideInputView(view)
+  self._transform_to_views[view.pvalue.producer].append(view)
+
+  def __repr__(self):
+views = ', '.join([
+str(elm) for elm in self._views.values()
+] if self._views.values() else [])
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81477)
Time Spent: 1.5h  (was: 1h 20m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81480=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81480
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:34
Start Date: 17/Mar/18 01:34
Worklog Time Spent: 10m 
  Work Description: mariapython commented on a change in pull request 
#4838: [BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175243849
 
 

 ##
 File path: sdks/python/apache_beam/testing/test_stream_test.py
 ##
 @@ -245,6 +247,82 @@ def fired_elements(elem):
 # TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
 self.assertEqual([('k', ['a'])], result)
 
+  def test_basic_execution_sideinputs_batch(self):
+
+# TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
+global result # pylint: disable=global-variable-undefined
+result = []
+
+def recorded_elements(elem):
+  result.append(elem)
+  return elem
+
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+main_stream = (p
+   | 'main TestStream' >> TestStream()
+   .advance_watermark_to(10)
+   .add_elements(['e']))
+side = (p
+| beam.Create([2, 1, 4])
+| beam.Map(lambda t: window.TimestampedValue(t, t)))
+
+class RecordFn(beam.DoFn):
+  def process(self,
+  elm=beam.DoFn.ElementParam,
+  ts=beam.DoFn.TimestampParam,
+  side=beam.DoFn.SideInputParam):
+yield (elm, ts, side)
+
+records = main_stream | beam.ParDo(RecordFn(), beam.pvalue.AsList(side)) | 
beam.Map(recorded_elements) # pylint: disable=line-too-long, unused-variable
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81480)
Time Spent: 2h  (was: 1h 50m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81479=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81479
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:34
Start Date: 17/Mar/18 01:34
Worklog Time Spent: 10m 
  Work Description: mariapython commented on a change in pull request 
#4838: [BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175243844
 
 

 ##
 File path: sdks/python/apache_beam/testing/test_stream_test.py
 ##
 @@ -245,6 +247,82 @@ def fired_elements(elem):
 # TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
 self.assertEqual([('k', ['a'])], result)
 
+  def test_basic_execution_sideinputs_batch(self):
+
+# TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
+global result # pylint: disable=global-variable-undefined
+result = []
+
+def recorded_elements(elem):
+  result.append(elem)
+  return elem
+
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+main_stream = (p
+   | 'main TestStream' >> TestStream()
+   .advance_watermark_to(10)
+   .add_elements(['e']))
+side = (p
+| beam.Create([2, 1, 4])
+| beam.Map(lambda t: window.TimestampedValue(t, t)))
+
+class RecordFn(beam.DoFn):
+  def process(self,
+  elm=beam.DoFn.ElementParam,
+  ts=beam.DoFn.TimestampParam,
+  side=beam.DoFn.SideInputParam):
+yield (elm, ts, side)
+
+records = main_stream | beam.ParDo(RecordFn(), beam.pvalue.AsList(side)) | 
beam.Map(recorded_elements) # pylint: disable=line-too-long, unused-variable
+p.run()
+
+# TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
+self.assertEqual([('e', Timestamp(10), [2, 1, 4])], result)
+
+  def test_basic_execution_sideinputs(self):
+
+# TODO(BEAM-3377): Remove after assert_that in streaming is fixed.
+global result # pylint: disable=global-variable-undefined
+result = []
+
+def recorded_elements(elem):
+  result.append(elem)
+  return elem
+
+options = PipelineOptions()
+options.view_as(StandardOptions).streaming = True
+p = TestPipeline(options=options)
+
+main_stream = (p
+   | 'main TestStream' >> TestStream()
+   .advance_watermark_to(10)
+   .add_elements(['e'])
+   .advance_processing_time(11))
+# TODO(mariagh): Fix this
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81479)
Time Spent: 1h 50m  (was: 1h 40m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81475=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81475
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:33
Start Date: 17/Mar/18 01:33
Worklog Time Spent: 10m 
  Work Description: mariapython commented on a change in pull request 
#4838: [BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175243829
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -217,11 +241,12 @@ def handle_result(
 self._side_inputs_container.add_values(
 view,
 committed_bundle.get_elements_iterable(make_copy=True))
-  if (self.get_execution_context(result.transform)
-  .watermarks.input_watermark
-  == WatermarkManager.WATERMARK_POS_INF):
-self._pending_unblocked_tasks.extend(
-self._side_inputs_container.finalize_value_and_get_tasks(view))
+
+  tasks = self._watermark_manager.update_watermarks(
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81475)
Time Spent: 1h 10m  (was: 1h)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81476=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81476
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:34
Start Date: 17/Mar/18 01:34
Worklog Time Spent: 10m 
  Work Description: mariapython commented on a change in pull request 
#4838: [BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175243831
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -99,6 +113,19 @@ def finalize_value_and_get_tasks(self, side_input):
   view.has_result = True
   return result
 
+  def update_watermarks_for_transform(self, ptransform, watermark):
+# Collect tasks that get unblocked as the workflow progresses
 
 Review comment:
   Done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81476)
Time Spent: 1h 20m  (was: 1h 10m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow #5167

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add Network Utility Methods

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

--
[...truncated 1.73 MB...]
2018-03-17T01:25:17.172 [INFO] Dependency-reduced POM written at: 

2018-03-17T01:25:17.916 [INFO] 
2018-03-17T01:25:17.916 [INFO] --- maven-dependency-plugin:3.0.2:analyze-only 
(default) @ beam-sdks-java-extensions-google-cloud-platform-core ---
2018-03-17T01:25:18.049 [INFO] No dependency problems found
[JENKINS] Archiving disabled
2018-03-17T01:25:19.740 [INFO] 
2018-03-17T01:25:19.740 [INFO] 

2018-03-17T01:25:19.740 [INFO] Building Apache Beam :: Runners 2.5.0-SNAPSHOT
2018-03-17T01:25:19.740 [INFO] 

2018-03-17T01:25:19.744 [INFO] 
2018-03-17T01:25:19.748 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-parent ---
2018-03-17T01:25:19.752 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md, 
**/sdks/python/apache_beam/portability/api/*pb2*.*, 
**/sdks/python/apache_beam/portability/common_urns.py, **/sdks/python/**/*.c, 
**/sdks/python/**/*.so, **/sdks/python/**/*.egg], excludes = [])
2018-03-17T01:25:19.985 [INFO] 
2018-03-17T01:25:19.985 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-maven-version) @ beam-runners-parent ---
2018-03-17T01:25:20.092 [INFO] 
2018-03-17T01:25:20.092 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce) @ beam-runners-parent ---
2018-03-17T01:25:20.095 [INFO] Adding ignore: module-info
2018-03-17T01:25:20.201 [INFO] 
2018-03-17T01:25:20.201 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-banned-dependencies) @ beam-runners-parent ---
2018-03-17T01:25:20.316 [INFO] 
2018-03-17T01:25:20.316 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-parent ---
2018-03-17T01:25:20.449 [INFO] 
2018-03-17T01:25:20.449 [INFO] --- maven-checkstyle-plugin:3.0.0:check 
(default) @ beam-runners-parent ---
2018-03-17T01:25:20.721 [INFO] Starting audit...
Audit done.
2018-03-17T01:25:20.830 [INFO] 
2018-03-17T01:25:20.830 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-runners-parent ---
2018-03-17T01:25:20.938 [INFO] 
2018-03-17T01:25:20.938 [INFO] --- maven-site-plugin:3.7:attach-descriptor 
(attach-descriptor) @ beam-runners-parent ---
2018-03-17T01:25:20.939 [INFO] No site descriptor found: nothing to attach.
2018-03-17T01:25:21.155 [INFO] 
2018-03-17T01:25:21.155 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-runners-parent ---
2018-03-17T01:25:21.157 [INFO] Skipping packaging of the jar
2018-03-17T01:25:21.369 [INFO] 
2018-03-17T01:25:21.370 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-runners-parent ---
2018-03-17T01:25:21.371 [INFO] Skipping packaging of the test-jar
2018-03-17T01:25:21.480 [INFO] 
2018-03-17T01:25:21.480 [INFO] --- maven-shade-plugin:3.1.0:shade 
(bundle-and-repackage) @ beam-runners-parent ---
2018-03-17T01:25:21.485 [INFO] Replacing original artifact with shaded artifact.
2018-03-17T01:25:21.593 [INFO] 
2018-03-17T01:25:21.593 [INFO] --- maven-dependency-plugin:3.0.2:analyze-only 
(default) @ beam-runners-parent ---
2018-03-17T01:25:21.594 [INFO] Skipping pom project
[JENKINS] Archiving disabled
2018-03-17T01:25:22.849 [INFO] 
2018-03-17T01:25:22.849 [INFO] 

2018-03-17T01:25:22.849 [INFO] Building Apache Beam :: Runners :: Core 
Construction Java 2.5.0-SNAPSHOT
2018-03-17T01:25:22.849 [INFO] 

2018-03-17T01:25:22.880 [INFO] 
2018-03-17T01:25:22.880 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-core-construction-java ---
2018-03-17T01:25:22.883 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md, 
**/sdks/python/apache_beam/portability/api/*pb2*.*, 
**/sdks/python/apache_beam/portability/common_urns.py, **/sdks/python/**/*.c, 
**/sdks/python/**/*.so, **/sdks/python/**/*.egg], excludes = [])
2018-03-17T01:25:23.019 [INFO] 
2018-03-17T01:25:23.019 [INFO] --- 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5166

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1475

[Apache Jenkins Server]

See 


Changes:

[rober] Correct EventTime encoding. time.(Time).Unix() returns seconds not

[ankurgoenka] Send error response for process bundle progress instead of 
throwing and

[Pablo] Fixing counter string name.

[tgroh] Add Network Utility Methods

[lcwik] Fix missing license header in test.

[aaltay] Enable input subscription in Python streaming wordcount (#4864)

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

--
[...truncated 63.08 KB...]
2018-03-17 00:56:56,038 e93ac1a0 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-17 00:57:13,983 e93ac1a0 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-17 00:57:17,435 e93ac1a0 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4cb724067b844934_01623175ebb0_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r4cb724067b844934_01623175ebb0_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r4cb724067b844934_01623175ebb0_1 ... (0s) Current status: DONE   
2018-03-17 00:57:17,435 e93ac1a0 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-17 00:57:37,475 e93ac1a0 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-17 00:57:41,228 e93ac1a0 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r46af790b8ba1496c_0162317647c7_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r46af790b8ba1496c_0162317647c7_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r46af790b8ba1496c_0162317647c7_1 ... (0s) Current status: DONE   
2018-03-17 00:57:41,229 e93ac1a0 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-17 00:58:05,893 e93ac1a0 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-17 00:58:09,311 e93ac1a0 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r3fb126fd4390e35c_01623176b688_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r3fb126fd4390e35c_01623176b688_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r3fb126fd4390e35c_01623176b688_1 ... (0s) Current status: DONE   
2018-03-17 00:58:09,312 e93ac1a0 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-17 00:58:32,454 e93ac1a0 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-17 00:58:35,805 e93ac1a0 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex #3711

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add Network Utility Methods

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

--
[...truncated 102.83 KB...]
2018-03-17T01:02:43.859 [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.015 s - in org.apache.beam.sdk.coders.DelegateCoderTest
2018-03-17T01:02:43.860 [INFO] Running 
org.apache.beam.sdk.io.SerializableAvroCodecFactoryTest
2018-03-17T01:02:43.869 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.006 s - in 
org.apache.beam.sdk.io.SerializableAvroCodecFactoryTest
2018-03-17T01:02:43.869 [INFO] Running 
org.apache.beam.sdk.coders.CustomCoderTest
2018-03-17T01:02:43.876 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.003 s - in org.apache.beam.sdk.coders.CustomCoderTest
2018-03-17T01:02:43.877 [INFO] Running 
org.apache.beam.sdk.transforms.reflect.OnTimerInvokersTest
2018-03-17T01:02:43.909 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.028 s - in 
org.apache.beam.sdk.transforms.reflect.OnTimerInvokersTest
2018-03-17T01:02:43.909 [INFO] Running 
org.apache.beam.sdk.runners.TransformTreeTest
2018-03-17T01:02:43.955 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.042 s - in org.apache.beam.sdk.runners.TransformTreeTest
2018-03-17T01:02:43.955 [INFO] Running 
org.apache.beam.sdk.options.ExperimentalOptionsTest
2018-03-17T01:02:43.961 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.002 s - in 
org.apache.beam.sdk.options.ExperimentalOptionsTest
2018-03-17T01:02:43.962 [INFO] Running 
org.apache.beam.sdk.coders.VarIntCoderTest
2018-03-17T01:02:43.970 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.004 s - in org.apache.beam.sdk.coders.VarIntCoderTest
2018-03-17T01:02:43.971 [INFO] Running 
org.apache.beam.sdk.transforms.windowing.PaneInfoTest
2018-03-17T01:02:43.983 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.008 s - in 
org.apache.beam.sdk.transforms.windowing.PaneInfoTest
2018-03-17T01:02:43.983 [INFO] Running 
org.apache.beam.sdk.io.LocalFileSystemTest
2018-03-17T01:02:44.026 [INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.039 s - in org.apache.beam.sdk.io.LocalFileSystemTest
2018-03-17T01:02:44.027 [INFO] Running 
org.apache.beam.sdk.coders.LengthPrefixCoderTest
2018-03-17T01:02:44.044 [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.013 s - in org.apache.beam.sdk.coders.LengthPrefixCoderTest
2018-03-17T01:02:44.044 [INFO] Running 
org.apache.beam.sdk.testing.CrashingRunnerTest
2018-03-17T01:02:44.054 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.006 s - in org.apache.beam.sdk.testing.CrashingRunnerTest
2018-03-17T01:02:44.055 [INFO] Running org.apache.beam.sdk.transforms.DoFnTest
2018-03-17T01:02:44.060 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.002 s - in org.apache.beam.sdk.transforms.DoFnTest
2018-03-17T01:02:44.061 [INFO] Running org.apache.beam.sdk.io.FileBasedSinkTest
2018-03-17T01:02:44.123 [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.058 s - in org.apache.beam.sdk.io.FileBasedSinkTest
2018-03-17T01:02:44.124 [INFO] Running 
org.apache.beam.sdk.testing.PaneExtractorsTest
2018-03-17T01:02:44.134 [INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.007 s - in org.apache.beam.sdk.testing.PaneExtractorsTest
2018-03-17T01:02:44.135 [INFO] Running 
org.apache.beam.sdk.testing.StaticWindowsTest
2018-03-17T01:02:44.144 [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.005 s - in org.apache.beam.sdk.testing.StaticWindowsTest
2018-03-17T01:02:44.145 [INFO] Running 
org.apache.beam.sdk.values.reflect.ReflectionGetterTest
2018-03-17T01:02:44.151 [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.002 s - in 
org.apache.beam.sdk.values.reflect.ReflectionGetterTest
2018-03-17T01:02:44.151 [INFO] Running org.apache.beam.sdk.coders.ListCoderTest
2018-03-17T01:02:44.165 [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.01 s - in org.apache.beam.sdk.coders.ListCoderTest
2018-03-17T01:02:44.166 [INFO] Running org.apache.beam.sdk.transforms.LatestTest
2018-03-17T01:02:44.198 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.028 s - in org.apache.beam.sdk.transforms.LatestTest
2018-03-17T01:02:44.198 [INFO] Running 
org.apache.beam.sdk.coders.DurationCoderTest
2018-03-17T01:02:44.207 [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 
0, Time elapsed: 0.005 s - in org.apache.beam.sdk.coders.DurationCoderTest
2018-03-17T01:02:44.208 [INFO] Running 

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81473=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81473
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:01
Start Date: 17/Mar/18 01:01
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #4884: [BEAM-3676] Add 
Portable State Service 
URL: https://github.com/apache/beam/pull/4884#issuecomment-373881777
 
 
   Fixed lukecwik@'s requested changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81473)
Time Spent: 2h 50m  (was: 2h 40m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81472=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81472
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:01
Start Date: 17/Mar/18 01:01
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #4884: [BEAM-3676] Add 
Portable State Service 
URL: https://github.com/apache/beam/pull/4884#issuecomment-373881777
 
 
   Fixed lukecwik's requested changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81472)
Time Spent: 2h 40m  (was: 2.5h)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81469=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81469
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:00
Start Date: 17/Mar/18 01:00
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175242602
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
 ##
 @@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import static com.google.common.base.Throwables.getStackTraceAsString;
+
+import io.grpc.stub.StreamObserver;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnStateGrpc;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** An implementation of the Beam Fn State service. */
+public class GrpcStateService extends BeamFnStateGrpc.BeamFnStateImplBase 
implements StateService {
+  private static final Logger LOG = 
LoggerFactory.getLogger(GrpcStateService.class);
+  private final ConcurrentHashMap
+  requestHandlers;
+
+  public GrpcStateService()
+  throws Exception {
+this.requestHandlers = new ConcurrentHashMap<>();
+  }
+
+  @Override
+  public void close() throws Exception {
+// TODO: Track multiple clients and disconnect them cleanly instead of 
forcing termination
+  }
+
+  @Override
+  public StreamObserver state(StreamObserver 
responseObserver) {
+return new Inbound(responseObserver);
+  }
+
+  @Override
+  public AutoCloseable registerForProcessBundleInstructionId(
+  String processBundleInstructionId, StateRequestHandler handler) {
+requestHandlers.put(processBundleInstructionId, handler);
+return () -> requestHandlers.remove(processBundleInstructionId);
+  }
+
+  /**
+   * An inbound {@link StreamObserver} which delegates requests to registered 
handlers.
+   *
+   * TODO: Handle when the client indicates completion or an error on the 
inbound stream and
+   * there are pending requests.
+   */
+  private class Inbound implements StreamObserver {
+private final StreamObserver outboundObserver;
+
+Inbound(StreamObserver outboundObserver) {
+  this.outboundObserver = outboundObserver;
+}
+
+@Override
+public void onNext(StateRequest request) {
+  CompletionStage responseStage = new 
CompletableFuture<>();
+  responseStage.whenCompleteAsync(
+  (StateResponse.Builder responseBuilder, Throwable t) ->
+  outboundObserver.onNext(
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81469)
Time Spent: 2h 10m  (was: 2h)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during 

Build failed in Jenkins: beam_PerformanceTests_Python #1031

[Apache Jenkins Server]

See 


Changes:

[rober] Correct EventTime encoding. time.(Time).Unix() returns seconds not

[ankurgoenka] Send error response for process bundle progress instead of 
throwing and

[Pablo] Fixing counter string name.

[tgroh] Add Network Utility Methods

[lcwik] Fix missing license header in test.

[aaltay] Enable input subscription in Python streaming wordcount (#4864)

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

--
[...truncated 1.29 KB...]
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8260125647765366079.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4639779576775086749.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1212160134747752640.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins597370827803029033.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/b2/8a/9b1fa583782c63b89d21c4bf32dfdee40fa30be1339201f0acfe3fb21510/setuptools-38.6.0-py2.py3-none-any.whl#md5=e2125311cb1b4adc8cc26f678caa88a2
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4136048971795611849.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6864935174128403318.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81467
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:00
Start Date: 17/Mar/18 01:00
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175242579
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateService.java
 ##
 @@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.runners.fnexecution.FnService;
+
+/**
+ * The {@link StateService} is able to delegate {@link StateRequest}s to a set 
of registered
+ * handlers. Any request for an unregistered process bundle instruction id is 
automatically failed.
+ */
+public interface StateService extends FnService {
 
 Review comment:
   I renamed to StateDelegator and decoupled it from FnService instead..


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81467)
Time Spent: 1h 50m  (was: 1h 40m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81468=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81468
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:00
Start Date: 17/Mar/18 01:00
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175242595
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/package-info.java
 ##
 @@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** State api services. */
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81468)
Time Spent: 2h  (was: 1h 50m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81471=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81471
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:00
Start Date: 17/Mar/18 01:00
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175242611
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
 ##
 @@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import static com.google.common.base.Throwables.getStackTraceAsString;
+
+import io.grpc.stub.StreamObserver;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnStateGrpc;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** An implementation of the Beam Fn State service. */
+public class GrpcStateService extends BeamFnStateGrpc.BeamFnStateImplBase 
implements StateService {
+  private static final Logger LOG = 
LoggerFactory.getLogger(GrpcStateService.class);
+  private final ConcurrentHashMap
+  requestHandlers;
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81471)
Time Spent: 2.5h  (was: 2h 20m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81470=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81470
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 01:00
Start Date: 17/Mar/18 01:00
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175242605
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
 ##
 @@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import static com.google.common.base.Throwables.getStackTraceAsString;
+
+import io.grpc.stub.StreamObserver;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnStateGrpc;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** An implementation of the Beam Fn State service. */
+public class GrpcStateService extends BeamFnStateGrpc.BeamFnStateImplBase 
implements StateService {
+  private static final Logger LOG = 
LoggerFactory.getLogger(GrpcStateService.class);
+  private final ConcurrentHashMap
+  requestHandlers;
+
+  public GrpcStateService()
+  throws Exception {
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81470)
Time Spent: 2h 20m  (was: 2h 10m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_JDBC #338

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81466=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81466
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:57
Start Date: 17/Mar/18 00:57
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175242455
 
 

 ##
 File path: 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java
 ##
 @@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.nio.charset.StandardCharsets;
+import java.util.concurrent.CompletableFuture;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.sdk.fn.test.TestStreams;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.*;
+
+/** Tests for {@link 
org.apache.beam.runners.fnexecution.state.GrpcStateService} */
+@RunWith(JUnit4.class)
+public class GrpcStateServiceTest {
+  private static long TIMEOUT = 30 * 1000;
+
+  private GrpcStateService stateService;
+
+  @Mock
+  private StreamObserver responseObserver;
+
+  @Mock
+  private StateRequestHandler handler;
+
+  @Before
+  public void setUp() throws Exception {
+MockitoAnnotations.initMocks(this);
+stateService = new GrpcStateService();
+  }
+
+  /**
+   * After a handler has been registered with
+   * {@link GrpcStateService#registerForProcessBundleInstructionId(String, 
StateRequestHandler)},
+   * the {@link GrpcStateService} should delegate requests through
+   * {@link GrpcStateService#state(StreamObserver)} to the registered handler.
+   */
+  @Test
+  public void testStateRequestsHandledByRegisteredHandlers() throws Exception {
+// register handler
+String bundleInstructionId = "bundle_instruction";
+stateService.registerForProcessBundleInstructionId(bundleInstructionId, 
handler);
+
+// open state stream
+StreamObserver requestObserver = stateService.state(responseObserver);
+
+// send state request
+BeamFnApi.StateRequest request =
+
BeamFnApi.StateRequest.newBuilder().setInstructionReference(bundleInstructionId).build();
+requestObserver.onNext(request);
+
+// assert behavior
+verify(handler).accept(eq(request), any());
+  }
+
+  @Test
+  public void testHandlerResponseSentToStateStream() throws Exception {
+// define handler behavior
+ByteString expectedResponseData =
+ByteString.copyFrom("EXPECTED_RESPONSE_DATA", StandardCharsets.UTF_8);
+String bundleInstructionId = "EXPECTED_BUNDLE_INSTRUCTION_ID";
+BeamFnApi.StateResponse.Builder expectedBuilder =
+BeamFnApi.StateResponse
+.newBuilder()
+
.setGet(BeamFnApi.StateGetResponse.newBuilder().setData(expectedResponseData));
+StateRequestHandler dummyHandler =
+(request, result) -> 
result.toCompletableFuture().complete(expectedBuilder);
+
+// define observer behavior
+CompletableFuture onNextCalled = new CompletableFuture<>();
+StreamObserver recordingResponseObserver =
+TestStreams
+.withOnNext(next -> {
+  synchronized (onNextCalled) {
 
 Review comment:
   Using a BlockingDeque instead in order to poll with a timeout.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #6223

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #6225

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

--
[...truncated 344.88 KB...]
2018-03-17T00:42:57.431 [INFO] 

2018-03-17T00:42:57.431 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2018-03-17T00:42:57.431 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.431 [INFO] 

2018-03-17T00:42:57.431 [INFO] 
2018-03-17T00:42:57.431 [INFO] 

2018-03-17T00:42:57.431 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2018-03-17T00:42:57.432 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] 
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2018-03-17T00:42:57.432 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] 
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] Skipping Apache Beam :: Examples :: Java
2018-03-17T00:42:57.432 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] 
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2018-03-17T00:42:57.432 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] 
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Jackson
2018-03-17T00:42:57.432 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] 
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Join library
2018-03-17T00:42:57.432 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] 
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Sketching
2018-03-17T00:42:57.432 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] 
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.432 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Sorter
2018-03-17T00:42:57.432 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.432 [INFO] 

2018-03-17T00:42:57.433 [INFO] 
2018-03-17T00:42:57.433 [INFO] 

2018-03-17T00:42:57.433 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: SQL
2018-03-17T00:42:57.433 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.433 [INFO] 

2018-03-17T00:42:57.433 [INFO] 
2018-03-17T00:42:57.433 [INFO] 

2018-03-17T00:42:57.433 [INFO] Skipping Apache Beam :: SDKs :: Java :: Nexmark
2018-03-17T00:42:57.433 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:42:57.433 [INFO] 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #4433

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

--
[...truncated 110.78 KB...]
2018-03-17T00:42:54.594 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2018-03-17T00:42:54.594 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2018-03-17T00:42:54.595 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2018-03-17T00:42:54.595 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4 from the shaded jar.
2018-03-17T00:42:54.595 [INFO] Including 
org.apache.commons:commons-compress:jar:1.14 in the shaded jar.
2018-03-17T00:42:54.595 [INFO] Including 
org.apache.commons:commons-lang3:jar:3.6 in the shaded jar.
2018-03-17T00:42:54.595 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2018-03-17T00:42:54.596 [INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded 
jar.
2018-03-17T00:42:54.596 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2018-03-17T00:42:54.596 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2018-03-17T00:43:05.273 [INFO] Replacing original artifact with shaded artifact.
2018-03-17T00:43:05.273 [INFO] Replacing 

 with 

2018-03-17T00:43:05.274 [INFO] Replacing original test artifact with shaded 
test artifact.
2018-03-17T00:43:05.274 [INFO] Replacing 

 with 

2018-03-17T00:43:05.274 [INFO] Dependency-reduced POM written at: 

2018-03-17T00:43:05.912 [INFO] 
2018-03-17T00:43:05.912 [INFO] --- maven-dependency-plugin:3.0.2:analyze-only 
(default) @ beam-sdks-java-core ---
2018-03-17T00:43:06.730 [INFO] No dependency problems found
[JENKINS] Archiving disabled
2018-03-17T00:43:09.417 [INFO] 
2018-03-17T00:43:09.417 [INFO] 

2018-03-17T00:43:09.417 [INFO] Building Apache Beam :: Runners 2.5.0-SNAPSHOT
2018-03-17T00:43:09.417 [INFO] 

2018-03-17T00:43:09.421 [INFO] 
2018-03-17T00:43:09.421 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-parent ---
2018-03-17T00:43:09.426 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md, 
**/sdks/python/apache_beam/portability/api/*pb2*.*, 
**/sdks/python/apache_beam/portability/common_urns.py, **/sdks/python/**/*.c, 
**/sdks/python/**/*.so, **/sdks/python/**/*.egg], excludes = [])
2018-03-17T00:43:09.707 [INFO] 
2018-03-17T00:43:09.707 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-maven-version) @ beam-runners-parent ---
2018-03-17T00:43:11.565 [INFO] 
2018-03-17T00:43:11.565 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce) @ beam-runners-parent ---
2018-03-17T00:43:11.568 [INFO] Adding ignore: module-info
2018-03-17T00:43:11.789 [INFO] 
2018-03-17T00:43:11.789 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-banned-dependencies) @ beam-runners-parent ---
2018-03-17T00:43:11.901 [INFO] 
2018-03-17T00:43:11.901 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-parent ---
2018-03-17T00:43:12.029 [INFO] 
2018-03-17T00:43:12.029 [INFO] --- maven-checkstyle-plugin:3.0.0:check 
(default) @ beam-runners-parent ---
2018-03-17T00:43:12.334 [INFO] Starting audit...
Audit done.
2018-03-17T00:43:12.485 [INFO] 
2018-03-17T00:43:12.485 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-runners-parent ---
2018-03-17T00:43:12.621 [INFO] 
2018-03-17T00:43:12.621 [INFO] --- maven-site-plugin:3.7:attach-descriptor 
(attach-descriptor) @ beam-runners-parent ---
2018-03-17T00:43:12.622 [INFO] No site descriptor found: nothing to attach.
2018-03-17T00:43:12.836 [INFO] 
2018-03-17T00:43:12.836 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-runners-parent ---
2018-03-17T00:43:12.837 [INFO] 

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81459=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81459
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:42
Start Date: 17/Mar/18 00:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175241095
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/package-info.java
 ##
 @@ -0,0 +1,20 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/** State api services. */
 
 Review comment:
   nit: `api` -> `API`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81459)
Time Spent: 1h 10m  (was: 1h)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81458=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81458
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:42
Start Date: 17/Mar/18 00:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175241253
 
 

 ##
 File path: 
runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/state/GrpcStateServiceTest.java
 ##
 @@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import com.google.protobuf.ByteString;
+import io.grpc.stub.StreamObserver;
+import java.nio.charset.StandardCharsets;
+import java.util.concurrent.CompletableFuture;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi;
+import org.apache.beam.sdk.fn.test.TestStreams;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.ArgumentCaptor;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.junit.Assert.assertThat;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.eq;
+import static org.mockito.Mockito.*;
+
+/** Tests for {@link 
org.apache.beam.runners.fnexecution.state.GrpcStateService} */
+@RunWith(JUnit4.class)
+public class GrpcStateServiceTest {
+  private static long TIMEOUT = 30 * 1000;
+
+  private GrpcStateService stateService;
+
+  @Mock
+  private StreamObserver responseObserver;
+
+  @Mock
+  private StateRequestHandler handler;
+
+  @Before
+  public void setUp() throws Exception {
+MockitoAnnotations.initMocks(this);
+stateService = new GrpcStateService();
+  }
+
+  /**
+   * After a handler has been registered with
+   * {@link GrpcStateService#registerForProcessBundleInstructionId(String, 
StateRequestHandler)},
+   * the {@link GrpcStateService} should delegate requests through
+   * {@link GrpcStateService#state(StreamObserver)} to the registered handler.
+   */
+  @Test
+  public void testStateRequestsHandledByRegisteredHandlers() throws Exception {
+// register handler
+String bundleInstructionId = "bundle_instruction";
+stateService.registerForProcessBundleInstructionId(bundleInstructionId, 
handler);
+
+// open state stream
+StreamObserver requestObserver = stateService.state(responseObserver);
+
+// send state request
+BeamFnApi.StateRequest request =
+
BeamFnApi.StateRequest.newBuilder().setInstructionReference(bundleInstructionId).build();
+requestObserver.onNext(request);
+
+// assert behavior
+verify(handler).accept(eq(request), any());
+  }
+
+  @Test
+  public void testHandlerResponseSentToStateStream() throws Exception {
+// define handler behavior
+ByteString expectedResponseData =
+ByteString.copyFrom("EXPECTED_RESPONSE_DATA", StandardCharsets.UTF_8);
+String bundleInstructionId = "EXPECTED_BUNDLE_INSTRUCTION_ID";
+BeamFnApi.StateResponse.Builder expectedBuilder =
+BeamFnApi.StateResponse
+.newBuilder()
+
.setGet(BeamFnApi.StateGetResponse.newBuilder().setData(expectedResponseData));
+StateRequestHandler dummyHandler =
+(request, result) -> 
result.toCompletableFuture().complete(expectedBuilder);
+
+// define observer behavior
+CompletableFuture onNextCalled = new CompletableFuture<>();
+StreamObserver recordingResponseObserver =
+TestStreams
+.withOnNext(next -> {
+  synchronized (onNextCalled) {
 
 Review comment:
   Follow the pattern here: 
https://github.com/apache/beam/blob/cb13361a6d4452e7f5e31af5974e8b90316a654a/sdks/java/harness/src/test/java/org/apache/beam/fn/harness/logging/BeamFnLoggingClientTest.java#L100
   
   Use a concurrent linked queue.

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81457=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81457
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:42
Start Date: 17/Mar/18 00:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175241059
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/StateService.java
 ##
 @@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.runners.fnexecution.FnService;
+
+/**
+ * The {@link StateService} is able to delegate {@link StateRequest}s to a set 
of registered
+ * handlers. Any request for an unregistered process bundle instruction id is 
automatically failed.
+ */
+public interface StateService extends FnService {
 
 Review comment:
   Rename to FnStateService mirroring existing naming like FnDataService


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81457)
Time Spent: 1h  (was: 50m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81462=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81462
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:42
Start Date: 17/Mar/18 00:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175240862
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
 ##
 @@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import static com.google.common.base.Throwables.getStackTraceAsString;
+
+import io.grpc.stub.StreamObserver;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnStateGrpc;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** An implementation of the Beam Fn State service. */
+public class GrpcStateService extends BeamFnStateGrpc.BeamFnStateImplBase 
implements StateService {
+  private static final Logger LOG = 
LoggerFactory.getLogger(GrpcStateService.class);
+  private final ConcurrentHashMap
+  requestHandlers;
+
+  public GrpcStateService()
+  throws Exception {
+this.requestHandlers = new ConcurrentHashMap<>();
+  }
+
+  @Override
+  public void close() throws Exception {
+// TODO: Track multiple clients and disconnect them cleanly instead of 
forcing termination
+  }
+
+  @Override
+  public StreamObserver state(StreamObserver 
responseObserver) {
+return new Inbound(responseObserver);
+  }
+
+  @Override
+  public AutoCloseable registerForProcessBundleInstructionId(
+  String processBundleInstructionId, StateRequestHandler handler) {
+requestHandlers.put(processBundleInstructionId, handler);
+return () -> requestHandlers.remove(processBundleInstructionId);
+  }
+
+  /**
+   * An inbound {@link StreamObserver} which delegates requests to registered 
handlers.
+   *
+   * TODO: Handle when the client indicates completion or an error on the 
inbound stream and
+   * there are pending requests.
+   */
+  private class Inbound implements StreamObserver {
+private final StreamObserver outboundObserver;
+
+Inbound(StreamObserver outboundObserver) {
+  this.outboundObserver = outboundObserver;
+}
+
+@Override
+public void onNext(StateRequest request) {
+  CompletionStage responseStage = new 
CompletableFuture<>();
+  responseStage.whenCompleteAsync(
+  (StateResponse.Builder responseBuilder, Throwable t) ->
+  outboundObserver.onNext(
 
 Review comment:
   Add a comment that this isn't thread safe since the outboundObserver isn't 
thread safe.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81462)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> 

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81461=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81461
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:42
Start Date: 17/Mar/18 00:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175240645
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
 ##
 @@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import static com.google.common.base.Throwables.getStackTraceAsString;
+
+import io.grpc.stub.StreamObserver;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnStateGrpc;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** An implementation of the Beam Fn State service. */
+public class GrpcStateService extends BeamFnStateGrpc.BeamFnStateImplBase 
implements StateService {
+  private static final Logger LOG = 
LoggerFactory.getLogger(GrpcStateService.class);
+  private final ConcurrentHashMap
+  requestHandlers;
 
 Review comment:
   nit: move to line above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81461)
Time Spent: 1.5h  (was: 1h 20m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81460=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81460
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:42
Start Date: 17/Mar/18 00:42
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on a change in pull request #4884: 
[BEAM-3676] Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884#discussion_r175240660
 
 

 ##
 File path: 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/GrpcStateService.java
 ##
 @@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.fnexecution.state;
+
+import static com.google.common.base.Throwables.getStackTraceAsString;
+
+import io.grpc.stub.StreamObserver;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.CompletionStage;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateRequest;
+import org.apache.beam.model.fnexecution.v1.BeamFnApi.StateResponse;
+import org.apache.beam.model.fnexecution.v1.BeamFnStateGrpc;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** An implementation of the Beam Fn State service. */
+public class GrpcStateService extends BeamFnStateGrpc.BeamFnStateImplBase 
implements StateService {
+  private static final Logger LOG = 
LoggerFactory.getLogger(GrpcStateService.class);
+  private final ConcurrentHashMap
+  requestHandlers;
+
+  public GrpcStateService()
+  throws Exception {
 
 Review comment:
   No longer throws exception


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81460)
Time Spent: 1h 20m  (was: 1h 10m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink #5256

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add Network Utility Methods

[tgroh] fixup! Add Network Utility Methods

[tgroh] Remove Future#set(null) in fn-execution

[herohde] [BEAM-3869] Fix Go Dataflow break due to zero timestamps

--
[...truncated 131.74 KB...]
2018-03-17T00:29:17.713 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-annotations:jar:2.8.9 from the shaded jar.
2018-03-17T00:29:17.713 [INFO] Excluding 
com.fasterxml.jackson.core:jackson-databind:jar:2.8.9 from the shaded jar.
2018-03-17T00:29:17.714 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2018-03-17T00:29:17.714 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2018-03-17T00:29:23.564 [INFO] Replacing original artifact with shaded artifact.
2018-03-17T00:29:23.565 [INFO] Replacing 

 with 

2018-03-17T00:29:23.565 [INFO] Replacing original test artifact with shaded 
test artifact.
2018-03-17T00:29:23.566 [INFO] Replacing 

 with 

2018-03-17T00:29:23.566 [INFO] Dependency-reduced POM written at: 

2018-03-17T00:29:24.546 [INFO] 
2018-03-17T00:29:24.546 [INFO] --- maven-dependency-plugin:3.0.2:analyze-only 
(default) @ beam-sdks-java-extensions-google-cloud-platform-core ---
2018-03-17T00:29:24.820 [INFO] No dependency problems found
[JENKINS] Archiving disabled
2018-03-17T00:29:27.425 [INFO] 
2018-03-17T00:29:27.425 [INFO] 

2018-03-17T00:29:27.425 [INFO] Building Apache Beam :: Runners 2.5.0-SNAPSHOT
2018-03-17T00:29:27.425 [INFO] 

2018-03-17T00:29:27.429 [INFO] 
2018-03-17T00:29:27.430 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-parent ---
2018-03-17T00:29:27.434 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md, 
**/sdks/python/apache_beam/portability/api/*pb2*.*, 
**/sdks/python/apache_beam/portability/common_urns.py, **/sdks/python/**/*.c, 
**/sdks/python/**/*.so, **/sdks/python/**/*.egg], excludes = [])
2018-03-17T00:29:28.871 [INFO] 
2018-03-17T00:29:28.872 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-maven-version) @ beam-runners-parent ---
2018-03-17T00:29:28.980 [INFO] 
2018-03-17T00:29:28.980 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce) @ beam-runners-parent ---
2018-03-17T00:29:28.982 [INFO] Adding ignore: module-info
2018-03-17T00:29:29.090 [INFO] 
2018-03-17T00:29:29.090 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-banned-dependencies) @ beam-runners-parent ---
2018-03-17T00:29:29.203 [INFO] 
2018-03-17T00:29:29.203 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-parent ---
2018-03-17T00:29:29.569 [INFO] 
2018-03-17T00:29:29.569 [INFO] --- maven-checkstyle-plugin:3.0.0:check 
(default) @ beam-runners-parent ---
2018-03-17T00:29:30.186 [INFO] Starting audit...
Audit done.
2018-03-17T00:29:30.313 [INFO] 
2018-03-17T00:29:30.314 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-runners-parent ---
2018-03-17T00:29:30.430 [INFO] 
2018-03-17T00:29:30.430 [INFO] --- maven-site-plugin:3.7:attach-descriptor 
(attach-descriptor) @ beam-runners-parent ---
2018-03-17T00:29:30.431 [INFO] No site descriptor found: nothing to attach.
2018-03-17T00:29:30.973 [INFO] 
2018-03-17T00:29:30.973 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 
beam-runners-parent ---
2018-03-17T00:29:30.975 [INFO] Skipping packaging of the jar
2018-03-17T00:29:31.196 [INFO] 
2018-03-17T00:29:31.196 [INFO] --- maven-jar-plugin:3.0.2:test-jar 
(default-test-jar) @ beam-runners-parent ---

[jira] [Resolved] (BEAM-3869) Go Dataflow jobs fail with EventTime error

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3869.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> Go Dataflow jobs fail with EventTime error
> --
>
> Key: BEAM-3869
> URL: https://issues.apache.org/jira/browse/BEAM-3869
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Critical
> Fix For: 2.5.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Suspect that the stricter checking in 
> https://github.com/apache/beam/pull/4843 for the join example:
> 2018/03/16 15:28:30 Failed to execute job: received a zero EventTime, which 
> is unencodable



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81455=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81455
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:23
Start Date: 17/Mar/18 00:23
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #4801: [BEAM-3738] Enable 
py3 lint
URL: https://github.com/apache/beam/pull/4801
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index 5ed623b2293..5ef1683e81b 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -204,7 +204,6 @@ def generate_common_urns():
 ]),
 setup_requires=REQUIRED_SETUP_PACKAGES,
 install_requires=REQUIRED_PACKAGES,
-python_requires='>=2.7,<3.0',
 test_suite='nose.collector',
 tests_require=REQUIRED_TEST_PACKAGES,
 extras_require={
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 857e7b0e571..7d4152602f8 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -17,8 +17,7 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27gcp,py27cython,lint_py2,lint_py3,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
@@ -103,7 +102,6 @@ whitelist_externals=time
 commands =
   python --version
   pip --version
-  time pip install -e .[test]
   time {toxinidir}/run_pylint.sh
 passenv = TRAVIS*
 
@@ -116,12 +114,13 @@ deps=
   isort==4.2.15
   flake8==3.5.0
 whitelist_externals=time
+basepython=python3
 commands =
-  time pip install -e .[test]
+  python --version
+  pip --version
   time {toxinidir}/run_mini_py3lint.sh
 passenv = TRAVIS*
 
-
 [testenv:docs]
 deps=
   nose==1.3.7


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81455)
Time Spent: 5h 40m  (was: 5.5h)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81454=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81454
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:23
Start Date: 17/Mar/18 00:23
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #4801: [BEAM-3738] Enable py3 
lint
URL: https://github.com/apache/beam/pull/4801#issuecomment-373878311
 
 
   Thank you @udim. Closing this PR in favor of #4877.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81454)
Time Spent: 5.5h  (was: 5h 20m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3759) Add support for PaneInfo descriptor in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3759?focusedWorklogId=81453=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81453
 ]

ASF GitHub Bot logged work on BEAM-3759:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:19
Start Date: 17/Mar/18 00:19
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #4763: [BEAM-3759] Add 
support for PaneInfo in WindowedValues
URL: https://github.com/apache/beam/pull/4763#issuecomment-373877868
 
 
   @charlesccychen any updates on this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81453)
Time Spent: 2h  (was: 1h 50m)

> Add support for PaneInfo descriptor in Python SDK
> -
>
> Key: BEAM-3759
> URL: https://issues.apache.org/jira/browse/BEAM-3759
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Affects Versions: 2.3.0
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> The PaneInfo descriptor allows a user to determine which particular 
> triggering emitted a value.  This allows the user to differentiate between 
> speculative (early), on-time (at end of window) and late value emissions 
> coming out of a GroupByKey.  We should add support for this feature in the 
> Python SDK.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #6224

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add Network Utility Methods

[tgroh] fixup! Add Network Utility Methods

--
[...truncated 345.07 KB...]
2018-03-17T00:14:01.027 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Solr
2018-03-17T00:14:01.028 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: SDKs :: Java :: IO :: 
Tika
2018-03-17T00:14:01.028 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Starter
2018-03-17T00:14:01.028 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: Examples :: Java
2018-03-17T00:14:01.028 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: SDKs :: Java :: Maven 
Archetypes :: Examples
2018-03-17T00:14:01.028 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Jackson
2018-03-17T00:14:01.028 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Join library
2018-03-17T00:14:01.028 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Sketching
2018-03-17T00:14:01.028 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] 
2018-03-17T00:14:01.028 [INFO] 

2018-03-17T00:14:01.028 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: Sorter
2018-03-17T00:14:01.029 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.029 [INFO] 

2018-03-17T00:14:01.029 [INFO] 
2018-03-17T00:14:01.029 [INFO] 

2018-03-17T00:14:01.029 [INFO] Skipping Apache Beam :: SDKs :: Java :: 
Extensions :: SQL
2018-03-17T00:14:01.029 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.029 [INFO] 

2018-03-17T00:14:01.029 [INFO] 
2018-03-17T00:14:01.029 [INFO] 

2018-03-17T00:14:01.029 [INFO] Skipping Apache Beam :: SDKs :: Java :: Nexmark
2018-03-17T00:14:01.029 [INFO] This project has been banned from the build due 
to previous failures.
2018-03-17T00:14:01.029 [INFO] 

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81452=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81452
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:13
Start Date: 17/Mar/18 00:13
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4838: 
[BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175239845
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -99,6 +113,19 @@ def finalize_value_and_get_tasks(self, side_input):
   view.has_result = True
   return result
 
+  def update_watermarks_for_transform(self, ptransform, watermark):
+# Collect tasks that get unblocked as the workflow progresses
+unblocked_tasks = []
+for view in self._transform_to_views[ptransform]:
+  unblocked_tasks.extend(self._update_watermarks_for_view(view, watermark))
+return unblocked_tasks
+
+  def _update_watermarks_for_view(self, view, watermark):
+unblocked_tasks = []
+if watermark.input_watermark == WatermarkManager.WATERMARK_POS_INF:
 
 Review comment:
   Thank you. Sounds good, let's do that.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81452)
Time Spent: 1h  (was: 50m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81451=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81451
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:10
Start Date: 17/Mar/18 00:10
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #4884: [BEAM-3676] Add 
Portable State Service 
URL: https://github.com/apache/beam/pull/4884#issuecomment-373876789
 
 
   R: @tgroh @lukecwik


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81451)
Time Spent: 50m  (was: 40m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81450
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:09
Start Date: 17/Mar/18 00:09
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #4884: [BEAM-3676] Add 
Portable State Service 
URL: https://github.com/apache/beam/pull/4884#issuecomment-373876789
 
 
   R: @tghroh, @lukecwik


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81450)
Time Spent: 40m  (was: 0.5h)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81449=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81449
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:09
Start Date: 17/Mar/18 00:09
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #4884: [BEAM-3676] Add 
Portable State Service 
URL: https://github.com/apache/beam/pull/4884#issuecomment-373876789
 
 
   R: tgroh@
   
   CC: lukecwik@


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81449)
Time Spent: 0.5h  (was: 20m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81448
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:08
Start Date: 17/Mar/18 00:08
Worklog Time Spent: 10m 
  Work Description: axelmagn commented on issue #4884: [BEAM-3676] Add 
Portable State Service 
URL: https://github.com/apache/beam/pull/4884#issuecomment-373876789
 
 
   R: tgroh@
   CC: lukecwik


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81448)
Time Spent: 20m  (was: 10m)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3676) FlinkRunner: Portable state service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?focusedWorklogId=81447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81447
 ]

ASF GitHub Bot logged work on BEAM-3676:


Author: ASF GitHub Bot
Created on: 17/Mar/18 00:07
Start Date: 17/Mar/18 00:07
Worklog Time Spent: 10m 
  Work Description: axelmagn opened a new pull request #4884: [BEAM-3676] 
Add Portable State Service 
URL: https://github.com/apache/beam/pull/4884
 
 
   This adds a java implementation of a portable state service in 
`runners/java-fn-execution`, which can be used among runners for the 
portability framework.  It delegates the handling of state requests through a 
handler interface that can be registered against the service, so that runners 
can register their own handler implementations against it.
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [x] Write a pull request description that is detailed enough to 
understand:
  - [x] What the pull request does
  - [x] Why it does it
  - [x] How it does it
  - [x] Why this approach
- [x] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81447)
Time Spent: 10m
Remaining Estimate: 0h

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3565) Add utilities for producing a collection of PTransforms that can execute in a single SDK Harness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3565?focusedWorklogId=81441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81441
 ]

ASF GitHub Bot logged work on BEAM-3565:


Author: ASF GitHub Bot
Created on: 16/Mar/18 23:57
Start Date: 16/Mar/18 23:57
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #4777: 
[BEAM-3565] Add FusedPipeline#toPipeline
URL: https://github.com/apache/beam/pull/4777#discussion_r175238617
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/QueryablePipeline.java
 ##
 @@ -161,6 +179,40 @@ private static boolean isPrimitiveTransform(PTransform 
transform) {
 return network;
   }
 
+  // This enables a naive implementation of topological sort, instead of doing 
something clever.
+  // Nodes with lower weight precede nodes with higher weight, and are 
unrelated to nodes with
+  // equal weight
+  private final LoadingCache nodeWeights =
+  CacheBuilder.newBuilder()
+  .build(
+  new CacheLoader() {
+@Override
+public Long load(@Nonnull PTransformNode transformNode) {
+  long parentWeight = 0L;
+  for (String inputPCollectionId :
+  transformNode.getTransform().getInputsMap().values()) {
+PTransformNode upstream =
+getProducer(
+PipelineNode.pCollection(
+inputPCollectionId,
+
components.getPcollectionsOrThrow(inputPCollectionId)));
+parentWeight += nodeWeights.getUnchecked(upstream);
+  }
+  return 1 + parentWeight;
+}
+  });
+
+  public Iterable getTopologicallyOrderedTransforms() {
+return pipelineNetwork
+.nodes()
+.stream()
+.filter(node -> node instanceof PTransformNode)
+.map(PTransformNode.class::cast)
+.collect(
+Collectors.toCollection(
+() -> new 
TreeSet<>(Comparator.comparingLong(nodeWeights::getUnchecked;
 
 Review comment:
   This comparator is really extremely inconsistent with equals, and will make 
it so there only looks like there's one root transform.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81441)
Time Spent: 13h  (was: 12h 50m)

> Add utilities for producing a collection of PTransforms that can execute in a 
> single SDK Harness
> 
>
> Key: BEAM-3565
> URL: https://issues.apache.org/jira/browse/BEAM-3565
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>Priority: Major
>  Labels: portability
> Fix For: 2.4.0
>
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> An SDK Harness executes some ("fused") collection of PTransforms. The java 
> runner libraries should provide some way to take a Pipeline that executes in 
> both a runner and an environment and construct a collection of transforms 
> which can execute within a single environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4880: Remove Future#set(null) in fn-execution

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit cb13361a6d4452e7f5e31af5974e8b90316a654a
Merge: 4bd5006 84f043f
Author: Thomas Groh 
AuthorDate: Fri Mar 16 16:56:58 2018 -0700

Merge pull request #4880: Remove Future#set(null) in fn-execution

Remove Future#set(null) in fn-execution

 .../src/main/resources/beam/findbugs-filter.xml| 41 --
 .../sdk/fn/data/BeamFnDataInboundObserver.java |  2 +-
 .../data/CompletableFutureInboundDataClient.java   | 12 ++---
 .../fn/data/SettableFutureInboundDataClient.java   | 63 --
 .../CompletableFutureInboundDataClientTest.java|  9 ++--
 .../fn/harness/control/BeamFnControlClient.java|  7 ++-
 .../fn/harness/logging/BeamFnLoggingClient.java|  4 +-
 .../beam/fn/harness/BeamFnDataReadRunnerTest.java  |  2 +-
 .../data/BeamFnDataInboundObserverTest.java|  1 +
 9 files changed, 21 insertions(+), 120 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-3869) Go Dataflow jobs fail with EventTime error

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3869?focusedWorklogId=81440=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81440
 ]

ASF GitHub Bot logged work on BEAM-3869:


Author: ASF GitHub Bot
Created on: 16/Mar/18 23:56
Start Date: 16/Mar/18 23:56
Worklog Time Spent: 10m 
  Work Description: lukecwik closed pull request #4883: [BEAM-3869] Fix Go 
Dataflow break due to zero timestamps
URL: https://github.com/apache/beam/pull/4883
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/go/pkg/beam/core/runtime/exec/coder.go 
b/sdks/go/pkg/beam/core/runtime/exec/coder.go
index 504f6e6a868..e41be11f94f 100644
--- a/sdks/go/pkg/beam/core/runtime/exec/coder.go
+++ b/sdks/go/pkg/beam/core/runtime/exec/coder.go
@@ -282,6 +282,9 @@ func (c *kvDecoder) Decode(r io.Reader) (FullValue, error) {
 func EncodeWindowedValueHeader(c *coder.Coder, t typex.EventTime, w io.Writer) 
error {
// Encoding: Timestamp, Window, Pane (header) + Element
 
+   if (time.Time)(t).IsZero() {
+   t = typex.EventTime(time.Now())
+   }
if err := coder.EncodeEventTime(t, w); err != nil {
return err
}
diff --git a/sdks/go/pkg/beam/runners/dataflow/translate.go 
b/sdks/go/pkg/beam/runners/dataflow/translate.go
index aae0654b082..17948e080e5 100644
--- a/sdks/go/pkg/beam/runners/dataflow/translate.go
+++ b/sdks/go/pkg/beam/runners/dataflow/translate.go
@@ -309,7 +309,7 @@ func translateEdge(edge *graph.MultiEdge) (string, 
properties, error) {
// URL Query-escaped windowed _unnested_ value. It is read back 
in
// a nested context at runtime.
var buf bytes.Buffer
-   if err := exec.EncodeWindowedValueHeader(c, 
beam.EventTime(time.Time{}), ); err != nil {
+   if err := exec.EncodeWindowedValueHeader(c, 
beam.EventTime(time.Now()), ); err != nil {
return "", properties{}, err
}
value := string(append(buf.Bytes(), edge.Value...))


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81440)
Time Spent: 0.5h  (was: 20m)

> Go Dataflow jobs fail with EventTime error
> --
>
> Key: BEAM-3869
> URL: https://issues.apache.org/jira/browse/BEAM-3869
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Critical
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Suspect that the stricter checking in 
> https://github.com/apache/beam/pull/4843 for the join example:
> 2018/03/16 15:28:30 Failed to execute job: received a zero EventTime, which 
> is unencodable



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (4bd5006 -> cb13361)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4bd5006  [BEAM-3869] Fix Go Dataflow break due to zero timestamps
 add 84f043f  Remove Future#set(null) in fn-execution
 new cb13361  Merge pull request #4880: Remove Future#set(null) in 
fn-execution

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../src/main/resources/beam/findbugs-filter.xml| 41 --
 .../sdk/fn/data/BeamFnDataInboundObserver.java |  2 +-
 .../data/CompletableFutureInboundDataClient.java   | 12 ++---
 .../fn/data/SettableFutureInboundDataClient.java   | 63 --
 .../CompletableFutureInboundDataClientTest.java|  9 ++--
 .../fn/harness/control/BeamFnControlClient.java|  7 ++-
 .../fn/harness/logging/BeamFnLoggingClient.java|  4 +-
 .../beam/fn/harness/BeamFnDataReadRunnerTest.java  |  2 +-
 .../data/BeamFnDataInboundObserverTest.java|  1 +
 9 files changed, 21 insertions(+), 120 deletions(-)
 rename sdks/java/{harness/src/main/java/org/apache/beam/fn/harness => 
fn-execution/src/main/java/org/apache/beam/sdk/fn}/data/CompletableFutureInboundDataClient.java
 (84%)
 delete mode 100644 
sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/SettableFutureInboundDataClient.java
 rename sdks/java/{harness/src/test/java/org/apache/beam/fn/harness => 
fn-execution/src/test/java/org/apache/beam/sdk/fn}/data/CompletableFutureInboundDataClientTest.java
 (94%)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Assigned] (BEAM-3676) FlinkRunner: Portable state service

[Axel Magnuson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Axel Magnuson reassigned BEAM-3676:
---

Assignee: Axel Magnuson  (was: Aljoscha Krettek)

> FlinkRunner: Portable state service
> ---
>
> Key: BEAM-3676
> URL: https://issues.apache.org/jira/browse/BEAM-3676
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Axel Magnuson
>Priority: Major
>
> The State API is an implementation of BeamFnState that exposes pipeline state 
> to SDK harnesses. Because it is used for side inputs, this service will also 
> need to be tied into side inputs/outputs during the translation phase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #4432

[Apache Jenkins Server]

See 


Changes:

[ankurgoenka] Send error response for process bundle progress instead of 
throwing and

[tgroh] Add Network Utility Methods

[tgroh] fixup! Add Network Utility Methods

--
[...truncated 510.85 KB...]
2018-03-16T23:39:37.793 [INFO] Excluding 
org.codehaus.jackson:jackson-core-asl:jar:1.9.13 from the shaded jar.
2018-03-16T23:39:37.793 [INFO] Excluding 
org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 from the shaded jar.
2018-03-16T23:39:37.793 [INFO] Excluding 
com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded jar.
2018-03-16T23:39:37.793 [INFO] Excluding 
org.xerial.snappy:snappy-java:jar:1.1.4 from the shaded jar.
2018-03-16T23:39:37.793 [INFO] Including 
org.apache.commons:commons-compress:jar:1.14 in the shaded jar.
2018-03-16T23:39:37.793 [INFO] Including 
org.apache.commons:commons-lang3:jar:3.6 in the shaded jar.
2018-03-16T23:39:37.793 [INFO] Excluding joda-time:joda-time:jar:2.4 from the 
shaded jar.
2018-03-16T23:39:37.793 [INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded 
jar.
2018-03-16T23:39:37.793 [INFO] Excluding 
com.google.auto.service:auto-service:jar:1.0-rc2 from the shaded jar.
2018-03-16T23:39:37.793 [INFO] Excluding com.google.auto:auto-common:jar:0.3 
from the shaded jar.
2018-03-16T23:39:45.055 [INFO] Replacing original artifact with shaded artifact.
2018-03-16T23:39:45.056 [INFO] Replacing 

 with 

2018-03-16T23:39:45.056 [INFO] Replacing original test artifact with shaded 
test artifact.
2018-03-16T23:39:45.056 [INFO] Replacing 

 with 

2018-03-16T23:39:45.056 [INFO] Dependency-reduced POM written at: 

2018-03-16T23:39:45.344 [INFO] 
2018-03-16T23:39:45.344 [INFO] --- maven-dependency-plugin:3.0.2:analyze-only 
(default) @ beam-sdks-java-core ---
2018-03-16T23:39:45.910 [INFO] No dependency problems found
[JENKINS] Archiving disabled
2018-03-16T23:39:50.922 [INFO] 
2018-03-16T23:39:50.922 [INFO] 

2018-03-16T23:39:50.922 [INFO] Building Apache Beam :: Runners 2.5.0-SNAPSHOT
2018-03-16T23:39:50.922 [INFO] 

2018-03-16T23:39:50.926 [INFO] 
2018-03-16T23:39:50.926 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-runners-parent ---
2018-03-16T23:39:50.931 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md, 
**/sdks/python/apache_beam/portability/api/*pb2*.*, 
**/sdks/python/apache_beam/portability/common_urns.py, **/sdks/python/**/*.c, 
**/sdks/python/**/*.so, **/sdks/python/**/*.egg], excludes = [])
2018-03-16T23:39:51.203 [INFO] 
2018-03-16T23:39:51.203 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-maven-version) @ beam-runners-parent ---
2018-03-16T23:39:51.468 [INFO] 
2018-03-16T23:39:51.469 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce) @ beam-runners-parent ---
2018-03-16T23:39:51.471 [INFO] Adding ignore: module-info
2018-03-16T23:39:51.579 [INFO] 
2018-03-16T23:39:51.579 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-banned-dependencies) @ beam-runners-parent ---
2018-03-16T23:39:51.705 [INFO] 
2018-03-16T23:39:51.705 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-runners-parent ---
2018-03-16T23:39:52.033 [INFO] 
2018-03-16T23:39:52.033 [INFO] --- maven-checkstyle-plugin:3.0.0:check 
(default) @ beam-runners-parent ---
2018-03-16T23:39:52.193 [INFO] Starting audit...
Audit done.
2018-03-16T23:39:52.581 [INFO] 
2018-03-16T23:39:52.581 [INFO] --- 
build-helper-maven-plugin:3.0.0:regex-properties (render-artifact-id) @ 
beam-runners-parent ---
2018-03-16T23:39:52.698 [INFO] 
2018-03-16T23:39:52.698 [INFO] --- maven-site-plugin:3.7:attach-descriptor 
(attach-descriptor) @ beam-runners-parent ---
2018-03-16T23:39:52.699 [INFO] No site descriptor found: nothing to attach.
2018-03-16T23:39:52.916 [INFO] 
2018-03-16T23:39:52.916 [INFO] --- maven-jar-plugin:3.0.2:jar (default-jar) @ 

[jira] [Work logged] (BEAM-3865) Incorrect timestamp on merging window outputs.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3865?focusedWorklogId=81435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81435
 ]

ASF GitHub Bot logged work on BEAM-3865:


Author: ASF GitHub Bot
Created on: 16/Mar/18 23:37
Start Date: 16/Mar/18 23:37
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on a change in pull request 
#4879: [BEAM-3865] Fix watermark hold handling bug.
URL: https://github.com/apache/beam/pull/4879#discussion_r175236867
 
 

 ##
 File path: sdks/python/apache_beam/transforms/trigger_test.py
 ##
 @@ -81,16 +84,8 @@ def bundle_data(data, size):
 
 if not groupings:
   groupings = [1]
+grouping = list(groupings) + [-group_by for group_by in groupings]
 
 Review comment:
   
   What is this?  This is currently a no-op.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81435)
Time Spent: 20m  (was: 10m)

> Incorrect timestamp on merging window outputs.
> --
>
> Key: BEAM-3865
> URL: https://issues.apache.org/jira/browse/BEAM-3865
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Robert Bradshaw
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Looks like we're setting multiple watermark holds with one arbitrarily being 
> held. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3867) Add support for decompressing and reading tar files

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403120#comment-16403120
 ] 

Eugene Kirpichov commented on BEAM-3867:


I think you'd simply need to add something to the canonical Compression enum 
[https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Compression.java]
 and make sure that ".tar.gz" some how has precedence over simply ".gz"

> Add support for decompressing and reading tar files
> ---
>
> Key: BEAM-3867
> URL: https://issues.apache.org/jira/browse/BEAM-3867
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Sajeevan Achuthan
>Priority: Major
>
> See below for a thread regarding this.
> https://lists.apache.org/thread.html/eb006101b9d3cb6bf88ec1f8f29e2aaec96ab5760fb7041336fa541a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Add Network Utility Methods

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c6f1d3d6275c6e4acf02025ecf3e8135b5bb0ebc
Merge: 8f32183 f3a9f91
Author: Lukasz Cwik 
AuthorDate: Fri Mar 16 16:28:43 2018 -0700

Add Network Utility Methods

 .../runners/core/construction/graph/Networks.java  | 249 +
 .../core/construction/graph/NetworksTest.java  | 239 
 2 files changed, 488 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] branch master updated (8f32183 -> c6f1d3d)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8f32183  Merge pull request #4604 from angoenka/progress_report_error
 add d78ea30  Add Network Utility Methods
 add f3a9f91  fixup! Add Network Utility Methods
 new c6f1d3d  Add Network Utility Methods

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/core/construction/graph/Networks.java  | 249 +
 .../core/construction/graph/NetworksTest.java  | 239 
 2 files changed, 488 insertions(+)
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/Networks.java
 create mode 100644 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/NetworksTest.java

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Assigned] (BEAM-3867) Add support for decompressing and reading tar files

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-3867:


Assignee: Sajeevan Achuthan

> Add support for decompressing and reading tar files
> ---
>
> Key: BEAM-3867
> URL: https://issues.apache.org/jira/browse/BEAM-3867
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-core
>Reporter: Chamikara Jayalath
>Assignee: Sajeevan Achuthan
>Priority: Major
>
> See below for a thread regarding this.
> https://lists.apache.org/thread.html/eb006101b9d3cb6bf88ec1f8f29e2aaec96ab5760fb7041336fa541a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4446

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3303) Go windowing support

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403101#comment-16403101
 ] 

Henning Rohde commented on BEAM-3303:
-

We should also ensure that event time is properly propagated in the runtime.

> Go windowing support
> 
>
> Key: BEAM-3303
> URL: https://issues.apache.org/jira/browse/BEAM-3303
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>
> Add support for Window.into and windowing strategies on Node. Implement the 
> various windowing strategies Beam has: GlobalWindow, FixedWindows, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3225) Non deterministic behaviour of AfterProcessingTime trigger with multiple group by transformations

[Pawel Bartoszek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403089#comment-16403089
 ] 

Pawel Bartoszek commented on BEAM-3225:
---

[~dawidwys] I think this issue might have been caused by 
https://issues.apache.org/jira/browse/BEAM-3863 

> Non deterministic behaviour of AfterProcessingTime trigger with multiple 
> group by transformations
> -
>
> Key: BEAM-3225
> URL: https://issues.apache.org/jira/browse/BEAM-3225
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core, runner-flink
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Pawel Bartoszek
>Assignee: Aljoscha Krettek
>Priority: Critical
>
> *Context*
> I run my [test 
> code|https://gist.github.com/pbartoszek/fe6b1d7501333a2a4b385a1424f6bd80] 
> against different triggers and runners. My original problem was that when 
> writing to a file sink files weren't always produced in a deterministic way. 
> Please refer to this 
> [BEAM-3151|https://issues.apache.org/jira/browse/BEAM-3151] When I started 
> looking at WriteFiles class I noticed that file sink implementation includes 
> some multiple GroupByKey transformations. Knowing that I wrote my test code 
> that is using multiple GroupByKey transformations to conclude that this is a 
> bit buggy(?) support of After(Synchronised)ProcessingTime triggers by 
> GroupByKey that also influence the file sink. When I run my job using 
> Dataflow runner I was getting expected output.
> *About test code*
> The job is counting how many A and B elements it received within 30 sec 
> windows using Count.perElement. Then I am using GroupByKey to fire every time 
> count has increased.
> Below I outlined the expected standard output: 
> {code:java}
> Let's assume all events are received in the same 30 seconds window.
> A -> After count KV{A, 1} -> Final group by KV{A, [1]}
> A -> After count KV{A, 2} -> Final group by KV{A, [1,2]}
> A -> After count KV{A, 3} -> Final group by KV{A, [1,2,3]}
> B -> After count KV{B, 1} -> Final group by KV{B, [1]}
> With my trigger configuration I would expect that for every new element 
> 'After count' is printed with new value followed by 'Final group by' with new 
> counter. Final group by represents the history of counters then.{code}
>  
> *Problem*
> 'Final group by' trigger doesn't always go off although trigger set up would 
> suggest that. This behaviour is different for different runners and Beam 
> versions. 
>  
> *My observations when using Pubsub*
> Trigger configuration
> {code:java}
> Window.into(FixedWindows.of(standardSeconds(30)))
> 
> .triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane())) 
>  
> .withAllowedLateness(standardSeconds(5), 
> Window.ClosingBehavior.FIRE_IF_NON_EMPTY)
> .accumulatingFiredPanes())
> {code}
>  
>  Beam 2.0 Flink Runner
> {code:java}
> 2017-11-16T14:51:44.294Z After count KV{A, 1} 
> [2017-11-16T14:51:30.000Z..2017-11-16T14:52:00.000Z)
> 2017-11-16T14:51:53.036Z Received Element A 
> [2017-11-16T14:51:30.000Z..2017-11-16T14:52:00.000Z)
> 2017-11-16T14:51:53.143Z After count KV{A, 2} 
> [2017-11-16T14:51:30.000Z..2017-11-16T14:52:00.000Z)
> 2017-11-16T14:51:53.246Z Final group by KV{A, [1, 2]} 
> [2017-11-16T14:51:30.000Z..2017-11-16T14:52:00.000Z)
> 2017-11-16T14:52:03.522Z Received Element A 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:03.629Z After count KV{A, 1} 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:03.732Z Final group by KV{A, [1]} 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:07.270Z Received Element A 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:07.372Z After count KV{A, 2} 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:07.476Z Final group by KV{A, [1, 2]} 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:10.394Z Received Element A 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:10.501Z After count KV{A, 3} 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:10.602Z Final group by KV{A, [1, 2, 3]} 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:13.296Z Received Element A 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:13.402Z After count KV{A, 4} 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:13.507Z Final group by KV{A, [1, 2, 3, 4]} 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z)
> 2017-11-16T14:52:14.951Z Received Element A 
> [2017-11-16T14:52:00.000Z..2017-11-16T14:52:30.000Z) <--- Expected to see 
> 'After 

[jira] [Work logged] (BEAM-3869) Go Dataflow jobs fail with EventTime error

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3869?focusedWorklogId=81427=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81427
 ]

ASF GitHub Bot logged work on BEAM-3869:


Author: ASF GitHub Bot
Created on: 16/Mar/18 23:03
Start Date: 16/Mar/18 23:03
Worklog Time Spent: 10m 
  Work Description: herohde opened a new pull request #4883: [BEAM-3869] 
Fix Go Dataflow break due to zero timestamps
URL: https://github.com/apache/beam/pull/4883
 
 
   The merge of #4693 broke Go Dataflow.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81427)
Time Spent: 10m
Remaining Estimate: 0h

> Go Dataflow jobs fail with EventTime error
> --
>
> Key: BEAM-3869
> URL: https://issues.apache.org/jira/browse/BEAM-3869
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Critical
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Suspect that the stricter checking in 
> https://github.com/apache/beam/pull/4843 for the join example:
> 2018/03/16 15:28:30 Failed to execute job: received a zero EventTime, which 
> is unencodable



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3869) Go Dataflow jobs fail with EventTime error

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3869?focusedWorklogId=81428=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81428
 ]

ASF GitHub Bot logged work on BEAM-3869:


Author: ASF GitHub Bot
Created on: 16/Mar/18 23:03
Start Date: 16/Mar/18 23:03
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #4883: [BEAM-3869] Fix Go 
Dataflow break due to zero timestamps
URL: https://github.com/apache/beam/pull/4883#issuecomment-373867754
 
 
   R: @lostluck @lukecwik 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81428)
Time Spent: 20m  (was: 10m)

> Go Dataflow jobs fail with EventTime error
> --
>
> Key: BEAM-3869
> URL: https://issues.apache.org/jira/browse/BEAM-3869
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Critical
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Suspect that the stricter checking in 
> https://github.com/apache/beam/pull/4843 for the join example:
> 2018/03/16 15:28:30 Failed to execute job: received a zero EventTime, which 
> is unencodable



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3869) Go Dataflow jobs fail with EventTime error

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-3869:

Priority: Critical  (was: Major)

> Go Dataflow jobs fail with EventTime error
> --
>
> Key: BEAM-3869
> URL: https://issues.apache.org/jira/browse/BEAM-3869
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Critical
>
> Suspect that the stricter checking in 
> https://github.com/apache/beam/pull/4843 for the join example:
> 2018/03/16 15:28:30 Failed to execute job: received a zero EventTime, which 
> is unencodable



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (a7bc72d -> 8f32183)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a7bc72d  Enable input subscription in Python streaming wordcount 
(#4864)
 add eed5a5a  Send error response for process bundle progress instead of 
throwing and catching exception
 new 8f32183  Merge pull request #4604 from angoenka/progress_report_error

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/worker/sdk_worker.py | 20 +---
 1 file changed, 17 insertions(+), 3 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #6222

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81413
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 16/Mar/18 22:33
Start Date: 16/Mar/18 22:33
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on a change in pull request 
#4838: [BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175229449
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -99,6 +113,19 @@ def finalize_value_and_get_tasks(self, side_input):
   view.has_result = True
   return result
 
+  def update_watermarks_for_transform(self, ptransform, watermark):
+# Collect tasks that get unblocked as the workflow progresses
+unblocked_tasks = []
+for view in self._transform_to_views[ptransform]:
+  unblocked_tasks.extend(self._update_watermarks_for_view(view, watermark))
+return unblocked_tasks
+
+  def _update_watermarks_for_view(self, view, watermark):
+unblocked_tasks = []
+if watermark.input_watermark == WatermarkManager.WATERMARK_POS_INF:
 
 Review comment:
   Yes, this is part one of a refactor that will deliver the streaming side 
input functionality.  I've synced with Maria, and she has the other parts of 
the refactor after this one goes in.  Perhaps we should clarify the title of 
the PR though.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81413)
Time Spent: 50m  (was: 40m)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=81410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81410
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 16/Mar/18 22:31
Start Date: 16/Mar/18 22:31
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4838: 
[BEAM-3818] Add support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#discussion_r175229093
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/evaluation_context.py
 ##
 @@ -99,6 +113,19 @@ def finalize_value_and_get_tasks(self, side_input):
   view.has_result = True
   return result
 
+  def update_watermarks_for_transform(self, ptransform, watermark):
+# Collect tasks that get unblocked as the workflow progresses
+unblocked_tasks = []
+for view in self._transform_to_views[ptransform]:
+  unblocked_tasks.extend(self._update_watermarks_for_view(view, watermark))
+return unblocked_tasks
+
+  def _update_watermarks_for_view(self, view, watermark):
+unblocked_tasks = []
+if watermark.input_watermark == WatermarkManager.WATERMARK_POS_INF:
 
 Review comment:
   Would not waiting for `WatermarkManager.WATERMARK_POS_INF` unblock side 
inputs only when they are completely finished?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81410)
Time Spent: 40m  (was: 0.5h)

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3869) Go Dataflow jobs fail with EventTime error

[Henning Rohde (JIRA)]

Henning Rohde created BEAM-3869:
---

 Summary: Go Dataflow jobs fail with EventTime error
 Key: BEAM-3869
 URL: https://issues.apache.org/jira/browse/BEAM-3869
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Henning Rohde
Assignee: Henning Rohde


Suspect that the stricter checking in https://github.com/apache/beam/pull/4843 
for the join example:

2018/03/16 15:28:30 Failed to execute job: received a zero EventTime, which is 
unencodable



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=81409=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81409
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 16/Mar/18 22:26
Start Date: 16/Mar/18 22:26
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-373861600
 
 
   Changes are made base on comments. @tgroh 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81409)
Time Spent: 69h  (was: 68h 50m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 69h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=81408=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81408
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 16/Mar/18 22:19
Start Date: 16/Mar/18 22:19
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-373860350
 
 
   @aaltay Changes are made base on comments:
   
   1. rename files being "run_release_candidate_python_*.sh"
   2. stop checking md5 files
   3. update gcloud before installing beam sdk.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81408)
Time Spent: 13h 10m  (was: 13h)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 13h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3868) Move state and time logic out of the ParDoEvaluator

[Batkhuyag Batsaikhan (JIRA)]

Batkhuyag Batsaikhan created BEAM-3868:
--

 Summary: Move state and time logic out of the ParDoEvaluator
 Key: BEAM-3868
 URL: https://issues.apache.org/jira/browse/BEAM-3868
 Project: Beam
  Issue Type: Improvement
  Components: runner-direct
Reporter: Batkhuyag Batsaikhan
Assignee: Batkhuyag Batsaikhan
 Fix For: 2.4.0


ParDoEvaluator has State and Timer logic that ideally belongs to 
StatefulParDoEvaluator. If possible, move these logic to its appropriate place.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3867) Add support for decompressing and reading tar files

[Chamikara Jayalath (JIRA)]

Chamikara Jayalath created BEAM-3867:


 Summary: Add support for decompressing and reading tar files
 Key: BEAM-3867
 URL: https://issues.apache.org/jira/browse/BEAM-3867
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-core
Reporter: Chamikara Jayalath


See below for a thread regarding this.

https://lists.apache.org/thread.html/eb006101b9d3cb6bf88ec1f8f29e2aaec96ab5760fb7041336fa541a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5165

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4445

[Apache Jenkins Server]

See 


--
[...truncated 294.33 KB...]
+import hamcrest as hc
 from apache_beam.metrics.cells import DistributionData
 from apache_beam.metrics.cells import DistributionResult
 from apache_beam.metrics.execution import MetricKey
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.899312
+++ 
:after
2018-03-16 21:58:17.518353
@@ -18,7 +18,6 @@
 import unittest
 
 import hamcrest as hc
-
 from apache_beam.metrics.cells import DistributionData
 from apache_beam.metrics.cells import DistributionResult
 from apache_beam.metrics.execution import MetricKey
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.879312
+++ 
:after
  2018-03-16 21:58:18.304872
@@ -16,9 +16,8 @@
 #
 import unittest
 
+import apache_beam.runners.dataflow.internal.clients.dataflow as dataflow
 import hamcrest as hc
-
-import apache_beam.runners.dataflow.internal.clients.dataflow as dataflow
 from apache_beam.internal.gcp.json_value import to_json_value
 from apache_beam.runners.dataflow.internal.clients.dataflow import 
message_matchers
 
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-02-09 17:13:39.414380
+++ 
:after
2018-03-16 21:58:18.623723
@@ -21,15 +21,14 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples import wordcount
 from apache_beam.examples import wordcount_fnapi
 from apache_beam.testing.pipeline_verifiers import FileChecksumMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
 from apache_beam.testing.test_utils import delete_files
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class WordCountIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
 2018-01-24 00:22:36.719312
+++ 
:after
  2018-03-16 21:58:19.191871
@@ -21,14 +21,13 @@
 import time
 import unittest
 
-from hamcrest.core.core.allof import all_of
-from nose.plugins.attrib import attr
-
 from apache_beam.examples.cookbook import bigquery_tornadoes
 from apache_beam.io.gcp.tests import utils
 from apache_beam.io.gcp.tests.bigquery_matcher import BigqueryMatcher
 from apache_beam.testing.pipeline_verifiers import PipelineStateMatcher
 from apache_beam.testing.test_pipeline import TestPipeline
+from hamcrest.core.core.allof import all_of
+from nose.plugins.attrib import attr
 
 
 class BigqueryTornadoesIT(unittest.TestCase):
ERROR: 

 Imports are incorrectly sorted.
--- 
:before
   2018-01-24 00:22:36.959311
+++ 
:after
2018-03-16 21:58:19.333058
@@ -20,12 +20,11 @@
 import logging
 import unittest
 
-from hamcrest.core.assert_that import assert_that as hc_assert_that
-from hamcrest.core.base_matcher import BaseMatcher
-
 from 

[jira] [Work logged] (BEAM-3861) Build test infra for end-to-end streaming test in Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3861?focusedWorklogId=81389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81389
 ]

ASF GitHub Bot logged work on BEAM-3861:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:45
Start Date: 16/Mar/18 21:45
Worklog Time Spent: 10m 
  Work Description: markflyhigh commented on issue #4874: [BEAM-3861] 
Improve test infra in Python SDK for streaming end-to-end test
URL: https://github.com/apache/beam/pull/4874#issuecomment-373853545
 
 
   Rebased against master. It's ready to review.
   Please take a look @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81389)
Time Spent: 20m  (was: 10m)

> Build test infra for end-to-end streaming test in Python SDK
> 
>
> Key: BEAM-3861
> URL: https://issues.apache.org/jira/browse/BEAM-3861
> Project: Beam
>  Issue Type: Task
>  Components: testing
>Reporter: Mark Liu
>Assignee: Mark Liu
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3708

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81383=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81383
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:28
Start Date: 16/Mar/18 21:28
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175218594
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
 
 Review comment:
   How are these extra packages installed? Is it a tox feature?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81383)
Time Spent: 5h 20m  (was: 5h 10m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81381
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:28
Start Date: 16/Mar/18 21:28
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175214899
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -113,11 +113,8 @@ def get_version():
 'futures>=3.1.1,<4.0.0',
 ]
 
-REQUIRED_SETUP_PACKAGES = [
-'nose>=1.0',
 
 Review comment:
   Is not this required by setup, because test_suite references it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81381)
Time Spent: 5h 10m  (was: 5h)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81379=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81379
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:28
Start Date: 16/Mar/18 21:28
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175218288
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
+# Pass these environment variables to the test environment.
+passenv = TRAVIS*
+# Don't warn that these commands aren't installed.
+whitelist_externals =
+  find
+  time
+
 [testenv:py27]
-# autocomplete_test depends on nose when invoked directly.
-deps =
-  nose==1.3.7
-  grpcio-tools==1.3.5
-whitelist_externals=find
 commands =
-  python --version
 
 Review comment:
   Printing python and pip version was very useful in debugging issues. Can we 
keep them?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81379)
Time Spent: 5h  (was: 4h 50m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81380
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:28
Start Date: 16/Mar/18 21:28
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175198814
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -64,7 +64,7 @@ build.dependsOn buildPython
 task lint (dependsOn: 'setupTest') {
   doLast {
 exec {
-  commandLine 'tox', '-e', 'lint', '-c', 'tox.ini'
+  commandLine 'tox', '-e', 'py27-lint', '-c', 'tox.ini'
 
 Review comment:
   Should we also run the py3 lint?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81380)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81385
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:28
Start Date: 16/Mar/18 21:28
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175215623
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
+# Pass these environment variables to the test environment.
+passenv = TRAVIS*
 
 Review comment:
   I believe we do not need this anymore.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81385)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81384
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:28
Start Date: 16/Mar/18 21:28
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175215562
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
 
 Review comment:
   Is there a need for this? Would not affect local development?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81384)
Time Spent: 5h 20m  (was: 5h 10m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81382=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81382
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:28
Start Date: 16/Mar/18 21:28
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #4877: 
[BEAM-3738] Enable py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#discussion_r175218495
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -17,142 +17,101 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-# TODO (after BEAM-3671) add lint_py3 back in.
-envlist = py27,py27gcp,py27cython,lint_py2,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
 # Disable all errors and warnings except for the ones related to blank lines.
 # pylint does not check the number of blank lines.
 select = E3
 
+# Shared environment options.
+[testenv]
+# Set [] options for pip install, e.g., pip install apache-beam[test].
+extras = test
+# Always recreate the virtual environment.
+recreate = True
+# Pass these environment variables to the test environment.
+passenv = TRAVIS*
+# Don't warn that these commands aren't installed.
+whitelist_externals =
+  find
+  time
+
 [testenv:py27]
-# autocomplete_test depends on nose when invoked directly.
-deps =
-  nose==1.3.7
-  grpcio-tools==1.3.5
-whitelist_externals=find
 commands =
-  python --version
-  pip --version
-  # Clean up all previous python generated files.
-  - find apache_beam -type f -name '*.pyc' -delete
-  pip install -e .[test]
+  {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
-passenv = TRAVIS*
+  {toxinidir}/run_tox_cleanup.sh
 
-[testenv:py27cython]
+[testenv:py27-cython]
 # cython tests are only expected to work in linux (2.x and 3.x)
 # If we want to add other platforms in the future, it should be:
 # `platform = linux2|darwin|...`
 # See https://docs.python.org/2/library/sys.html#sys.platform for platform 
codes
 platform = linux2
-# autocomplete_test depends on nose when invoked directly.
 deps =
-  nose==1.3.7
-  grpcio-tools==1.3.5
   cython==0.25.2
-whitelist_externals=
-  find
-  time
 commands =
-  python --version
-  pip --version
-  # Clean up all previous python generated files.
-  - find apache_beam -type f -name '*.pyc' -delete
-  # Clean up all previous cython generated files.
-  - find apache_beam -type f -name '*.c' -delete
-  - find apache_beam -type f -name '*.so' -delete
-  - find target/build -type f -name '*.c' -delete
-  - find target/build -type f -name '*.so' -delete
-  time pip install -e .[test]
+  {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
-  # Clean up all cython generated files. Ignore if deletion fails.
-  - find apache_beam -type f -name '*.c' -delete
-  - find apache_beam -type f -name '*.so' -delete
-  - find target/build -type f -name '*.c' -delete
-  - find target/build -type f -name '*.so' -delete
-passenv = TRAVIS*
+  {toxinidir}/run_tox_cleanup.sh
 
-[testenv:py27gcp]
-# autocomplete_test depends on nose when invoked directly.
-deps =
-  nose==1.3.7
-whitelist_externals=find
+[testenv:py27-gcp]
+extras = test,gcp
 commands =
-  python --version
-  pip --version
-  pip install -e .[test,gcp]
-  # Clean up all previous python generated files.
-  - find apache_beam -type f -name '*.pyc' -delete
+  {toxinidir}/run_tox_cleanup.sh
   python apache_beam/examples/complete/autocomplete_test.py
   python setup.py test
-passenv = TRAVIS*
+  {toxinidir}/run_tox_cleanup.sh
 
-[testenv:lint_py2]
-deps=
-  nose==1.3.7
+[testenv:py27-lint]
+deps =
   pycodestyle==2.3.1
   pylint==1.7.2
   future==0.16.0
   isort==4.2.15
   flake8==3.5.0
-whitelist_externals=time
 commands =
   python --version
-  pip --version
-  time pip install -e .[test]
   time {toxinidir}/run_pylint.sh
-passenv = TRAVIS*
 
-[testenv:lint_py3]
-deps=
-  nose==1.3.7
+[testenv:py3-lint]
+deps =
   pycodestyle==2.3.1
   pylint==1.7.2
   future==0.16.0
   isort==4.2.15
   flake8==3.5.0
-whitelist_externals=time
+setenv =
+BEAM_EXPERIMENTAL_PY3=1
 commands =
-  time pip install -e .[test]
+  python --version
   time {toxinidir}/run_mini_py3lint.sh
-passenv = TRAVIS*
-
 
 [testenv:docs]
-deps=
-  nose==1.3.7
-  grpcio-tools==1.3.5
+extras = docs
+deps =
   Sphinx==1.6.5
   sphinx_rtd_theme==0.2.4
-whitelist_externals=time
 commands =
   python --version
   pip --version
   time pip install -e .[test,gcp,docs]
 
 Review comment:
   Should we use `extras` instead of `[test,gcp,docs]`?


This is an automated message from the Apache Git Service.
To respond to the 

[jira] [Work logged] (BEAM-3840) Get Python Mobile-Gaming Running on Core Runners

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3840?focusedWorklogId=81373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81373
 ]

ASF GitHub Bot logged work on BEAM-3840:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:12
Start Date: 16/Mar/18 21:12
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4853: [BEAM-3840] Get 
python mobile-gaming automating on core runners
URL: https://github.com/apache/beam/pull/4853#issuecomment-373846316
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81373)
Time Spent: 13h  (was: 12h 50m)

> Get Python Mobile-Gaming Running on Core Runners
> 
>
> Key: BEAM-3840
> URL: https://issues.apache.org/jira/browse/BEAM-3840
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python
>Affects Versions: 2.5.0
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 13h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated: Enable input subscription in Python streaming wordcount (#4864)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new a7bc72d  Enable input subscription in Python streaming wordcount 
(#4864)
a7bc72d is described below

commit a7bc72d4a3493d59d88923a99762031a1d81b07d
Author: Mark Liu 
AuthorDate: Fri Mar 16 14:10:54 2018 -0700

Enable input subscription in Python streaming wordcount (#4864)

* Enable input subscription in Python streaming wordcount
---
 .../apache_beam/examples/streaming_wordcount.py   | 19 ++-
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/sdks/python/apache_beam/examples/streaming_wordcount.py 
b/sdks/python/apache_beam/examples/streaming_wordcount.py
index 43d19f3..12f7351 100644
--- a/sdks/python/apache_beam/examples/streaming_wordcount.py
+++ b/sdks/python/apache_beam/examples/streaming_wordcount.py
@@ -43,13 +43,18 @@ def run(argv=None):
   """Build and run the pipeline."""
   parser = argparse.ArgumentParser()
   parser.add_argument(
-  '--input_topic', required=True,
-  help=('Input PubSub topic of the form '
-'"projects//topics/".'))
-  parser.add_argument(
   '--output_topic', required=True,
   help=('Output PubSub topic of the form '
 '"projects//topic/".'))
+  group = parser.add_mutually_exclusive_group(required=True)
+  group.add_argument(
+  '--input_topic',
+  help=('Input PubSub topic of the form '
+'"projects//topics/".'))
+  group.add_argument(
+  '--input_subscription',
+  help=('Input PubSub subscription of the form '
+'"projects//subscriptions/."'))
   known_args, pipeline_args = parser.parse_known_args(argv)
   options = PipelineOptions(pipeline_args)
   options.view_as(StandardOptions).streaming = True
@@ -57,7 +62,11 @@ def run(argv=None):
   with beam.Pipeline(options=options) as p:
 
 # Read from PubSub into a PCollection.
-lines = p | beam.io.ReadStringsFromPubSub(known_args.input_topic)
+if known_args.input_subscription:
+  lines = p | beam.io.ReadStringsFromPubSub(
+  subscription=known_args.input_subscription)
+else:
+  lines = p | beam.io.ReadStringsFromPubSub(topic=known_args.input_topic)
 
 # Capitalize the characters in each line.
 def count_ones(word_ones):

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81372=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81372
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:10
Start Date: 16/Mar/18 21:10
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #4877: [BEAM-3738] Enable 
py3 lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#issuecomment-373845974
 
 
   Love the refactor, thanks @udim!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81372)
Time Spent: 4h 50m  (was: 4h 40m)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3738) Enable Py3 linting in Jenkins

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3738?focusedWorklogId=81369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-81369
 ]

ASF GitHub Bot logged work on BEAM-3738:


Author: ASF GitHub Bot
Created on: 16/Mar/18 21:04
Start Date: 16/Mar/18 21:04
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #4877: [BEAM-3738] Enable py3 
lint and cleanup tox.ini.
URL: https://github.com/apache/beam/pull/4877#issuecomment-373844621
 
 
   Missing license was fixed in #4881, retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 81369)
Time Spent: 4h 40m  (was: 4.5h)

> Enable Py3 linting in Jenkins
> -
>
> Key: BEAM-3738
> URL: https://issues.apache.org/jira/browse/BEAM-3738
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, testing
>Reporter: holdenk
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4872 from bjchambers/cherrypick-4829

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch release-2.4.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d224bfbb73149523bbead16cb1006881a3deb69c
Merge: fadab7f 13a3dd1
Author: Robert Bradshaw 
AuthorDate: Fri Mar 16 14:01:27 2018 -0700

Merge pull request #4872 from bjchambers/cherrypick-4829

Cherrypick pr/4829 and pr/4849 [BEAM-3806] direct runner fixes.

 .../beam/runners/direct/WatermarkManager.java  | 38 ---
 .../beam/runners/direct/WatermarkManagerTest.java  | 54 ++
 .../org/apache/beam/sdk/transforms/ParDoTest.java  | 33 +
 3 files changed, 108 insertions(+), 17 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch release-2.4.0 updated (fadab7f -> d224bfb)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch release-2.4.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


from fadab7f  Merge pull request #4847 from charlesccychen/revert-pr-4666-cp
 add 2cdcfbe  [BEAM-3806] Fix direct-runner hang (#4829)
 add 13a3dd1  Annotate ParDoTest.duplicateTimerSetting with UsesTestStream
 new d224bfb  Merge pull request #4872 from bjchambers/cherrypick-4829

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/direct/WatermarkManager.java  | 38 ---
 .../beam/runners/direct/WatermarkManagerTest.java  | 54 ++
 .../org/apache/beam/sdk/transforms/ParDoTest.java  | 33 +
 3 files changed, 108 insertions(+), 17 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Fix missing license header in test.

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 468d924c4ddb11d440de5f14375888c3045113f7
Merge: 984f70d 734ca7d
Author: Lukasz Cwik 
AuthorDate: Fri Mar 16 13:51:21 2018 -0700

Fix missing license header in test.

 sdks/go/pkg/beam/core/graph/coder/time_test.go | 15 +++
 1 file changed, 15 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.