[jira] [Created] (BEAM-5202) register UDF/UDAF with ServiceLoader

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-5202:


 Summary: register UDF/UDAF with ServiceLoader
 Key: BEAM-5202
 URL: https://issues.apache.org/jira/browse/BEAM-5202
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


create a ServiceLoader for UDF/UDAF register, if the autoLoad option is set to 
TRUE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4511) Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4511?focusedWorklogId=137237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137237
 ]

ASF GitHub Bot logged work on BEAM-4511:


Author: ASF GitHub Bot
Created on: 23/Aug/18 03:10
Start Date: 23/Aug/18 03:10
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #6266: 
[BEAM-4511] added py3 tox env for first test
URL: https://github.com/apache/beam/pull/6266#discussion_r212171098
 
 

 ##
 File path: sdks/python/setup.py
 ##
 @@ -99,9 +100,15 @@ def get_version():
   except ImportError:
 cythonize = lambda *args, **kwargs: []
 
+REQUIRED_PACKAGES_PY2 = [
+'avro>=1.8.1,<2.0.0'
+]
+
+REQUIRED_PACKAGES_PY3 = [
+'avro-python3>=1.8.1,<2.0.0'
+]
 
-REQUIRED_PACKAGES = [
-'avro>=1.8.1,<2.0.0',
+REQUIRED_PACKAGES_PY23 = [
 
 Review comment:
   How about the following names:
   ```
   REQUIRED_PACKAGES = ...
   REQUIRED_PACKAGES_PY2_ONLY = ...
   REQUIRED_PACKAGES_PY3_ONLY = ...
   
   REQUIRED_PACKAGES = REQUIRED_PACKAGES + ...
```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137237)
Time Spent: 20m  (was: 10m)

> Create a tox environment that uses Py3 interpreter for pre/post commit test 
> suites, once codebase supports Py3. 
> 
>
> Key: BEAM-4511
> URL: https://issues.apache.org/jira/browse/BEAM-4511
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4511) Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4511?focusedWorklogId=137238=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137238
 ]

ASF GitHub Bot logged work on BEAM-4511:


Author: ASF GitHub Bot
Created on: 23/Aug/18 03:10
Start Date: 23/Aug/18 03:10
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #6266: 
[BEAM-4511] added py3 tox env for first test
URL: https://github.com/apache/beam/pull/6266#discussion_r212170236
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -53,6 +53,16 @@ commands =
   python setup.py nosetests
   {toxinidir}/scripts/run_tox_cleanup.sh
 
+[testenv:py3]
+setenv =
+BEAM_EXPERIMENTAL_PY3=1
+commands =
+  python --version
 
 Review comment:
   How do we tell Jenkins which version of Python interpreter will be used 
here? Is it inferred from the name of the test suite (`testenv:py3`)?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137238)
Time Spent: 20m  (was: 10m)

> Create a tox environment that uses Py3 interpreter for pre/post commit test 
> suites, once codebase supports Py3. 
> 
>
> Key: BEAM-4511
> URL: https://issues.apache.org/jira/browse/BEAM-4511
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4823) Create SNS IO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137231=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137231
 ]

ASF GitHub Bot logged work on BEAM-4823:


Author: ASF GitHub Bot
Created on: 23/Aug/18 01:10
Start Date: 23/Aug/18 01:10
Worklog Time Spent: 10m 
  Work Description: jhalaria commented on issue #6202: [BEAM-4823] - Adds a 
Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#issuecomment-415239022
 
 
   @iemejia - I rebased. Build is still failing & I don't see errors. Do you 
have any recommendations on how to figure this out?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137231)
Time Spent: 4h  (was: 3h 50m)

> Create SNS IO 
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_Nexmark_Dataflow #230

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1323

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #5814

[Apache Jenkins Server]

See 


--
[...truncated 1.04 MB...]
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_simple_type_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_functions_as_regular_generator 
(apache_beam.typehints.typehints_test.TestGeneratorWrapper) ... ok
test_compatibility (apache_beam.typehints.typehints_test.TupleHintTestCase) ... 
ok
test_compatibility_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok
test_getitem_invalid_ellipsis_type_param 

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #874

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PreCommit_Java_Cron #259

[Apache Jenkins Server]

See 


Changes:

[jxlewis] Add Amazon SqsIO

--
[...truncated 17.54 MB...]
INFO: 2018-08-23T00:17:53.094Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign,
 through flatten s13-u58, into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.134Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.179Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.217Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.261Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.309Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
 with random key
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.345Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.389Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.432Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.462Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize 
into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.506Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.551Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
 into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add 
void key/AddKeys/Map
Aug 23, 2018 12:17:54 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-23T00:17:53.574Z: Fusing consumer 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #901

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137186=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137186
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 22:52
Start Date: 22/Aug/18 22:52
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415211052
 
 
   Run Reproducible Env Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137186)
Time Spent: 6h 20m  (was: 6h 10m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4511) Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4511?focusedWorklogId=137187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137187
 ]

ASF GitHub Bot logged work on BEAM-4511:


Author: ASF GitHub Bot
Created on: 22/Aug/18 22:52
Start Date: 22/Aug/18 22:52
Worklog Time Spent: 10m 
  Work Description: Fematich opened a new pull request #6266: [BEAM-4511] 
added py3 tox env for first test
URL: https://github.com/apache/beam/pull/6266
 
 
   This pull request adds the first test for the new py3 tox environment. This 
is is part of a series of PRs with goal to make Apache Beam PY3 compatible. The 
proposal with the outlined approach has been documented 
[here](https://s.apache.org/beam-python-3).
   
   R: @aaltay @tvalentyn @RobbeSneyders @charlesccychen
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137187)
Time Spent: 10m
Remaining Estimate: 0h

> Create a tox environment that uses Py3 interpreter for pre/post commit test 
> suites, once codebase supports Py3. 
> 
>
> Key: BEAM-4511

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137183=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137183
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 22:45
Start Date: 22/Aug/18 22:45
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415207857
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137183)
Time Spent: 6h 10m  (was: 6h)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4511) Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.

[Matthias Feys (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias Feys reassigned BEAM-4511:
---

Assignee: Matthias Feys

> Create a tox environment that uses Py3 interpreter for pre/post commit test 
> suites, once codebase supports Py3. 
> 
>
> Key: BEAM-4511
> URL: https://issues.apache.org/jira/browse/BEAM-4511
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Matthias Feys
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137168=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137168
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 22:04
Start Date: 22/Aug/18 22:04
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415196754
 
 
   Run Reproducible Env Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137168)
Time Spent: 6h  (was: 5h 50m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 6h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137166=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137166
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 22:04
Start Date: 22/Aug/18 22:04
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415196647
 
 
   Run Docker Image Cleanup beam6


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137166)
Time Spent: 5h 50m  (was: 5h 40m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137161
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 21:41
Start Date: 22/Aug/18 21:41
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415190598
 
 
   Run Reproducible Env Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137161)
Time Spent: 5h 40m  (was: 5.5h)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137160
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 21:39
Start Date: 22/Aug/18 21:39
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415189972
 
 
   Run Docker Image Cleanup beam6


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137160)
Time Spent: 5.5h  (was: 5h 20m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137158
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 21:35
Start Date: 22/Aug/18 21:35
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415188867
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137158)
Time Spent: 5h 20m  (was: 5h 10m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #5813

[Apache Jenkins Server]

See 


Changes:

[jxlewis] Add Amazon SqsIO

--
[...truncated 1.05 MB...]
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_simple_type_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_functions_as_regular_generator 
(apache_beam.typehints.typehints_test.TestGeneratorWrapper) ... ok
test_compatibility (apache_beam.typehints.typehints_test.TupleHintTestCase) ... 
ok
test_compatibility_arbitrary_length 
(apache_beam.typehints.typehints_test.TupleHintTestCase) ... ok

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137155=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137155
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 21:32
Start Date: 22/Aug/18 21:32
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415187873
 
 
   Run Reproducible Env Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137155)
Time Spent: 5h 10m  (was: 5h)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137153=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137153
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 21:28
Start Date: 22/Aug/18 21:28
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415186660
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137153)
Time Spent: 5h  (was: 4h 50m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #900

[Apache Jenkins Server]

See 


--
[...truncated 18.99 MB...]
INFO: Adding Create123/Read(CreateSource) as step s10
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
Aug 22, 2018 9:22:32 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
Aug 22, 2018 9:22:32 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0822212227-c042ab05/output/results/staging/
Aug 22, 2018 9:22:32 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71190 bytes, hash Zo_7AKz-qoWVDEJTCNVW-g> to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0822212227-c042ab05/output/results/staging/pipeline-Zo_7AKz-qoWVDEJTCNVW-g.pb

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_OUT
Dataflow SDK version: 2.7.0-SNAPSHOT

org.apache.beam.sdk.transforms.ViewTest > testSingletonSideInput STANDARD_ERROR
Aug 22, 

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137150=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137150
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 21:18
Start Date: 22/Aug/18 21:18
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415183823
 
 
   Run Reproducible Env Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137150)
Time Spent: 4h 50m  (was: 4h 40m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137149
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 21:14
Start Date: 22/Aug/18 21:14
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415182629
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137149)
Time Spent: 4h 40m  (was: 4.5h)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5201) datastoreio discards entity key

[Rib Rdb (JIRA)]

Rib Rdb created BEAM-5201:
-

 Summary: datastoreio discards entity key
 Key: BEAM-5201
 URL: https://issues.apache.org/jira/browse/BEAM-5201
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Rib Rdb
Assignee: Henning Rohde


datastoreio discards the entity key and just returns a collection of decoded 
entities.

This might work for some use cases but often you need to know the key of the 
entity you're operating on, e.g. if you need to update it or store a reference 
to it.

Maybe it should return a PCollection<*datastore.Key, T> instead of just 
PCollection?

Or PCollection using key.Encode(), since the go sdk doesn't seem to 
support pointers yet.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5200) datastoreio doesn't work with entities containing keys

[Rib Rdb (JIRA)]

Rib Rdb created BEAM-5200:
-

 Summary: datastoreio doesn't work with entities containing keys
 Key: BEAM-5200
 URL: https://issues.apache.org/jira/browse/BEAM-5200
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Rib Rdb
Assignee: Henning Rohde


If I trie to map over an entity struct that contains a *datastore.Key field, I 
get the error:

panic: type value main.Foo must be a concrete type

 

It looks like this is because typex.IsConcrete doesn't support pointers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1322

[Apache Jenkins Server]

See 


Changes:

[jxlewis] Add Amazon SqsIO

--
[...truncated 20.38 MB...]
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.438Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.480Z: Elided trivial flatten 
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.526Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.583Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.610Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.646Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.682Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.729Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.788Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.869Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.919Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Aug 22, 2018 9:04:47 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T21:04:44.958Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write
 

[jira] [Created] (BEAM-5199) Better error message for golang non-concrete types

[Rib Rdb (JIRA)]

Rib Rdb created BEAM-5199:
-

 Summary: Better error message for golang non-concrete types
 Key: BEAM-5199
 URL: https://issues.apache.org/jira/browse/BEAM-5199
 Project: Beam
  Issue Type: Improvement
  Components: sdk-go
Reporter: Rib Rdb
Assignee: Henning Rohde


Currently I get an error like:

panic: type value main.Foo must be a concrete type

It would be helpful to tell me the name of the field in Foo that's causing the 
problem.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_Nexmark_Dataflow #229

[Apache Jenkins Server]

See 


Changes:

[jxlewis] Add Amazon SqsIO

--
[...truncated 1.45 MB...]
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 as step s24
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)
 as step s25
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map
 as step s26
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)/GroupByKey
 as step s27
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)/Combine.GroupedValues
 as step s28
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map
 as step s29
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.CreatePCollectionView/ParDo(StreamingPCollectionViewWriter)
 as step s30
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/TempFilePrefixView/Combine.GloballyAsSingletonView/View.CreatePCollectionView/CreateDataflowView
 as step s31
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding savePerfsToBigQuery/BatchLoads/rewindowIntoGlobal/Window.Assign as 
step s32
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding savePerfsToBigQuery/BatchLoads/WriteBundlesToFiles as step s33
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding savePerfsToBigQuery/BatchLoads/GroupByDestination as step s34
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding savePerfsToBigQuery/BatchLoads/WriteGroupedRecords as step s35
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding savePerfsToBigQuery/BatchLoads/FlattenFiles as step s36
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/ReifyResults/View.AsIterable/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)
 as step s37
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/ReifyResults/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map
 as step s38
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/ReifyResults/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)/GroupByKey
 as step s39
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/ReifyResults/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)/Combine.GroupedValues
 as step s40
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
savePerfsToBigQuery/BatchLoads/ReifyResults/View.AsIterable/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map
 as step s41
Aug 22, 2018 9:00:27 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 

[jira] [Work logged] (BEAM-5194) Pipeline options with multi value are not deserialized correctly from map

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5194?focusedWorklogId=137145=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137145
 ]

ASF GitHub Bot logged work on BEAM-5194:


Author: ASF GitHub Bot
Created on: 22/Aug/18 20:55
Start Date: 22/Aug/18 20:55
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6262: [BEAM-5194] Fix 
Pipeline options with multi value deserialization from map
URL: https://github.com/apache/beam/pull/6262#issuecomment-415176588
 
 
   R: @aaltay @pabloem 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137145)
Time Spent: 20m  (was: 10m)

> Pipeline options with multi value are not deserialized correctly from map
> -
>
> Key: BEAM-5194
> URL: https://issues.apache.org/jira/browse/BEAM-5194
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/7c41e0a915083bd3b1fe52c2a417fa38a00e6463/sdks/python/apache_beam/options/pipeline_options.py#L171]
>  
> Multiple options are converted to strings and added to flags which causes 
> wrong deserialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #873

[Apache Jenkins Server]

See 


Changes:

[jxlewis] Add Amazon SqsIO

--
[...truncated 77.42 KB...]
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
:54:
 DeprecationWarning: options is deprecated since First stable release. 
References to .options will not be supported
  print('Found: %s.' % self.build_console_url(pipeline.options))
WARNING:root:Waiting indefinitely for streaming job.
WARNING:root:Waiting indefinitely for streaming job.
WARNING:root:Waiting indefinitely for streaming job.
WARNING:root:Waiting indefinitely for streaming job.
WARNING:root:Waiting indefinitely for streaming job.
WARNING:root:Waiting indefinitely for streaming job.
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... FAIL
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was 

[jira] [Work logged] (BEAM-4823) Create SNS IO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4823?focusedWorklogId=137141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137141
 ]

ASF GitHub Bot logged work on BEAM-4823:


Author: ASF GitHub Bot
Created on: 22/Aug/18 20:21
Start Date: 22/Aug/18 20:21
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #6202: [BEAM-4823] - Adds a 
Sink to write to Amazon's SNS
URL: https://github.com/apache/beam/pull/6202#issuecomment-415166566
 
 
   Oups I somehow screwed this one while merging another PR can you please 
rebase/fix.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137141)
Time Spent: 3h 50m  (was: 3h 40m)

> Create SNS IO 
> --
>
> Key: BEAM-4823
> URL: https://issues.apache.org/jira/browse/BEAM-4823
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Affects Versions: Not applicable
>Reporter: Ankit Jhalaria
>Assignee: Ankit Jhalaria
>Priority: Minor
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4828) Add Amazon SqsIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-4828.

   Resolution: Fixed
Fix Version/s: 2.7.0

> Add Amazon SqsIO
> 
>
> Key: BEAM-4828
> URL: https://issues.apache.org/jira/browse/BEAM-4828
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-aws
>Reporter: John Rudolf Lewis
>Assignee: John Rudolf Lewis
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> Add an SQS source
>  
> For people who would like to follow progress or help out: 
> [https://github.com/JohnRudolfLewis/beam/tree/Add-SqsIO]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4828) Add Amazon SqsIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4828?focusedWorklogId=137137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137137
 ]

ASF GitHub Bot logged work on BEAM-4828:


Author: ASF GitHub Bot
Created on: 22/Aug/18 20:16
Start Date: 22/Aug/18 20:16
Worklog Time Spent: 10m 
  Work Description: iemejia closed pull request #6101: [BEAM-4828] Add 
Amazon SqsIO
URL: https://github.com/apache/beam/pull/6101
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/io/amazon-web-services/build.gradle 
b/sdks/java/io/amazon-web-services/build.gradle
index 973d115dbf2..3ea755966cd 100644
--- a/sdks/java/io/amazon-web-services/build.gradle
+++ b/sdks/java/io/amazon-web-services/build.gradle
@@ -1,3 +1,5 @@
+import groovy.json.JsonOutput
+
 /*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
@@ -29,6 +31,7 @@ dependencies {
   shadow project(path: ":beam-sdks-java-core", configuration: "shadow")
   shadow "com.amazonaws:aws-java-sdk-core:$aws_java_sdk_version"
   shadow "com.amazonaws:aws-java-sdk-s3:$aws_java_sdk_version"
+  shadow "com.amazonaws:aws-java-sdk-sqs:$aws_java_sdk_version"
   shadow library.java.jackson_core
   shadow library.java.jackson_annotations
   shadow library.java.jackson_databind
@@ -42,5 +45,13 @@ dependencies {
   shadowTest library.java.mockito_core
   shadowTest library.java.junit
   shadowTest library.java.slf4j_jdk14
-  testCompile "io.findify:s3mock_2.11:0.2.4"
+  testCompile "io.findify:s3mock_2.12:0.2.4"
+  shadowTest group: 'org.elasticmq', name: 'elasticmq-rest-sqs_2.12', version: 
'0.14.1'
+}
+
+test {
+  systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
+  '--awsRegion=us-west-2',
+  '--awsCredentialsProvider={"@type" : "AWSStaticCredentialsProvider", 
"awsAccessKeyId" : "key_id_value","awsSecretKey" : "secret_value"}'
+  ])
 }
diff --git 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsCheckpointMark.java
 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsCheckpointMark.java
new file mode 100644
index 000..d9a67c066d7
--- /dev/null
+++ 
b/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsCheckpointMark.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.aws.sqs;
+
+import com.amazonaws.services.sqs.model.Message;
+import com.google.common.base.Objects;
+import com.google.common.collect.ImmutableList;
+import java.io.Serializable;
+import java.util.Collection;
+import java.util.List;
+import java.util.Optional;
+import org.apache.beam.sdk.io.UnboundedSource;
+
+class SqsCheckpointMark implements UnboundedSource.CheckpointMark, 
Serializable {
+
+  private final List messagesToDelete;
+  private final transient Optional reader;
+
+  public SqsCheckpointMark(SqsUnboundedReader reader, Collection 
messagesToDelete) {
+this.reader = Optional.of(reader);
+this.messagesToDelete = ImmutableList.copyOf(messagesToDelete);
+;
+  }
+
+  @Override
+  public void finalizeCheckpoint() {
+reader.ifPresent(r -> r.delete(messagesToDelete));
+  }
+
+  List getMessagesToDelete() {
+return messagesToDelete;
+  }
+
+  @Override
+  public boolean equals(Object o) {
+if (this == o) {
+  return true;
+}
+if (o == null || getClass() != o.getClass()) {
+  return false;
+}
+SqsCheckpointMark that = (SqsCheckpointMark) o;
+return Objects.equal(messagesToDelete, that.messagesToDelete);
+  }
+
+  @Override
+  public int hashCode() {
+return Objects.hashCode(messagesToDelete);
+  }
+}
diff --git 
a/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsConfiguration.java
 

[jira] [Work logged] (BEAM-4828) Add Amazon SqsIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4828?focusedWorklogId=137139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137139
 ]

ASF GitHub Bot logged work on BEAM-4828:


Author: ASF GitHub Bot
Created on: 22/Aug/18 20:17
Start Date: 22/Aug/18 20:17
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #6101: [BEAM-4828] Add 
Amazon SqsIO
URL: https://github.com/apache/beam/pull/6101#issuecomment-415165384
 
 
   Fantastic @JohnRudolfLewis great work !
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137139)
Time Spent: 12h 10m  (was: 12h)

> Add Amazon SqsIO
> 
>
> Key: BEAM-4828
> URL: https://issues.apache.org/jira/browse/BEAM-4828
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-aws
>Reporter: John Rudolf Lewis
>Assignee: John Rudolf Lewis
>Priority: Major
>  Time Spent: 12h 10m
>  Remaining Estimate: 0h
>
> Add an SQS source
>  
> For people who would like to follow progress or help out: 
> [https://github.com/JohnRudolfLewis/beam/tree/Add-SqsIO]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6101: [BEAM-4828] Add Amazon SqsIO

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 35f7d0f4045abe5becee8c3c2529eacc251009e2
Merge: 6255ce1 3f9214a
Author: Ismaël Mejía 
AuthorDate: Wed Aug 22 22:16:39 2018 +0200

Merge pull request #6101: [BEAM-4828] Add Amazon SqsIO

 sdks/java/io/amazon-web-services/build.gradle  |  13 +-
 .../beam/sdk/io/aws/sqs/SqsCheckpointMark.java |  65 +++
 .../beam/sdk/io/aws/sqs/SqsConfiguration.java  |  78 
 .../java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java | 216 +
 .../beam/sdk/io/aws/sqs/SqsUnboundedReader.java| 166 
 .../beam/sdk/io/aws/sqs/SqsUnboundedSource.java|  93 +
 .../apache/beam/sdk/io/aws/sqs/package-info.java   |  19 ++
 .../org/apache/beam/sdk/io/aws/sqs/SqsIOTest.java  | 139 +
 8 files changed, 788 insertions(+), 1 deletion(-)

[beam] branch master updated (6255ce1 -> 35f7d0f)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 6255ce1  Merge pull request #6194: [BEAM-3654] Port FilterExamplesTest 
off DoFnTester
 add 3f9214a  Add Amazon SqsIO
 new 35f7d0f  Merge pull request #6101: [BEAM-4828] Add Amazon SqsIO

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/amazon-web-services/build.gradle  |  13 +-
 .../beam/sdk/io/aws/sqs/SqsCheckpointMark.java}|  45 +++--
 .../beam/sdk/io/aws/sqs/SqsConfiguration.java  |  78 
 .../java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java | 216 +
 .../beam/sdk/io/aws/sqs/SqsUnboundedReader.java| 166 
 .../beam/sdk/io/aws/sqs/SqsUnboundedSource.java|  93 +
 .../apache/beam/sdk/io/aws/sqs}/package-info.java  |   5 +-
 .../org/apache/beam/sdk/io/aws/sqs/SqsIOTest.java  | 139 +
 8 files changed, 734 insertions(+), 21 deletions(-)
 copy 
sdks/java/io/{google-cloud-platform/src/test/java/com/google/cloud/spanner/FakeBatchTransactionId.java
 => 
amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsCheckpointMark.java}
 (50%)
 create mode 100644 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsConfiguration.java
 create mode 100644 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java
 create mode 100644 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedReader.java
 create mode 100644 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs/SqsUnboundedSource.java
 copy 
{runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/utils
 => 
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/sqs}/package-info.java
 (89%)
 create mode 100644 
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sqs/SqsIOTest.java

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137136=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137136
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 20:04
Start Date: 22/Aug/18 20:04
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415161612
 
 
   Run Reproducible Env Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137136)
Time Spent: 4.5h  (was: 4h 20m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137123
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 19:01
Start Date: 22/Aug/18 19:01
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415143383
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137123)
Time Spent: 4h 20m  (was: 4h 10m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5198) add functions to add/remove/list tables in BeamSqlEnv

[Xu Mingmin (JIRA)]

Xu Mingmin created BEAM-5198:


 Summary: add functions to add/remove/list tables in BeamSqlEnv
 Key: BEAM-5198
 URL: https://issues.apache.org/jira/browse/BEAM-5198
 Project: Beam
  Issue Type: Task
  Components: dsl-sql
Reporter: Xu Mingmin
Assignee: Xu Mingmin


We use a customized CLI implementation to submit streaming queries with 
BeamSql, the capability to add/remove/list tables in BeamSqlEnv is helpful for 
user-interactive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1321

[Apache Jenkins Server]

See 


--
[...truncated 19.82 MB...]
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.367Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike) into SpannerIO.Write/Write mutations to 
Cloud Spanner/Keys sample as 
view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.409Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.456Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Extract
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.512Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.559Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.605Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Reify into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Partial
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.651Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial into 
SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.694Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.741Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.788Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.812Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-08-22T18:43:27.862Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map
Aug 22, 2018 6:43:32 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process

[jira] [Work logged] (BEAM-4130) Portable Flink runner JobService entry point in a Docker container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4130?focusedWorklogId=137111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137111
 ]

ASF GitHub Bot logged work on BEAM-4130:


Author: ASF GitHub Bot
Created on: 22/Aug/18 18:47
Start Date: 22/Aug/18 18:47
Worklog Time Spent: 10m 
  Work Description: mxm opened a new pull request #6265: [BEAM-4130] Bring 
up Job Server container for Python jobs
URL: https://github.com/apache/beam/pull/6265
 
 
   This starts the Job Server in a container when Python pipelines are run and 
no
   Job Server endpoint has been specified. The Job Server has the docker 
binaries
   and socket mounted from the host to bring up SDK harness containers directly 
on
   the host. This feature is currently only is available for Flink.
   
   Due to host networking not being available on MacOs, we need to explicitly 
map
   ports from the host system to the container. On Linux this is not necessary
   because host networking works as expected. Generally, getting Docker support 
to
   work on MacOs proved to be a bit tricky.
   
   You can test this via:
   
   ```
gradle :beam-sdks-python:portableWordCount
   ```
   
   Some rough edges on MacOS:
   
   - At times, the JobService socket is not available, even though 
`grpc.channel_ready_future(channel).result()` in the portable_runner returns.
   - The port range for the SDK harness is restricted to be between 8100 and 
8200 (configurable) due to the necessary explicit port forwarding
   
   Potentially problematic on Linux:
   
   - The Docker binary might not be statically compiled and depend on shared
 libraries, though this needs to be tested
   
   CC @angoenka 
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137111)
Time Spent: 5h  (was: 4h 50m)

> Portable 

[jira] [Work logged] (BEAM-2930) Flink support for portable side input

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?focusedWorklogId=137110=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137110
 ]

ASF GitHub Bot logged work on BEAM-2930:


Author: ASF GitHub Bot
Created on: 22/Aug/18 18:40
Start Date: 22/Aug/18 18:40
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6208: [BEAM-2930] Side input 
support for Flink portable streaming.
URL: https://github.com/apache/beam/pull/6208#issuecomment-415136625
 
 
   The PR is ready for review. I made the changes to use GBK to materialize the 
side input, which adheres to the windowing strategy and is equivalent to what 
the old runner does. @lukecwik @angoenka 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137110)
Time Spent: 4h 10m  (was: 4h)

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PreCommit_Java_Cron #258

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-5148) Implement MongoDB IO for Python SDK

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589202#comment-16589202
 ] 

Chamikara Jayalath commented on BEAM-5148:
--

Thanks. Looks good in general. Could you send this for review in the form of a 
Beam pull request so that I can provide detailed comments. 

Also, I tried to assign this Jira to you but seems like you currently don't 
have the Beam contributor role assigned to your Jira account. Could you send a 
request through dev list or Slack for this ? (a PMC member can add you). 

> Implement MongoDB IO for Python SDK
> ---
>
> Key: BEAM-5148
> URL: https://issues.apache.org/jira/browse/BEAM-5148
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Affects Versions: 3.0.0
>Reporter: Pascal Gula
>Assignee: Chamikara Jayalath
>Priority: Major
> Fix For: Not applicable
>
>
> Currently Java SDK has MongoDB support but Python SDK does not. With current 
> portability efforts other runners may soon be able to use Python SDK. Having 
> mongoDB support will allow these runners to execute large scale jobs using it.
> Since we need this IO components @ Peat, we started working on a PyPi package 
> available at this repository: [https://github.com/PEAT-AI/beam-extended]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1320

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137085=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137085
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:53
Start Date: 22/Aug/18 17:53
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415121154
 
 
   Run Docker Image Cleanup beam6


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137085)
Time Spent: 4h 10m  (was: 4h)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137072
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:41
Start Date: 22/Aug/18 17:41
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415117035
 
 
   Run Reproducible Env Test


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137072)
Time Spent: 4h  (was: 3h 50m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3954) Get Jenkins agents dockerized

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3954?focusedWorklogId=137070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137070
 ]

ASF GitHub Bot logged work on BEAM-3954:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:36
Start Date: 22/Aug/18 17:36
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #6254: Do Not Merge 
[BEAM-3954] Reproducible environment for Beam Jenkins tests
URL: https://github.com/apache/beam/pull/6254#issuecomment-415115593
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137070)
Time Spent: 3h 50m  (was: 3h 40m)

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4828) Add Amazon SqsIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4828?focusedWorklogId=137069=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137069
 ]

ASF GitHub Bot logged work on BEAM-4828:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:34
Start Date: 22/Aug/18 17:34
Worklog Time Spent: 10m 
  Work Description: JohnRudolfLewis commented on issue #6101: [BEAM-4828] 
Add Amazon SqsIO
URL: https://github.com/apache/beam/pull/6101#issuecomment-415115058
 
 
   @iemejia squashed and ready to merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137069)
Time Spent: 11h 50m  (was: 11h 40m)

> Add Amazon SqsIO
> 
>
> Key: BEAM-4828
> URL: https://issues.apache.org/jira/browse/BEAM-4828
> Project: Beam
>  Issue Type: New Feature
>  Components: io-java-aws
>Reporter: John Rudolf Lewis
>Assignee: John Rudolf Lewis
>Priority: Major
>  Time Spent: 11h 50m
>  Remaining Estimate: 0h
>
> Add an SQS source
>  
> For people who would like to follow progress or help out: 
> [https://github.com/JohnRudolfLewis/beam/tree/Add-SqsIO]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5197) Flaky test: UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest

[Thomas Weise (JIRA)]

Thomas Weise created BEAM-5197:
--

 Summary: Flaky test: 
UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest
 Key: BEAM-5197
 URL: https://issues.apache.org/jira/browse/BEAM-5197
 Project: Beam
  Issue Type: Improvement
  Components: runner-flink
Reporter: Thomas Weise
Assignee: Aljoscha Krettek


{code:java}
java.lang.NullPointerException
at 
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:52)
at 
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService$1.compare(TestProcessingTimeService.java:49)
at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)
at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)
at java.util.PriorityQueue.offer(PriorityQueue.java:345)
at 
org.apache.flink.streaming.runtime.tasks.TestProcessingTimeService.registerTimer(TestProcessingTimeService.java:93)
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.setNextWatermarkTimer(UnboundedSourceWrapper.java:452)
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:225)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at 
org.apache.beam.runners.flink.streaming.UnboundedSourceWrapperTest$ParameterizedUnboundedSourceWrapperTest.testValueEmission(UnboundedSourceWrapperTest.java:153)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #1322

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #1308

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-5156) Apache Beam on dataflow runner can't find Tensorflow for workers

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589137#comment-16589137
 ] 

Ankur Goenka edited comment on BEAM-5156 at 8/22/18 5:17 PM:
-

Check the thread 
[https://lists.apache.org/thread.html/0cbf73d696e0d3a5bb8e93618ac9d6bb81daecf2c9c8e11ee220c8ae@%3Cdev.beam.apache.org%3E]

 I am suspecting the same issue here.

Please try using --experiments worker_threads=100 after fixing the setup.py, 
please use 


was (Author: angoenka):
Check the thread 
[https://lists.apache.org/thread.html/0cbf73d696e0d3a5bb8e93618ac9d6bb81daecf2c9c8e11ee220c8ae@%3Cdev.beam.apache.org%3E]

 

Please try using --experiments worker_threads=100 after fixing the setup.py, 
please use 

> Apache Beam on dataflow runner can't find Tensorflow for workers
> 
>
> Key: BEAM-5156
> URL: https://issues.apache.org/jira/browse/BEAM-5156
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
> Environment: google cloud compute instance running linux
>Reporter: Thomas Johns
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0, 2.6.0
>
>
> Adding serialized tensorflow model to apache beam pipeline with python sdk 
> but it can not find any version of tensorflow when applied to dataflow runner 
> although it is not a problem locally. Tried various versions of tensorflow 
> from 1.6 to 1.10. I thought it might be a conflicting package some where so I 
> removed all other packages and tried to just install tensorflow and same 
> problem.
> Could not find a version that satisfies the requirement tensorflow==1.6.0 
> (from -r reqtest.txt (line 59)) (from versions: )No matching distribution 
> found for tensorflow==1.6.0 (from -r reqtest.txt (line 59))



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5002) Beam Dependency Update Request: org.slf4j

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5002?focusedWorklogId=137059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137059
 ]

ASF GitHub Bot logged work on BEAM-5002:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:16
Start Date: 22/Aug/18 17:16
Worklog Time Spent: 10m 
  Work Description: yifanzou closed pull request #6203: Do Not Merge 
[BEAM-5002] upgrade slf4j
URL: https://github.com/apache/beam/pull/6203
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index bc22bad8abd..15dd818d57b 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -389,10 +389,10 @@ class BeamModulePlugin implements Plugin {
 proto_google_cloud_pubsub_v1: 
"com.google.api.grpc:proto-google-cloud-pubsub-v1:$generated_grpc_ga_version",
 proto_google_cloud_spanner_admin_database_v1: 
"com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:$generated_grpc_beta_version",
 proto_google_common_protos  : 
"com.google.api.grpc:proto-google-common-protos:$proto_google_common_protos",
-slf4j_api   : 
"org.slf4j:slf4j-api:1.7.25",
-slf4j_simple: 
"org.slf4j:slf4j-simple:1.7.25",
-slf4j_jdk14 : 
"org.slf4j:slf4j-jdk14:1.7.25",
-slf4j_log4j12   : 
"org.slf4j:slf4j-log4j12:1.7.25",
+slf4j_api   : 
"org.slf4j:slf4j-api:1.8.0-beta2",
+slf4j_simple: 
"org.slf4j:slf4j-simple:1.8.0-beta2",
+slf4j_jdk14 : 
"org.slf4j:slf4j-jdk14:1.8.0-beta2",
+slf4j_log4j12   : 
"org.slf4j:slf4j-log4j12:1.8.0-beta2",
 snappy_java : 
"org.xerial.snappy:snappy-java:1.1.4",
 spark_core  : 
"org.apache.spark:spark-core_2.11:$spark_version",
 spark_network_common: 
"org.apache.spark:spark-network-common_2.11:$spark_version",


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137059)
Time Spent: 40m  (was: 0.5h)

> Beam Dependency Update Request: org.slf4j
> -
>
> Key: BEAM-5002
> URL: https://issues.apache.org/jira/browse/BEAM-5002
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> 2018-07-25 20:33:53.630864
> Please review and upgrade the org.slf4j to the latest version None 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5156) Apache Beam on dataflow runner can't find Tensorflow for workers

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589137#comment-16589137
 ] 

Ankur Goenka commented on BEAM-5156:


Check the thread 
[https://lists.apache.org/thread.html/0cbf73d696e0d3a5bb8e93618ac9d6bb81daecf2c9c8e11ee220c8ae@%3Cdev.beam.apache.org%3E]

 

Please try using --experiments worker_threads=100 after fixing the setup.py, 
please use 

> Apache Beam on dataflow runner can't find Tensorflow for workers
> 
>
> Key: BEAM-5156
> URL: https://issues.apache.org/jira/browse/BEAM-5156
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
> Environment: google cloud compute instance running linux
>Reporter: Thomas Johns
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0, 2.6.0
>
>
> Adding serialized tensorflow model to apache beam pipeline with python sdk 
> but it can not find any version of tensorflow when applied to dataflow runner 
> although it is not a problem locally. Tried various versions of tensorflow 
> from 1.6 to 1.10. I thought it might be a conflicting package some where so I 
> removed all other packages and tried to just install tensorflow and same 
> problem.
> Could not find a version that satisfies the requirement tensorflow==1.6.0 
> (from -r reqtest.txt (line 59)) (from versions: )No matching distribution 
> found for tensorflow==1.6.0 (from -r reqtest.txt (line 59))



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3654) Port FilterExamplesTest off DoFnTester

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3654?focusedWorklogId=137049=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137049
 ]

ASF GitHub Bot logged work on BEAM-3654:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:06
Start Date: 22/Aug/18 17:06
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev closed pull request #6194: [BEAM-3654] 
Port FilterExamplesTest off DoFnTester
URL: https://github.com/apache/beam/pull/6194
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/examples/java/src/test/java/org/apache/beam/examples/cookbook/FilterExamplesTest.java
 
b/examples/java/src/test/java/org/apache/beam/examples/cookbook/FilterExamplesTest.java
index ec98967fb5f..dedc0e31335 100644
--- 
a/examples/java/src/test/java/org/apache/beam/examples/cookbook/FilterExamplesTest.java
+++ 
b/examples/java/src/test/java/org/apache/beam/examples/cookbook/FilterExamplesTest.java
@@ -18,14 +18,17 @@
 package org.apache.beam.examples.cookbook;
 
 import com.google.api.services.bigquery.model.TableRow;
-import java.util.Arrays;
-import java.util.List;
 import 
org.apache.beam.examples.cookbook.FilterExamples.FilterSingleMonthDataFn;
 import org.apache.beam.examples.cookbook.FilterExamples.ProjectionFn;
-import org.apache.beam.sdk.transforms.DoFnTester;
-import org.hamcrest.CoreMatchers;
-import org.junit.Assert;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.ValidatesRunner;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.junit.Rule;
 import org.junit.Test;
+import org.junit.experimental.categories.Category;
 import org.junit.runner.RunWith;
 import org.junit.runners.JUnit4;
 
@@ -54,8 +57,6 @@
   .set("year", "2014")
   .set("mean_temp", "45.3")
   .set("tornado", true);
-  static final TableRow[] ROWS_ARRAY = new TableRow[] {row1, row2, row3};
-  static final List ROWS = Arrays.asList(ROWS_ARRAY);
 
   private static final TableRow outRow1 =
   new TableRow().set("year", 2014).set("month", 6).set("day", 
21).set("mean_temp", 85.3);
@@ -63,22 +64,28 @@
   new TableRow().set("year", 2014).set("month", 7).set("day", 
20).set("mean_temp", 75.4);
   private static final TableRow outRow3 =
   new TableRow().set("year", 2014).set("month", 6).set("day", 
18).set("mean_temp", 45.3);
-  private static final TableRow[] PROJROWS_ARRAY = new TableRow[] {outRow1, 
outRow2, outRow3};
+
+  @Rule public TestPipeline p = TestPipeline.create();
 
   @Test
-  public void testProjectionFn() throws Exception {
-DoFnTester projectionFn = DoFnTester.of(new 
ProjectionFn());
-List results = projectionFn.processBundle(ROWS_ARRAY);
-Assert.assertThat(results, CoreMatchers.hasItem(outRow1));
-Assert.assertThat(results, CoreMatchers.hasItem(outRow2));
-Assert.assertThat(results, CoreMatchers.hasItem(outRow3));
+  @Category(ValidatesRunner.class)
+  public void testProjectionFn() {
+PCollection input = p.apply(Create.of(row1, row2, row3));
+
+PCollection results = input.apply(ParDo.of(new ProjectionFn()));
+
+PAssert.that(results).containsInAnyOrder(outRow1, outRow2, outRow3);
+p.run().waitUntilFinish();
   }
 
   @Test
-  public void testFilterSingleMonthDataFn() throws Exception {
-DoFnTester filterSingleMonthDataFn =
-DoFnTester.of(new FilterSingleMonthDataFn(7));
-List results = 
filterSingleMonthDataFn.processBundle(PROJROWS_ARRAY);
-Assert.assertThat(results, CoreMatchers.hasItem(outRow2));
+  @Category(ValidatesRunner.class)
+  public void testFilterSingleMonthDataFn() {
+PCollection input = p.apply(Create.of(outRow1, outRow2, 
outRow3));
+
+PCollection results = input.apply(ParDo.of(new 
FilterSingleMonthDataFn(7)));
+
+PAssert.that(results).containsInAnyOrder(outRow2);
+p.run().waitUntilFinish();
   }
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137049)
Time Spent: 1.5h  (was: 1h 20m)

> Port FilterExamplesTest off DoFnTester
> --
>
> Key: BEAM-3654
> URL: https://issues.apache.org/jira/browse/BEAM-3654
> Project: Beam
>  Issue Type: Sub-task
> 

[jira] [Work logged] (BEAM-3654) Port FilterExamplesTest off DoFnTester

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3654?focusedWorklogId=137051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137051
 ]

ASF GitHub Bot logged work on BEAM-3654:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:06
Start Date: 22/Aug/18 17:06
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev edited a comment on issue #6194: 
[BEAM-3654] Port FilterExamplesTest off DoFnTester
URL: https://github.com/apache/beam/pull/6194#issuecomment-415105500
 
 
   LGTM, thank you for your contribution @SokolovMS !


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137051)
Time Spent: 1h 40m  (was: 1.5h)

> Port FilterExamplesTest off DoFnTester
> --
>
> Key: BEAM-3654
> URL: https://issues.apache.org/jira/browse/BEAM-3654
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Mikhail
>Priority: Major
>  Labels: beginner, newbie, starter
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6194: [BEAM-3654] Port FilterExamplesTest off DoFnTester

[aromanenko]

This is an automated email from the ASF dual-hosted git repository.

aromanenko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 6255ce1de063bd90700eb1d0c50bd92698a19cd1
Merge: 90ca21e c288be9
Author: Alexey Romanenko <33895511+aromanenko-...@users.noreply.github.com>
AuthorDate: Wed Aug 22 19:05:58 2018 +0200

Merge pull request #6194: [BEAM-3654] Port FilterExamplesTest off DoFnTester

[BEAM-3654] Port FilterExamplesTest off DoFnTester

 .../beam/examples/cookbook/FilterExamplesTest.java | 45 +-
 1 file changed, 26 insertions(+), 19 deletions(-)

[beam] branch master updated (90ca21e -> 6255ce1)

[aromanenko]

This is an automated email from the ASF dual-hosted git repository.

aromanenko pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 90ca21e  Merge pull request #6232: [BEAM-5196] Add MD5 consistency 
check on S3 uploads (writes)
 add c288be9  [BEAM-3654] Port FilterExamplesTest off DoFnTester
 new 6255ce1  Merge pull request #6194: [BEAM-3654] Port FilterExamplesTest 
off DoFnTester

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/examples/cookbook/FilterExamplesTest.java | 45 +-
 1 file changed, 26 insertions(+), 19 deletions(-)

[jira] [Work logged] (BEAM-3654) Port FilterExamplesTest off DoFnTester

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3654?focusedWorklogId=137048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137048
 ]

ASF GitHub Bot logged work on BEAM-3654:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:05
Start Date: 22/Aug/18 17:05
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev commented on issue #6194: [BEAM-3654] 
Port FilterExamplesTest off DoFnTester
URL: https://github.com/apache/beam/pull/6194#issuecomment-415105500
 
 
   LGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137048)
Time Spent: 1h 20m  (was: 1h 10m)

> Port FilterExamplesTest off DoFnTester
> --
>
> Key: BEAM-3654
> URL: https://issues.apache.org/jira/browse/BEAM-3654
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-java
>Reporter: Kenneth Knowles
>Assignee: Mikhail
>Priority: Major
>  Labels: beginner, newbie, starter
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5060) Issues with aws KPL while writing to kinesis using beam

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5060?focusedWorklogId=137041=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137041
 ]

ASF GitHub Bot logged work on BEAM-5060:


Author: ASF GitHub Bot
Created on: 22/Aug/18 17:00
Start Date: 22/Aug/18 17:00
Worklog Time Spent: 10m 
  Work Description: aromanenko-dev opened a new pull request #6264: 
[BEAM-5060] Fix issue with KinesisIO deps
URL: https://github.com/apache/beam/pull/6264
 
 
   Guava should not be shaded for KinesisIO since AWS KPL (used inside 
KinesisIO.Write) has own Guava deps.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [X] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137041)
Time Spent: 10m
Remaining Estimate: 0h

> Issues with aws KPL while writing to kinesis using beam
> ---
>
> Key: BEAM-5060
> URL: https://issues.apache.org/jira/browse/BEAM-5060
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-aws
>Affects Versions: 2.5.0
>Reporter: Varsha Thanooj
>Assignee: Alexey Romanenko
>Priority: Major
> Attachments: pom.xml

[jira] [Assigned] (BEAM-2930) Flink support for portable side input

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde reassigned BEAM-2930:
---

Assignee: Thomas Weise

> Flink support for portable side input
> -
>
> Key: BEAM-2930
> URL: https://issues.apache.org/jira/browse/BEAM-2930
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4683) Integrate support for timers using the portability APIs into Spark

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-4683.
---
   Resolution: Later
Fix Version/s: Not applicable

> Integrate support for timers using the portability APIs into Spark
> --
>
> Key: BEAM-4683
> URL: https://issues.apache.org/jira/browse/BEAM-4683
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Luke Cwik
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>
> Consider using the code produced in BEAM-4658 to support timers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5190) Python pipeline options are not picked correctly by PortableRunner

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589097#comment-16589097
 ] 

Thomas Weise commented on BEAM-5190:


Here is the stack trace that I see several operators stuck at:
{code:java}
at 
org.apache.beam.sdk.fn.data.CompletableFutureInboundDataClient.awaitCompletion(CompletableFutureInboundDataClient.java:49)

at 
org.apache.beam.sdk.fn.data.BeamFnDataInboundObserver.awaitCompletion(BeamFnDataInboundObserver.java:90)

at 
org.apache.beam.runners.fnexecution.control.SdkHarnessClient$ActiveBundle.close(SdkHarnessClient.java:293)

at 
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator.$closeResource(ExecutableStageDoFnOperator.java:157){code}

> Python pipeline options are not picked correctly by PortableRunner
> --
>
> Key: BEAM-5190
> URL: https://issues.apache.org/jira/browse/BEAM-5190
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>
> Python SDK worker is deserializing the pipeline options to dictionary instead 
> of PipelineOptions
> Sample log
> [grpc-default-executor-2] INFO sdk_worker_main.main - Python sdk harness 
> started with pipeline_options: \{u'beam:option:flink_master:v1': u'[auto]', 
> u'beam:option:streaming:v1': False, u'beam:option:experiments:v1': 
> [u'beam_fn_api', u'worker_threads=50'], u'beam:option:dry_run:v1': False, 
> u'beam:option:runner:v1': None, u'beam:option:profile_memory:v1': False, 
> u'beam:option:runtime_type_check:v1': False, u'beam:option:region:v1': 
> u'us-central1', u'beam:option:options_id:v1': 1, u'beam:option:no_auth:v1': 
> False, u'beam:option:dataflow_endpoint:v1': 
> u'https://dataflow.googleapis.com', u'beam:option:sdk_location:v1': 
> u'/usr/local/google/home/goenka/d/work/beam/beam/sdks/python/dist/apache-beam-2.7.0.dev0.tar.gz',
>  u'beam:option:direct_runner_use_stacked_bundle:v1': True, 
> u'beam:option:save_main_session:v1': True, 
> u'beam:option:type_check_strictness:v1': u'DEFAULT_TO_ANY', 
> u'beam:option:profile_cpu:v1': False, u'beam:option:job_endpoint:v1': 
> u'localhost:8099', u'beam:option:job_name:v1': 
> u'BeamApp-goenka-0822071645-48ae1008', u'beam:option:temp_location:v1': 
> u'gs://clouddfe-goenka/tmp/', u'beam:option:app_name:v1': None, 
> u'beam:option:project:v1': u'google.com:clouddfe', 
> u'beam:option:pipeline_type_check:v1': True, 
> u'beam:option:staging_location:v1': u'gs://clouddfe-goenka/tmp/staging'} 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5062) Add ability to configure S3ClientOptions

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5062?focusedWorklogId=137003=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-137003
 ]

ASF GitHub Bot logged work on BEAM-5062:


Author: ASF GitHub Bot
Created on: 22/Aug/18 16:12
Start Date: 22/Aug/18 16:12
Worklog Time Spent: 10m 
  Work Description: yuppie-flu commented on issue #6122: [BEAM-5062] Add 
ability to provide custom S3ClientOptions
URL: https://github.com/apache/beam/pull/6122#issuecomment-415088991
 
 
   Hey @iemejia 
   I thought that a cleaner way would be to expose `S3ClientOptions` to client, 
not the whole `AmazonS3ClientBuilder`. You can check in this updated PR. 
However, it's impossible to pass `S3ClientOptions` to the builder as a whole 
object, so the PR includes "hack" method to pass them one-by-one.
   I created an improvement issues for `aws-skd-java` library: 
https://github.com/aws/aws-sdk-java/issues/1736
   When it's done, the hack can be removed.
   What do you think?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 137003)
Time Spent: 2h  (was: 1h 50m)

> Add ability to configure S3ClientOptions
> 
>
> Key: BEAM-5062
> URL: https://issues.apache.org/jira/browse/BEAM-5062
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Kirill Kozlov
>Assignee: Kirill Kozlov
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> It would be very useful to have an ability to configure 
> [S3ClientOptions|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/S3ClientOptions.html]
>  for Apache Beam jobs.
> For example, there are some implementations of S3, that does not support 
> virtual-hosted-style URLs for buckets, only path-style. Currently it's 
> impossible to enable path style access for amazon s3 client, which is used by 
> an apache-beam job.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3672) FlinkRunner: Implement an Artifact service using the Flink DistributedCache

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589062#comment-16589062
 ] 

Henning Rohde commented on BEAM-3672:
-

Moving this to a separate task, if it's optional and not imminent.

> FlinkRunner: Implement an Artifact service using the Flink DistributedCache
> ---
>
> Key: BEAM-3672
> URL: https://issues.apache.org/jira/browse/BEAM-3672
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> We need to have a DistributedCache-based artifact service to ship with the 
> portable Flink runner. The DistributedCache is a perfect fit for Flink 
> because it comes for free and is the mechanism that Flink already uses to 
> distribute its own artifacts.
>  
> The final artifact service implementation should be pluggable, but using the 
> DistributedCache allows the Flink runner to work without additional external 
> dependencies (beyond perhaps Docker).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3672) FlinkRunner: Implement an Artifact service using the Flink DistributedCache

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-3672:

Issue Type: Improvement  (was: Sub-task)
Parent: (was: BEAM-2889)

> FlinkRunner: Implement an Artifact service using the Flink DistributedCache
> ---
>
> Key: BEAM-3672
> URL: https://issues.apache.org/jira/browse/BEAM-3672
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Ben Sidhom
>Assignee: Aljoscha Krettek
>Priority: Minor
>
> We need to have a DistributedCache-based artifact service to ship with the 
> portable Flink runner. The DistributedCache is a perfect fit for Flink 
> because it comes for free and is the mechanism that Flink already uses to 
> distribute its own artifacts.
>  
> The final artifact service implementation should be pluggable, but using the 
> DistributedCache allows the Flink runner to work without additional external 
> dependencies (beyond perhaps Docker).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2951) Containerize job proxy

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-2951.
-
   Resolution: Won't Fix
Fix Version/s: Not applicable

We'll containerize each runner/runner proxy instead.

> Containerize job proxy
> --
>
> Key: BEAM-2951
> URL: https://issues.apache.org/jira/browse/BEAM-2951
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-core
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2886) Portable pipeline submission proxy

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589051#comment-16589051
 ] 

Henning Rohde commented on BEAM-2886:
-

I think this issue should be resolved, given we are going with a library (= 
what this issue was intended to capture) and separate runner-specific job 
servers. [~axelmagn] feel free to reopen or clarify, if that not correct.

> Portable pipeline submission proxy
> --
>
> Key: BEAM-2886
> URL: https://issues.apache.org/jira/browse/BEAM-2886
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Henning Rohde
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
> Fix For: 2.7.0
>
>
> Add an local extensible proxy for portable job submission as per 
> https://s.apache.org/beam-job-api as well as a runner/shim for each SDK 
> targeting it. This proxy is a stepping stone towards the longer-term goal of 
> native runner support of these APIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2886) Portable pipeline submission proxy

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-2886.
-
   Resolution: Fixed
Fix Version/s: 2.7.0

> Portable pipeline submission proxy
> --
>
> Key: BEAM-2886
> URL: https://issues.apache.org/jira/browse/BEAM-2886
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Henning Rohde
>Assignee: Axel Magnuson
>Priority: Major
>  Labels: portability
> Fix For: 2.7.0
>
>
> Add an local extensible proxy for portable job submission as per 
> https://s.apache.org/beam-job-api as well as a runner/shim for each SDK 
> targeting it. This proxy is a stepping stone towards the longer-term goal of 
> native runner support of these APIs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2885) Local Dataflow proxy for portable job submission

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-2885:

Summary: Local Dataflow proxy for portable job submission  (was: Support 
job+artifact APIs locally)

> Local Dataflow proxy for portable job submission
> 
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2885) Local Dataflow proxy for portable job submission

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde updated BEAM-2885:

Issue Type: Improvement  (was: Sub-task)
Parent: (was: BEAM-2884)

> Local Dataflow proxy for portable job submission
> 
>
> Key: BEAM-2885
> URL: https://issues.apache.org/jira/browse/BEAM-2885
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-dataflow
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> As per https://s.apache.org/beam-job-api, use local support for 
> submission-side. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2932) Spark support for portable side input

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2932.
---
   Resolution: Later
Fix Version/s: Not applicable

> Spark support for portable side input
> -
>
> Key: BEAM-2932
> URL: https://issues.apache.org/jira/browse/BEAM-2932
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2920) Spark support for portable user state

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2920.
---
   Resolution: Later
Fix Version/s: Not applicable

> Spark support for portable user state
> -
>
> Key: BEAM-2920
> URL: https://issues.apache.org/jira/browse/BEAM-2920
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2909) Spark supports portable progress reporting

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2909.
---
   Resolution: Later
Fix Version/s: Not applicable

> Spark supports portable progress reporting
> --
>
> Key: BEAM-2909
> URL: https://issues.apache.org/jira/browse/BEAM-2909
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2921) Gearpump support for portable user state

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2921.
---
   Resolution: Later
Fix Version/s: Not applicable

> Gearpump support for portable user state
> 
>
> Key: BEAM-2921
> URL: https://issues.apache.org/jira/browse/BEAM-2921
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-gearpump
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2910) Gearpump supports portable progress reporting

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2910.
---
   Resolution: Later
Fix Version/s: Not applicable

> Gearpump supports portable progress reporting
> -
>
> Key: BEAM-2910
> URL: https://issues.apache.org/jira/browse/BEAM-2910
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-gearpump
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2933) Gearpump support for portable side input

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2933.
---
   Resolution: Later
Fix Version/s: Not applicable

> Gearpump support for portable side input
> 
>
> Key: BEAM-2933
> URL: https://issues.apache.org/jira/browse/BEAM-2933
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-gearpump
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2931) Apex support for portable side input

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2931.
---
   Resolution: Later
Fix Version/s: Not applicable

> Apex support for portable side input
> 
>
> Key: BEAM-2931
> URL: https://issues.apache.org/jira/browse/BEAM-2931
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-apex
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2908) Apex supports portable progress reporting

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2908.
---
   Resolution: Later
Fix Version/s: Not applicable

> Apex supports portable progress reporting
> -
>
> Key: BEAM-2908
> URL: https://issues.apache.org/jira/browse/BEAM-2908
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-apex
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2919) Apex support for portable user state

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2919.
---
   Resolution: Later
Fix Version/s: Not applicable

> Apex support for portable user state
> 
>
> Key: BEAM-2919
> URL: https://issues.apache.org/jira/browse/BEAM-2919
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-apex
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2936) JStorm support for portable side input

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2936.
---
   Resolution: Later
Fix Version/s: Not applicable

> JStorm support for portable side input
> --
>
> Key: BEAM-2936
> URL: https://issues.apache.org/jira/browse/BEAM-2936
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-jstorm
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2924) JStorm support for portable user state

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2924.
---
   Resolution: Later
Fix Version/s: Not applicable

> JStorm support for portable user state
> --
>
> Key: BEAM-2924
> URL: https://issues.apache.org/jira/browse/BEAM-2924
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-jstorm
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2913) JStorm supports portable progress reporting

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2913.
---
   Resolution: Later
Fix Version/s: Not applicable

> JStorm supports portable progress reporting
> ---
>
> Key: BEAM-2913
> URL: https://issues.apache.org/jira/browse/BEAM-2913
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-jstorm
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5196) Add MD5 consistency check on S3 uploads (writes)

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5196?focusedWorklogId=136993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136993
 ]

ASF GitHub Bot logged work on BEAM-5196:


Author: ASF GitHub Bot
Created on: 22/Aug/18 15:35
Start Date: 22/Aug/18 15:35
Worklog Time Spent: 10m 
  Work Description: iemejia commented on issue #6232: [BEAM-5196]  Add MD5 
consistency check on S3 uploads (writes)
URL: https://github.com/apache/beam/pull/6232#issuecomment-415076310
 
 
   @toelen Totally agree, we have IT for most of the Google IOs, for AWS we 
have only Kinesis but it is not part of the Beam CI yet, to put this in place 
we will need some company donating a bit to pay the AWS bills (we considered a 
docker with localstack at some moment but well it be still not a 'real' test).
   
   In the meantime we need to prepare code for this (there is ongoing work on 
new AWS connectors e.g. SNS, SQS, Kinesis so this will matter).  All 
contributions in this front are welcome :) I mentioned @lgajowy in the PR 
because he lead the work on file-based tests (gs://, hdfs://) so if you are 
interested in contributing in this area maybe it is worth to take a look at 
what he did, in principle it should be straightforward to add s3. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 136993)
Time Spent: 40m  (was: 0.5h)

> Add MD5 consistency check on S3 uploads (writes)
> 
>
> Key: BEAM-5196
> URL: https://issues.apache.org/jira/browse/BEAM-5196
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-aws
>Reporter: Ismaël Mejía
>Assignee: Leen Toelen
>Priority: Minor
> Fix For: 2.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2923) MapReduce support for portable user state

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2923.
---
   Resolution: Later
Fix Version/s: Not applicable

> MapReduce support for portable user state
> -
>
> Key: BEAM-2923
> URL: https://issues.apache.org/jira/browse/BEAM-2923
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-mapreduce
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2912) MapReduce supports portable progress reporting

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2912.
---
   Resolution: Later
Fix Version/s: Not applicable

> MapReduce supports portable progress reporting
> --
>
> Key: BEAM-2912
> URL: https://issues.apache.org/jira/browse/BEAM-2912
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-mapreduce
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2935) MapReduce support for portable side input

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2935.
---
   Resolution: Later
Fix Version/s: Not applicable

> MapReduce support for portable side input
> -
>
> Key: BEAM-2935
> URL: https://issues.apache.org/jira/browse/BEAM-2935
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-mapreduce
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2922) Tez support for portable user state

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2922.
---
   Resolution: Later
Fix Version/s: Not applicable

> Tez support for portable user state
> ---
>
> Key: BEAM-2922
> URL: https://issues.apache.org/jira/browse/BEAM-2922
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-tez
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2911) Tez supports portable progress reporting

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2911.
---
   Resolution: Later
Fix Version/s: Not applicable

> Tez supports portable progress reporting
> 
>
> Key: BEAM-2911
> URL: https://issues.apache.org/jira/browse/BEAM-2911
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-tez
>Reporter: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-2934) Tez support for portable side input

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde closed BEAM-2934.
---
   Resolution: Later
Fix Version/s: Not applicable

> Tez support for portable side input
> ---
>
> Key: BEAM-2934
> URL: https://issues.apache.org/jira/browse/BEAM-2934
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-tez
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
> Fix For: Not applicable
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?focusedWorklogId=136992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136992
 ]

ASF GitHub Bot logged work on BEAM-5184:


Author: ASF GitHub Bot
Created on: 22/Aug/18 15:31
Start Date: 22/Aug/18 15:31
Worklog Time Spent: 10m 
  Work Description: lukecwik edited a comment on issue #6257: [BEAM-5184] 
Multimap side inputs with duplicate keys and values are being lost
URL: https://github.com/apache/beam/pull/6257#issuecomment-415074170
 
 
   I'll try to look at the failure in Dataflow, it's side input handling is 
different.
   
   The Jenkins logs for 
`org.apache.beam.sdk.transforms.ViewTest.testMultimapSideInputWithNonDeterministicKeyCoder`
 fail with
   
   ```
   Expected: iterable over [, , , 
, ] in any order
but: No item matches:  in [, , , ]
   ```
   
   The error message is implying a comparison issue since all the values do 
exist in the actual output 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 136992)
Time Spent: 1h 50m  (was: 1h 40m)

> Multimap side inputs with duplicate keys and values are being lost
> --
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?focusedWorklogId=136991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136991
 ]

ASF GitHub Bot logged work on BEAM-5184:


Author: ASF GitHub Bot
Created on: 22/Aug/18 15:30
Start Date: 22/Aug/18 15:30
Worklog Time Spent: 10m 
  Work Description: lukecwik edited a comment on issue #6257: [BEAM-5184] 
Multimap side inputs with duplicate keys and values are being lost
URL: https://github.com/apache/beam/pull/6257#issuecomment-415074170
 
 
   This seems to break Dataflow, it's side input handling is different.
   
   The Jenkins logs for 
`org.apache.beam.sdk.transforms.ViewTest.testMultimapSideInputWithNonDeterministicKeyCoder`
 fail with
   
   ```
   Expected: iterable over [, , , 
, ] in any order
but: No item matches:  in [, , , ]
   ```
   
   I'll try to take a look as to why this is failing as the error message is 
implying a comparison issue since all the values do exist in the actual output 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 136991)
Time Spent: 1h 40m  (was: 1.5h)

> Multimap side inputs with duplicate keys and values are being lost
> --
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5184) Multimap side inputs with duplicate keys and values are being lost

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5184?focusedWorklogId=136990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-136990
 ]

ASF GitHub Bot logged work on BEAM-5184:


Author: ASF GitHub Bot
Created on: 22/Aug/18 15:29
Start Date: 22/Aug/18 15:29
Worklog Time Spent: 10m 
  Work Description: lukecwik commented on issue #6257: [BEAM-5184] Multimap 
side inputs with duplicate keys and values are being lost
URL: https://github.com/apache/beam/pull/6257#issuecomment-415074170
 
 
   This seems to break Dataflow, it's side input handling is different.
   
   The Jenkins logs for 
org.apache.beam.sdk.transforms.ViewTest.testMultimapSideInputWithNonDeterministicKeyCoder
 fail with
   
   Expected: iterable over [, , , 
, ] in any order
but: No item matches:  in [, , , ]
   
   I'll try to take a look as to why this is failing as the error message is 
implying a comparison issue since all the values do exist in the actual output 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 136990)
Time Spent: 1.5h  (was: 1h 20m)

> Multimap side inputs with duplicate keys and values are being lost
> --
>
> Key: BEAM-5184
> URL: https://issues.apache.org/jira/browse/BEAM-5184
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Luke Cwik
>Assignee: Vaclav Plajt
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Side inputs with duplicate values are being lost due to the usage of a set 
> based multimap.
> [https://github.com/apache/beam/blob/05fb694f265dda0254d7256e938e508fec9ba098/sdks/java/core/src/main/java/org/apache/beam/sdk/values/PCollectionViews.java#L293]
>  
> Originating thread: 
> [https://lists.apache.org/thread.html/48bae7cf71bf6851622cdee0e8bc8619c79c4c2273ed63f288202169@%3Cdev.beam.apache.org%3E]
>  
> Please update the existing tests to exercise this scenario as well: 
> https://github.com/apache/beam/blob/9f23ffc97535e7255245f3852b9d2f0939df5a0a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ViewTest.java#L507



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5107) Support ES 6.x for ElasticsearchIO

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589016#comment-16589016
 ] 

Etienne Chauchot commented on BEAM-5107:


No problem [~dattran.vn01] thanks to contribute to Beam ! 

Please do not close you PR you can still rebase onto master and we will squash 
commits at the end. It was just a comment on the good way to sync with master 
for your future contributions.

> Support ES 6.x for ElasticsearchIO
> --
>
> Key: BEAM-5107
> URL: https://issues.apache.org/jira/browse/BEAM-5107
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-elasticsearch
>Reporter: Dat Tran
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> Elasticsearch has released 6.3.2 but ElasticsearchIO only supports 2x-5.x.
> We should support ES 6.x for ElasticsearchIO.
> https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
> https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5107) Support ES 6.x for ElasticsearchIO

[Etienne Chauchot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589016#comment-16589016
 ] 

Etienne Chauchot edited comment on BEAM-5107 at 8/22/18 3:29 PM:
-

No problem [~dattran.vn01] thanks to contribute to Beam ! 

Please do not close your PR you can still rebase onto master and we will squash 
commits at the end. It was just a comment on the good way to sync with master 
for your future contributions.


was (Author: echauchot):
No problem [~dattran.vn01] thanks to contribute to Beam ! 

Please do not close you PR you can still rebase onto master and we will squash 
commits at the end. It was just a comment on the good way to sync with master 
for your future contributions.

> Support ES 6.x for ElasticsearchIO
> --
>
> Key: BEAM-5107
> URL: https://issues.apache.org/jira/browse/BEAM-5107
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-elasticsearch
>Reporter: Dat Tran
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> Elasticsearch has released 6.3.2 but ElasticsearchIO only supports 2x-5.x.
> We should support ES 6.x for ElasticsearchIO.
> https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
> https://github.com/apache/beam/blob/master/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5105) Move load job poll to finishBundle() method to better parallelize execution

[Nick Orlove (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589002#comment-16589002
 ] 

Nick Orlove commented on BEAM-5105:
---

This is an issue I've experienced

> Move load job poll to finishBundle() method to better parallelize execution
> ---
>
> Key: BEAM-5105
> URL: https://issues.apache.org/jira/browse/BEAM-5105
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Priority: Major
>
> It appears that when we write to BigQuery using WriteTablesDoFn we start a 
> load job and wait for that job to finish.
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L318]
>  
> In cases where we are trying to write a PCollection of tables (for example, 
> when user use dynamic destinations feature) this relies on dynamic work 
> rebalancing to parallellize execution of load jobs. If the runner does not 
> support dynamic work rebalancing or does not execute dynamic work rebalancing 
> from some reason this could have significant performance drawbacks. For 
> example, scheduling times for load jobs will add up.
>  
> A better approach might be to start load jobs at process() method but wait 
> for all load jobs to finish at finishBundle() method. This will parallelize 
> any overheads as well as job execution (assuming more than one job is 
> schedule by BQ.).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5156) Apache Beam on dataflow runner can't find Tensorflow for workers

[Rasmi Elasmar (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16589003#comment-16589003
 ] 

Rasmi Elasmar commented on BEAM-5156:
-

[~cyberush], is the new problem related to this one? Can you share details?

> Apache Beam on dataflow runner can't find Tensorflow for workers
> 
>
> Key: BEAM-5156
> URL: https://issues.apache.org/jira/browse/BEAM-5156
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model
> Environment: google cloud compute instance running linux
>Reporter: Thomas Johns
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0, 2.6.0
>
>
> Adding serialized tensorflow model to apache beam pipeline with python sdk 
> but it can not find any version of tensorflow when applied to dataflow runner 
> although it is not a problem locally. Tried various versions of tensorflow 
> from 1.6 to 1.10. I thought it might be a conflicting package some where so I 
> removed all other packages and tried to just install tensorflow and same 
> problem.
> Could not find a version that satisfies the requirement tensorflow==1.6.0 
> (from -r reqtest.txt (line 59)) (from versions: )No matching distribution 
> found for tensorflow==1.6.0 (from -r reqtest.txt (line 59))



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)