Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1021

[Apache Jenkins Server]

See 


--
[...truncated 19.40 MB...]
INFO: 2018-07-05T00:38:32.678Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.707Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.740Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.770Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.801Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.832Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/ParDo(ToIsmMetadataRecordForKey) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForKeys/Read
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.862Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.892Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/WithKeys/AddKeys/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/ParDo(CollectWindows)
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.922Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Read
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.951Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Write mutations to Spanner into 
SpannerIO.Write/Write mutations to Cloud Spanner/Batch mutations together
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:32.986Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/Combine.GroupedValues/Partial
Jul 05, 2018 12:38:36 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:38:33.016Z: Fusing consumer 

Build failed in Jenkins: beam_PostCommit_Python_Verify #5512

[Apache Jenkins Server]

See 


--
[...truncated 1.12 MB...]
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 

Build failed in Jenkins: beam_PreCommit_Java_Cron #66

[Apache Jenkins Server]

See 

--
[...truncated 17.02 MB...]
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:18:59.837Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:18:59.858Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:18:59.878Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:18:59.901Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:18:59.927Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:18:59.949Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:18:59.972Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:18:59.995Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:19:00.022Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:19:00.043Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:19:00.065Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:19:00.089Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
Jul 05, 2018 12:19:04 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-05T00:19:00.118Z: Fusing consumer 

Build failed in Jenkins: beam_PreCommit_Python_Cron #69

[Apache Jenkins Server]

See 


--
[...truncated 833.37 KB...]
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_key_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 

[jira] [Work logged] (BEAM-4733) Python portable runner to pass pipeline options to job service

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4733?focusedWorklogId=119133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119133
 ]

ASF GitHub Bot logged work on BEAM-4733:


Author: ASF GitHub Bot
Created on: 04/Jul/18 18:53
Start Date: 04/Jul/18 18:53
Worklog Time Spent: 10m 
  Work Description: tweise opened a new pull request #5888: [BEAM-4733] 
Pass pipeline options from Python portable runner to job server.
URL: https://github.com/apache/beam/pull/5888
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119133)
Time Spent: 10m
Remaining Estimate: 0h

> Python portable runner to pass pipeline options to job service
> --
>
> Key: BEAM-4733
> URL: https://issues.apache.org/jira/browse/BEAM-4733
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The portable runner client currently drops the pipeline options. In the 
> prototype, it was implemented here: 
> [https://github.com/bsidhom/beam/commit/ce8a79122b98a7cbcf6fea7db4a5fe31b6e8248a#diff-f2f54a3d6ae6ef6a22e5d52621a133ed]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1020

[Apache Jenkins Server]

See 


--
[...truncated 19.51 MB...]
Jul 04, 2018 6:42:29 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-07-04_11_42_28-7165283263941704956
Jul 04, 2018 6:42:29 PM org.apache.beam.runners.dataflow.TestDataflowRunner 
run
INFO: Running Dataflow job 2018-07-04_11_42_28-7165283263941704956 with 0 
expected assertions.
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:28.423Z: Autoscaling is enabled for job 
2018-07-04_11_42_28-7165283263941704956. The number of workers will be between 
1 and 1000.
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:28.441Z: Autoscaling was automatically enabled for 
job 2018-07-04_11_42_28-7165283263941704956.
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:30.928Z: Checking required Cloud APIs are enabled.
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:31.230Z: Checking permissions granted to controller 
Service Account.
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.057Z: Worker configuration: n1-standard-1 in 
us-central1-f.
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.446Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.609Z: Expanding GroupByKey operations into 
optimizable parts.
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.631Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.812Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.837Z: Elided trivial flatten 
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.864Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.894Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.916Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.948Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:36.980Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jul 04, 2018 6:42:41 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:42:37.005Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

[jira] [Updated] (BEAM-4733) Python portable runner to pass pipeline options to job service

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise updated BEAM-4733:
---
Description: 
The portable runner client currently drops the pipeline options. In the 
prototype, it was implemented here: 
[https://github.com/bsidhom/beam/commit/ce8a79122b98a7cbcf6fea7db4a5fe31b6e8248a#diff-f2f54a3d6ae6ef6a22e5d52621a133ed]

 

> Python portable runner to pass pipeline options to job service
> --
>
> Key: BEAM-4733
> URL: https://issues.apache.org/jira/browse/BEAM-4733
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink, sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>
> The portable runner client currently drops the pipeline options. In the 
> prototype, it was implemented here: 
> [https://github.com/bsidhom/beam/commit/ce8a79122b98a7cbcf6fea7db4a5fe31b6e8248a#diff-f2f54a3d6ae6ef6a22e5d52621a133ed]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4733) Python portable runner to pass pipeline options to job service

[Thomas Weise (JIRA)]

Thomas Weise created BEAM-4733:
--

 Summary: Python portable runner to pass pipeline options to job 
service
 Key: BEAM-4733
 URL: https://issues.apache.org/jira/browse/BEAM-4733
 Project: Beam
  Issue Type: Bug
  Components: runner-flink, sdk-py-harness
Reporter: Thomas Weise
Assignee: Thomas Weise






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #5511

[Apache Jenkins Server]

See 


--
[...truncated 1.12 MB...]
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 

Build failed in Jenkins: beam_PreCommit_Java_Cron #65

[Apache Jenkins Server]

See 

--
[...truncated 16.45 MB...]
INFO: 2018-07-04T18:21:02.104Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.307Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.340Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:349#1d275f544daf228c
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.364Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.391Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.421Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.444Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.474Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.499Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.523Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.554Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.582Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.608Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 04, 2018 6:21:09 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T18:21:02.631Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Jul 

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=119117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119117
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 16:00
Start Date: 04/Jul/18 16:00
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on a change in pull request 
#5715: [BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200166318
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsource.py
 ##
 @@ -49,6 +48,13 @@
 
 __all__ = ['FileBasedSource']
 
+try:
+  unicode   # pylint: disable=unicode-builtin
 
 Review comment:
   Great, thank you!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119117)
Time Spent: 4h 10m  (was: 4h)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4732) Failed to construct instance from factory method SparkRunner#fromOptions

[Piyush Ratan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piyush Ratan updated BEAM-4732:
---
Attachment: Wordcount_lambda.java

>  Failed to construct instance from factory method SparkRunner#fromOptions
> -
>
> Key: BEAM-4732
> URL: https://issues.apache.org/jira/browse/BEAM-4732
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Affects Versions: 2.5.0
> Environment: Eclipse neon 3 in windows 10, Spark 2.1.0
>Reporter: Piyush Ratan
>Assignee: Amit Sela
>Priority: Critical
>  Labels: patch
> Attachments: Wordcount_lambda.java
>
>
> Unable to run Spark as a runner due to following errors- 
> {code:java}
> 3910 [main] INFO org.spark_project.jetty.server.handler.ContextHandler - 
> Started 
> o.s.j.s.ServletContextHandler@18324f97{/metrics/json,null,AVAILABLE,@Spark}
> Exception in thread "main" java.lang.RuntimeException: Failed to construct 
> instance from factory method SparkRunner#fromOptions(interface 
> org.apache.beam.sdk.options.PipelineOptions)
>  at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
>  at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
>  at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
>  at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
>  at com.svayamtech.bean_app.Wordcount_lambda.main(Wordcount_lambda.java:42)
> Caused by: java.lang.reflect.InvocationTargetException
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>  at java.lang.reflect.Method.invoke(Unknown Source)
>  at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
>  ... 4 more
> Caused by: java.lang.NoClassDefFoundError: 
> org/apache/beam/runners/core/construction/PipelineResources
>  at 
> org.apache.beam.runners.spark.SparkRunner.fromOptions(SparkRunner.java:129)
>  ... 9 more
> Caused by: java.lang.ClassNotFoundException: 
> org.apache.beam.runners.core.construction.PipelineResources
>  at java.net.URLClassLoader.findClass(Unknown Source)
>  at java.lang.ClassLoader.loadClass(Unknown Source)
>  at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
>  at java.lang.ClassLoader.loadClass(Unknown Source)
>  ... 10 more
> 4022 [Thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from 
> shutdown hook{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4732) Failed to construct instance from factory method SparkRunner#fromOptions

[Piyush Ratan (JIRA)]

Piyush Ratan created BEAM-4732:
--

 Summary:  Failed to construct instance from factory method 
SparkRunner#fromOptions
 Key: BEAM-4732
 URL: https://issues.apache.org/jira/browse/BEAM-4732
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Affects Versions: 2.5.0
 Environment: Eclipse neon 3 in windows 10, Spark 2.1.0
Reporter: Piyush Ratan
Assignee: Amit Sela


Unable to run Spark as a runner due to following errors- 
{code:java}
3910 [main] INFO org.spark_project.jetty.server.handler.ContextHandler - 
Started 
o.s.j.s.ServletContextHandler@18324f97{/metrics/json,null,AVAILABLE,@Spark}
Exception in thread "main" java.lang.RuntimeException: Failed to construct 
instance from factory method SparkRunner#fromOptions(interface 
org.apache.beam.sdk.options.PipelineOptions)
 at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
 at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
 at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
 at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
 at com.svayamtech.bean_app.Wordcount_lambda.main(Wordcount_lambda.java:42)
Caused by: java.lang.reflect.InvocationTargetException
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
 ... 4 more
Caused by: java.lang.NoClassDefFoundError: 
org/apache/beam/runners/core/construction/PipelineResources
 at org.apache.beam.runners.spark.SparkRunner.fromOptions(SparkRunner.java:129)
 ... 9 more
Caused by: java.lang.ClassNotFoundException: 
org.apache.beam.runners.core.construction.PipelineResources
 at java.net.URLClassLoader.findClass(Unknown Source)
 at java.lang.ClassLoader.loadClass(Unknown Source)
 at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
 at java.lang.ClassLoader.loadClass(Unknown Source)
 ... 10 more
4022 [Thread-1] INFO org.apache.spark.SparkContext - Invoking stop() from 
shutdown hook{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4707) Change fields and table names in nexmark perfkit tables

[Etienne Chauchot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Etienne Chauchot resolved BEAM-4707.

   Resolution: Fixed
Fix Version/s: 2.6.0

> Change fields and table names in nexmark perfkit tables
> ---
>
> Key: BEAM-4707
> URL: https://issues.apache.org/jira/browse/BEAM-4707
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Nexmark BQ tables for perfkit lack timestamp field. Also the table name 
> contains a boolean than shows the mode of execution. It would be better to 
> have batch or streaming label in place of the boolean.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4731) update flink submission webpage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4731?focusedWorklogId=119089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119089
 ]

ASF GitHub Bot logged work on BEAM-4731:


Author: ASF GitHub Bot
Created on: 04/Jul/18 14:15
Start Date: 04/Jul/18 14:15
Worklog Time Spent: 10m 
  Work Description: echauchot opened a new pull request #489: [BEAM-4731] 
update flink submission port because flink 1.5 new standard is port 8081
URL: https://github.com/apache/beam-site/pull/489
 
 
   With v1.5 flink changed their default submission port for the client from 
rpc to rest. Update beam doc accordingly. Submission to rpc port entails an 
eternal waiting of the submission, see 
   
   https://issues.apache.org/jira/browse/FLINK-9255
   
   
   
   Once your pull request has been opened and assigned a number, please edit the
   URL below, replacing `PULL_REQUEST_NUMBER` with the number of your pull 
request.
   
   
http://apache-beam-website-pull-requests.storage.googleapis.com/PULL_REQUEST_NUMBER/index.html
   
   Finally, it will help us expedite review of your Pull Request if you tag
   someone (e.g. @username) to look at it.
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119089)
Time Spent: 10m
Remaining Estimate: 0h

> update flink submission webpage
> ---
>
> Key: BEAM-4731
> URL: https://issues.apache.org/jira/browse/BEAM-4731
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> With v1.5 flink changed their default submission port for the client from rpc 
> to rest. Update beam doc accordingly. Submission to rpc port entails an 
> eternal waiting of the submission, see 
> https://issues.apache.org/jira/browse/FLINK-9255



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4731) update flink submission webpage

[Etienne Chauchot (JIRA)]

Etienne Chauchot created BEAM-4731:
--

 Summary: update flink submission webpage
 Key: BEAM-4731
 URL: https://issues.apache.org/jira/browse/BEAM-4731
 Project: Beam
  Issue Type: Bug
  Components: website
Reporter: Etienne Chauchot
Assignee: Etienne Chauchot


With v1.5 flink changed their default submission port for the client from rpc 
to rest. Update beam doc accordingly. Submission to rpc port entails an eternal 
waiting of the submission, see 

https://issues.apache.org/jira/browse/FLINK-9255



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119076=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119076
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 13:25
Start Date: 04/Jul/18 13:25
Worklog Time Spent: 10m 
  Work Description: superbobry commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200125909
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/executor.py
 ##
 @@ -22,19 +22,24 @@
 import collections
 import itertools
 import logging
-import Queue
+import queue
 import sys
 import threading
 import traceback
+from builtins import object
+from builtins import range
 from weakref import WeakValueDictionary
 
-import six
+from future import standard_library
+from future.utils import raise_
 
 from apache_beam.metrics.execution import MetricsContainer
 from apache_beam.runners.worker import statesampler
 from apache_beam.transforms import sideinputs
 from apache_beam.utils import counters
 
+standard_library.install_aliases()
 
 Review comment:
   How about 
   
   ```
   from future.moves.queue import Queue
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119076)
Time Spent: 7h 20m  (was: 7h 10m)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119075
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 13:24
Start Date: 04/Jul/18 13:24
Worklog Time Spent: 10m 
  Work Description: superbobry commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200125441
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -28,12 +31,13 @@
 import tempfile
 import time
 from datetime import datetime
-from StringIO import StringIO
+from io import BytesIO
 
 import pkg_resources
 from apitools.base.py import encoding
 from apitools.base.py import exceptions
-import six
+
+from future import standard_library
 
 Review comment:
   As discussed in other PR, `io` has been added in Python 2.6, so we don't 
need to `install_aliases` to import it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119075)
Time Spent: 7h 10m  (was: 7h)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119073
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 13:23
Start Date: 04/Jul/18 13:23
Worklog Time Spent: 10m 
  Work Description: superbobry commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200125206
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -28,12 +31,13 @@
 import tempfile
 import time
 from datetime import datetime
-from StringIO import StringIO
+from io import BytesIO
 
 Review comment:
   I think people usually use it in a qualified way.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119073)
Time Spent: 6h 50m  (was: 6h 40m)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 6h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119074=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119074
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 13:23
Start Date: 04/Jul/18 13:23
Worklog Time Spent: 10m 
  Work Description: superbobry commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200125206
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -28,12 +31,13 @@
 import tempfile
 import time
 from datetime import datetime
-from StringIO import StringIO
+from io import BytesIO
 
 Review comment:
   I think people usually use it in a qualified way, so +1 for `io.BytesIO`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119074)
Time Spent: 7h  (was: 6h 50m)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 7h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=119071=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119071
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 12:45
Start Date: 04/Jul/18 12:45
Worklog Time Spent: 10m 
  Work Description: superbobry commented on a change in pull request #5715: 
[BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200115213
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsource.py
 ##
 @@ -49,6 +48,13 @@
 
 __all__ = ['FileBasedSource']
 
+try:
+  unicode   # pylint: disable=unicode-builtin
 
 Review comment:
   This is done in #5869. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119071)
Time Spent: 4h  (was: 3h 50m)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1019

[Apache Jenkins Server]

See 


Changes:

[lukasz.gajowy] [BEAM-4137] Divide separate PipelineOptions for elasticsearch 
tests

[lukasz.gajowy] [BEAM-4137] Remove unused options from IOTestPipelineOptions

--
[...truncated 19.35 MB...]
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:45.358Z: Autoscaling is enabled for job 
2018-07-04_05_40_45-5550568250457718714. The number of workers will be between 
1 and 1000.
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:45.386Z: Autoscaling was automatically enabled for 
job 2018-07-04_05_40_45-5550568250457718714.
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:47.983Z: Checking required Cloud APIs are enabled.
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:48.156Z: Checking permissions granted to controller 
Service Account.
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:51.847Z: Worker configuration: n1-standard-1 in 
us-central1-f.
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.304Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.449Z: Expanding GroupByKey operations into 
optimizable parts.
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.483Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.829Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.859Z: Elided trivial flatten 
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.891Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.918Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.951Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:52.982Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:53.014Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jul 04, 2018 12:40:57 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:40:53.048Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Jul 04, 

Build failed in Jenkins: beam_PostCommit_Python_Verify #5510

[Apache Jenkins Server]

See 


Changes:

[lukasz.gajowy] [BEAM-4137] Divide separate PipelineOptions for elasticsearch 
tests

[lukasz.gajowy] [BEAM-4137] Remove unused options from IOTestPipelineOptions

--
[...truncated 1.11 MB...]
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 

Build failed in Jenkins: beam_PreCommit_Java_Cron #64

[Apache Jenkins Server]

See 


Changes:

[lukasz.gajowy] [BEAM-4137] Divide separate PipelineOptions for elasticsearch 
tests

[lukasz.gajowy] [BEAM-4137] Remove unused options from IOTestPipelineOptions

--
[...truncated 16.54 MB...]
INFO: 2018-07-04T12:17:24.831Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:24.862Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:349#1d275f544daf228c
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:24.894Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:24.917Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:24.944Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:24.972Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:25.005Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:25.036Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:25.059Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:25.084Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:25.107Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:25.137Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 04, 2018 12:17:27 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T12:17:25.162Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Jul 04, 

[jira] [Work logged] (BEAM-1251) Python 3 Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=119054=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119054
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 04/Jul/18 11:37
Start Date: 04/Jul/18 11:37
Worklog Time Spent: 10m 
  Work Description: cclauss opened a new pull request #5887: [BEAM-1251] 
Upgrade from buffer to memoryview (again)
URL: https://github.com/apache/beam/pull/5887
 
 
   This PR is a second shot at #4820 based on [the 
work](https://github.com/andrix/python-snappy/pull/72#) that @pitrou has done 
to allow python-snappy to support Python memoryviews.  This has in turn has 
enabled @martindurant to [release python-snappy 0.5.2 to 
PyPI](https://pypi.org/project/python-snappy/).
   
   This PR recommends the use of that release to upgrade from buffer to 
memoryview in avroio.py because buffer was removed in Python 3 in favor of 
memoryview which is supported in a  Python versions that Beam supports.
   
   Reviews please from @aaltay @holdenk @superbobry 
   
   flake8 testing of https://github.com/apache/beam on Python 3.6.3
   
   $ __flake8 . --count --select=E901,E999,F821,F822,F823 --show-source 
--statistics__
   ```
   ./sdks/python/apache_beam/io/avroio.py:343:34: F821 undefined name 'buffer'
 result = snappy.decompress(buffer(data)[:-4])
^
   1F821 undefined name 'buffer'
   1
   ```
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=119053=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119053
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 11:36
Start Date: 04/Jul/18 11:36
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on a change in pull request 
#5715: [BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200099176
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsource.py
 ##
 @@ -49,6 +48,13 @@
 
 __all__ = ['FileBasedSource']
 
+try:
+  unicode   # pylint: disable=unicode-builtin
 
 Review comment:
   I have created the issue 
[BEAM-4730](https://issues.apache.org/jira/browse/BEAM-4730) to apply this 
change to already ported packages.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119053)
Time Spent: 3h 50m  (was: 3h 40m)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4730) Replace try/except imports related to Py2/3 compatibility with from past.builtins imports

[Robbe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbe reassigned BEAM-4730:
---

Assignee: Robbe  (was: Ahmet Altay)

> Replace try/except imports related to Py2/3 compatibility with from 
> past.builtins imports
> -
>
> Key: BEAM-4730
> URL: https://issues.apache.org/jira/browse/BEAM-4730
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4730) Replace try/except imports related to Py2/3 compatibility with from past.builtins imports

[Robbe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbe reassigned BEAM-4730:
---

Assignee: Ahmet Altay  (was: Robbe)

> Replace try/except imports related to Py2/3 compatibility with from 
> past.builtins imports
> -
>
> Key: BEAM-4730
> URL: https://issues.apache.org/jira/browse/BEAM-4730
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4730) Replace try/except imports related to Py2/3 compatibility with from past.builtins imports

[Robbe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbe reassigned BEAM-4730:
---

Assignee: Robbe  (was: Ahmet Altay)

> Replace try/except imports related to Py2/3 compatibility with from 
> past.builtins imports
> -
>
> Key: BEAM-4730
> URL: https://issues.apache.org/jira/browse/BEAM-4730
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4730) Replace try/except imports related to Py2/3 compatibility with from past.builtins imports

[Robbe (JIRA)]

Robbe created BEAM-4730:
---

 Summary: Replace try/except imports related to Py2/3 compatibility 
with from past.builtins imports
 Key: BEAM-4730
 URL: https://issues.apache.org/jira/browse/BEAM-4730
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Robbe
Assignee: Ahmet Altay






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-4007) Futurize and fix python 2 compatibility for typehints subpackage

[Robbe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robbe resolved BEAM-4007.
-
   Resolution: Fixed
Fix Version/s: Not applicable

> Futurize and fix python 2 compatibility for typehints subpackage
> 
>
> Key: BEAM-4007
> URL: https://issues.apache.org/jira/browse/BEAM-4007
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 4h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=119050=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119050
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 11:25
Start Date: 04/Jul/18 11:25
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on a change in pull request 
#5715: [BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200096594
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsource.py
 ##
 @@ -235,11 +241,11 @@ class _SingleFileSource(iobase.BoundedSource):
 
   def __init__(self, file_based_source, file_name, start_offset, stop_offset,
min_bundle_size=0, splittable=True):
-if not isinstance(start_offset, integer_types):
+if not isinstance(start_offset, (int, long)):
 
 Review comment:
   If you want to replace this with just `int`, we would have to import `int` 
from (future.)`builtins`, which gives problems when used for typechecks. To 
stay consistent with other modules, which do use typechecks, I would advice 
against this.
   
   Instead, we can replace the try/except block with from `past.builtins import 
long` as mentioned in the comment above.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119050)
Time Spent: 3h 40m  (was: 3.5h)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=119048=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119048
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 11:18
Start Date: 04/Jul/18 11:18
Worklog Time Spent: 10m 
  Work Description: RobbeSneyders commented on a change in pull request 
#5715: [BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200094907
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsource.py
 ##
 @@ -49,6 +48,13 @@
 
 __all__ = ['FileBasedSource']
 
+try:
+  unicode   # pylint: disable=unicode-builtin
 
 Review comment:
   The try/except was indeed used to prevent problems with typechecking when 
using `builtins` from the future package.
   However, I tested this again, and it seems that there are no such problems 
using the `past.builtins`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119048)
Time Spent: 3.5h  (was: 3h 20m)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119045=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119045
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 11:06
Start Date: 04/Jul/18 11:06
Worklog Time Spent: 10m 
  Work Description: cclauss commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200090606
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -28,12 +31,13 @@
 import tempfile
 import time
 from datetime import datetime
-from StringIO import StringIO
+from io import BytesIO
 
 import pkg_resources
 from apitools.base.py import encoding
 from apitools.base.py import exceptions
-import six
+
+from future import standard_library
 
 Review comment:
   I do not believe that you can just write __from future import 
standard_library__.  I believe that you must write __from 
future.standard_library import install_aliases ; install_aliases()__ and 
(despite what the linters will tell you) the __install_aliases()__ call must 
happen before you import effected modules.  If I am wrong on this then please 
enlighten me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119045)
Time Spent: 6h 40m  (was: 6.5h)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119044=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119044
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 11:05
Start Date: 04/Jul/18 11:05
Worklog Time Spent: 10m 
  Work Description: cclauss commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200090606
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -28,12 +31,13 @@
 import tempfile
 import time
 from datetime import datetime
-from StringIO import StringIO
+from io import BytesIO
 
 import pkg_resources
 from apitools.base.py import encoding
 from apitools.base.py import exceptions
-import six
+
+from future import standard_library
 
 Review comment:
   I do not believe that you can just write __from future import 
standard_library__.  I believe that you must write __from 
future.standard_library import install_aliases ; install_aliases()__ and 
(despite what the linters will tell you) the __install_aliases()__ call must 
happen before you import effected modules.  If I am wrong on this the please 
enlighten me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119044)
Time Spent: 6.5h  (was: 6h 20m)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119043=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119043
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 11:05
Start Date: 04/Jul/18 11:05
Worklog Time Spent: 10m 
  Work Description: cclauss commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200090606
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -28,12 +31,13 @@
 import tempfile
 import time
 from datetime import datetime
-from StringIO import StringIO
+from io import BytesIO
 
 import pkg_resources
 from apitools.base.py import encoding
 from apitools.base.py import exceptions
-import six
+
+from future import standard_library
 
 Review comment:
   I do not believe that you can just write __from future import 
standard_library__.  I believe that you must write __from 
future.standard_library import install_aliases ; install_aliases()__ and 
(despite what the linters will tell you) that call must happen before you 
import effected modules.  If I am wrong on this the please enlighten me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119043)
Time Spent: 6h 20m  (was: 6h 10m)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=119028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119028
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 04/Jul/18 10:45
Start Date: 04/Jul/18 10:45
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r200085113
 
 

 ##
 File path: sdks/python/apache_beam/pipeline_test.py
 ##
 @@ -520,11 +520,11 @@ def test_dir(self):
 options = Breakfast()
 self.assertEquals(
 set(['from_dictionary', 'get_all_options', 'slices', 'style',
- 'view_as', 'display_data']),
+ 'view_as', 'display_data', 'next']),
 
 Review comment:
   This is because of the import `from builtins import object` in 
apache_beam/transforms/display.py. 
   
   This import adds an alias: 
   `next = __next__` for Python2 and Python3 compatibility.
   
   PipelineOptions (the tested class in this test) inherits from HasDisplayData 
class defined in the display.py module. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119028)
Time Spent: 1.5h  (was: 1h 20m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=119025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119025
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 04/Jul/18 10:42
Start Date: 04/Jul/18 10:42
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r200085113
 
 

 ##
 File path: sdks/python/apache_beam/pipeline_test.py
 ##
 @@ -520,11 +520,11 @@ def test_dir(self):
 options = Breakfast()
 self.assertEquals(
 set(['from_dictionary', 'get_all_options', 'slices', 'style',
- 'view_as', 'display_data']),
+ 'view_as', 'display_data', 'next']),
 
 Review comment:
   This is because of the import `from builtins import object` in 
apache_beam/transforms/display.py. 
   
   This import adds an alias: 
   `next = __next__` to make Python2 and Python3 object classes compatible
   
   PipelineOptions (the tested class in this test) inherits from HasDisplayData 
class defined in the display.py module. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119025)
Time Spent: 1h 20m  (was: 1h 10m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=119020=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119020
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 10:25
Start Date: 04/Jul/18 10:25
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5715: 
[BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200080425
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/gcsio.py
 ##
 @@ -483,6 +489,7 @@ def size(self):
 return self._size
 
   def get_range(self, start, end):
+self._download_stream.seek(0)
 
 Review comment:
   Yes, this is a difference in behavior between cStringIO and BytesIO. In 
BytesIO `truncate` doesn't move the file pointer, see 
[here](https://stackoverflow.com/questions/39109068/why-does-truncating-a-bytesio-mess-it-up)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119020)
Time Spent: 3h 20m  (was: 3h 10m)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=119019=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119019
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 10:24
Start Date: 04/Jul/18 10:24
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5715: 
[BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200080425
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/gcsio.py
 ##
 @@ -483,6 +489,7 @@ def size(self):
 return self._size
 
   def get_range(self, start, end):
+self._download_stream.seek(0)
 
 Review comment:
   Yes, this is a difference in behavior between cStringIO and BytesIO. BytesIO 
doesn't move the file pointer, see 
[here](https://stackoverflow.com/questions/39109068/why-does-truncating-a-bytesio-mess-it-up)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119019)
Time Spent: 3h 10m  (was: 3h)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=119018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119018
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 10:24
Start Date: 04/Jul/18 10:24
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5715: 
[BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200080425
 
 

 ##
 File path: sdks/python/apache_beam/io/gcp/gcsio.py
 ##
 @@ -483,6 +489,7 @@ def size(self):
 return self._size
 
   def get_range(self, start, end):
+self._download_stream.seek(0)
 
 Review comment:
   Yes, this is a difference in behavior between cStringIO and BytesIO. In 
BytesIO doesn't move the file pointer, see 
[here](https://stackoverflow.com/questions/39109068/why-does-truncating-a-bytesio-mess-it-up)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119018)
Time Spent: 3h  (was: 2h 50m)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119017=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119017
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 10:11
Start Date: 04/Jul/18 10:11
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200076690
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/executor.py
 ##
 @@ -22,19 +22,24 @@
 import collections
 import itertools
 import logging
-import Queue
+import queue
 import sys
 import threading
 import traceback
+from builtins import object
+from builtins import range
 from weakref import WeakValueDictionary
 
-import six
+from future import standard_library
+from future.utils import raise_
 
 from apache_beam.metrics.execution import MetricsContainer
 from apache_beam.runners.worker import statesampler
 from apache_beam.transforms import sideinputs
 from apache_beam.utils import counters
 
+standard_library.install_aliases()
 
 Review comment:
   pylint errors on placing `install_aliases()` above the imports. That's the 
reasoning for this location.
   I tested functionality of this placement and this order works as well:
   test script* (Python2):
   Without `install_aliases()`:
   ```
   >>> import sys
   >>> sys.intern
   Traceback (most recent call last):
 File "", line 1, in 
   AttributeError: 'module' object has no attribute 'intern'
   ```
   With `install_aliases()` after import sys
   ```
   >>> import sys
   >>> from future import standard_library
   >>> standard_library.install_aliases()
   >>> sys.intern
   
   ```
   *Based on [documented effect of aliased imports 
](http://python-future.org/standard_library_imports.html#aliased-imports) 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119017)
Time Spent: 6h 10m  (was: 6h)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119016=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119016
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 10:10
Start Date: 04/Jul/18 10:10
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200076690
 
 

 ##
 File path: sdks/python/apache_beam/runners/direct/executor.py
 ##
 @@ -22,19 +22,24 @@
 import collections
 import itertools
 import logging
-import Queue
+import queue
 import sys
 import threading
 import traceback
+from builtins import object
+from builtins import range
 from weakref import WeakValueDictionary
 
-import six
+from future import standard_library
+from future.utils import raise_
 
 from apache_beam.metrics.execution import MetricsContainer
 from apache_beam.runners.worker import statesampler
 from apache_beam.transforms import sideinputs
 from apache_beam.utils import counters
 
+standard_library.install_aliases()
 
 Review comment:
   pylint errors on placing `install_aliases()` above the imports. That's the 
reasoning for this location.
   I tested functionality of this placement and this order works as well:
   test script (Python2):
   Without `install_aliases()`:
   ```
   >>> import sys
   >>> sys.intern
   Traceback (most recent call last):
 File "", line 1, in 
   AttributeError: 'module' object has no attribute 'intern'
   ```
   With `install_aliases()` after import sys
   ```
   >>> import sys
   >>> from future import standard_library
   >>> standard_library.install_aliases()
   >>> sys.intern
   
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119016)
Time Spent: 6h  (was: 5h 50m)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 6h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119011=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119011
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 09:51
Start Date: 04/Jul/18 09:51
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200071447
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/bundle_processor.py
 ##
 @@ -103,7 +107,7 @@ def __init__(self, operation_name, step_name, consumers, 
counter_factory,
 # We must do this manually as we don't have a spec or spec.output_coders.
 self.receivers = [
 operations.ConsumerSet(self.counter_factory, self.step_name, 0,
-   next(consumers.itervalues()),
+   next(itervalues(consumers)),
 
 Review comment:
   I used itervalues based on the feedback of @tvalentyn 
[here](https://github.com/apache/beam/pull/5373#discussion_r195550396):
   
   > As we learned in #5053, Cython can translate iteration through items() 
without an intermediate object given a directive to interpret the code using 
Python3 semantics. However when the code is not cythonized, I would prefer to 
use future.util.iteritems() avoid impact on efficiency for Python 2. It's 
probably ok to use items() in tests, or when it's clear that the collection is 
small in size.
   
   Similar to iteritems, I am using itervalues here to avoid efficiency losses 
in Python2


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119011)
Time Spent: 5h 50m  (was: 5h 40m)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=119008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-119008
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 09:43
Start Date: 04/Jul/18 09:43
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200068862
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
 ##
 @@ -28,12 +31,13 @@
 import tempfile
 import time
 from datetime import datetime
-from StringIO import StringIO
+from io import BytesIO
 
 Review comment:
   Are there any guidelines on the style? If not, I'll make it consistent with 
the `io.BytesIO` 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 119008)
Time Spent: 5h 40m  (was: 5.5h)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4417) BigqueryIO Numeric datatype Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4417?focusedWorklogId=118995=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118995
 ]

ASF GitHub Bot logged work on BEAM-4417:


Author: ASF GitHub Bot
Created on: 04/Jul/18 09:01
Start Date: 04/Jul/18 09:01
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #5755: [BEAM-4417] Support 
BigQuery's NUMERIC type using Java
URL: https://github.com/apache/beam/pull/5755#issuecomment-402414151
 
 
   @ElliottBrossard we fixed the JIRA mentioned above 
([link](https://issues.apache.org/jira/browse/BEAM-4706)). FWIW, we run the 
test successfully on Dataflow and direct runners (see the comment in the JIRA 
issue) after your change was merged. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118995)
Time Spent: 4h 40m  (was: 4.5h)

> BigqueryIO Numeric datatype Support
> ---
>
> Key: BEAM-4417
> URL: https://issues.apache.org/jira/browse/BEAM-4417
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: Kishan Kumar
>Assignee: Chamikara Jayalath
>Priority: Critical
>  Labels: newbie, patch
> Fix For: 2.6.0
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> The BigQueryIO.read fails while parsing the data from the avro file generated 
> while reading the data from the table which has columns with *Numeric* 
> datatypes. 
> We have gone through the source code at Git-Hub and noticed that *Numeric 
> data type is not yet supported.* 
>  
> Caused by: com.google.common.base.VerifyException: Unsupported BigQuery type: 
> NUMERIC
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-4706) BigQueryTornadoesIT cannot be run using integrationTest and performanceTest tasks

[Lukasz Gajowy (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532438#comment-16532438
 ] 

Lukasz Gajowy edited comment on BEAM-4706 at 7/4/18 9:00 AM:
-

Now it is possible to run this test successfully using designated gradle tasks. 
Example runs: 

using perfkit high-level task:
{code:java}
./gradlew performanceTest -DpkbLocation="" 
-DintegrationTestPipelineOptions='["--tempLocation=gs:///", 
"--project=", "--tempRoot=gs:/"]' 
-DitModule=examples/java 
-DintegrationTest=org.apache.beam.examples.cookbook.BigQueryTornadoesIT 
-DintegrationTestRunner=dataflow
{code}
using low-level integration test task (all pipeline options have to be set 
manually):
 
{code:java}
./gradlew integrationTest -p examples/java/ 
-DintegrationTestPipelineOptions='["--tempLocation=gs://, 
"--project=", "--tempRoot=gs://", 
"--runner=TestDataflowRunner"]' --tests 
org.apache.beam.examples.cookbook.BigQueryTornadoesIT 
-DintegrationTestRunner=dataflow
{code}
 
it can be also run using direct runner (this is described in 
BigQueryTornadoesIT class' docs now). 

FYI: IOIT documentation is being improved in [a separate 
PR|https://github.com/apache/beam-site/pull/465]. It' doesn't cover 
BigQueryTornadoes though. I wrote the examples above to be used at least until 
appropriate docs appear.

 


was (Author: łukaszg):
Now it is possible to run this test using designated gradle tasks. Example 
runs: 

using perfkit high-level task:
{code}
./gradlew performanceTest -DpkbLocation="" 
-DintegrationTestPipelineOptions='["--tempLocation=gs:///", 
"--project=", "--tempRoot=gs:/"]' 
-DitModule=examples/java 
-DintegrationTest=org.apache.beam.examples.cookbook.BigQueryTornadoesIT 
-DintegrationTestRunner=dataflow
{code}
using low-level integration test task (all pipeline options have to be set 
manually):
 
{code}
./gradlew integrationTest -p examples/java/ 
-DintegrationTestPipelineOptions='["--tempLocation=gs://, 
"--project=", "--tempRoot=gs://", 
"--runner=TestDataflowRunner"]' --tests 
org.apache.beam.examples.cookbook.BigQueryTornadoesIT 
-DintegrationTestRunner=dataflow
{code}
 
it can be also run using direct runner (this is described in 
BigQueryTornadoesIT class' docs now). 

FYI: IOIT documentation is being improved in [a separate 
PR|https://github.com/apache/beam-site/pull/465]. It' doesn't cover 
BigQueryTornadoes though. I wrote the examples above to be used at least until 
appropriate docs appear.

 

> BigQueryTornadoesIT cannot be run using integrationTest and performanceTest 
> tasks
> -
>
> Key: BEAM-4706
> URL: https://issues.apache.org/jira/browse/BEAM-4706
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> It seems that we cannot run this test using tasks from BuildModulePlugin 
> designed to run such tests. Those are not included in build.gradle of 
> examples module. There's a possibility that some other tests are not able to 
> run due to this reason (this also has to be checked).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4706) BigQueryTornadoesIT cannot be run using integrationTest and performanceTest tasks

[Lukasz Gajowy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lukasz Gajowy closed BEAM-4706.
---
   Resolution: Fixed
Fix Version/s: 2.6.0

> BigQueryTornadoesIT cannot be run using integrationTest and performanceTest 
> tasks
> -
>
> Key: BEAM-4706
> URL: https://issues.apache.org/jira/browse/BEAM-4706
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
> Fix For: 2.6.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> It seems that we cannot run this test using tasks from BuildModulePlugin 
> designed to run such tests. Those are not included in build.gradle of 
> examples module. There's a possibility that some other tests are not able to 
> run due to this reason (this also has to be checked).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=118994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118994
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 08:58
Start Date: 04/Jul/18 08:58
Worklog Time Spent: 10m 
  Work Description: superbobry commented on a change in pull request #5715: 
[BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200055745
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsource.py
 ##
 @@ -49,6 +48,13 @@
 
 __all__ = ['FileBasedSource']
 
+try:
+  unicode   # pylint: disable=unicode-builtin
 
 Review comment:
   I'd humbly suggest reconsidering (unless there are indeed issues with using 
`past` in some modules) as try-except is much more verbose syntactically.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118994)
Time Spent: 2h 50m  (was: 2h 40m)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4137) Split IOTestPipelineOptions to multiple, test-specific files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4137?focusedWorklogId=118993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118993
 ]

ASF GitHub Bot logged work on BEAM-4137:


Author: ASF GitHub Bot
Created on: 04/Jul/18 08:55
Start Date: 04/Jul/18 08:55
Worklog Time Spent: 10m 
  Work Description: echauchot closed pull request #5876: [BEAM-4137] Divide 
elasticsearch pipeline options
URL: https://github.com/apache/beam/pull/5876
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
index 132e3cc9094..b3733aee34c 100644
--- 
a/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
+++ 
b/sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
@@ -21,31 +21,8 @@
 import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.testing.TestPipelineOptions;
 
-/**
- * This shared set of options is used so that the full suite of IO tests can 
be run in one pass - if
- * a test tries to read TestPipelineOptions, it must be able to understand all 
the options that were
- * passed on the command line.
- */
+/** Pipeline options common for IO integration tests. */
 public interface IOTestPipelineOptions extends TestPipelineOptions {
-  /* Elasticsearch */
-  @Description("Server name for Elasticsearch server (host name/ip address)")
-  @Default.String("elasticsearch-server-name")
-  String getElasticsearchServer();
-
-  void setElasticsearchServer(String value);
-
-  @Description("Http port for elasticsearch server")
-  @Default.Integer(9200)
-  Integer getElasticsearchHttpPort();
-
-  void setElasticsearchHttpPort(Integer value);
-
-  /* Solr */
-  @Description("Address of Zookeeper server for Solr")
-  @Default.String("zookeeper-server")
-  String getSolrZookeeperServer();
-
-  void setSolrZookeeperServer(String value);
 
   /* Used by most IOIT */
   @Description("Number records that will be written and read by the test")
diff --git 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
index 0bcf228eec4..258121874d3 100644
--- 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
+++ 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
@@ -25,7 +25,7 @@
 
 import java.util.List;
 import org.apache.beam.sdk.io.BoundedSource;
-import org.apache.beam.sdk.io.common.IOTestPipelineOptions;
+import 
org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOITCommon.ElasticsearchPipelineOptions;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.testing.SourceTestUtils;
 import org.apache.beam.sdk.testing.TestPipeline;
@@ -58,7 +58,7 @@
  */
 public class ElasticsearchIOIT {
   private static RestClient restClient;
-  private static IOTestPipelineOptions options;
+  private static ElasticsearchPipelineOptions options;
   private static ConnectionConfiguration readConnectionConfiguration;
   private static ConnectionConfiguration writeConnectionConfiguration;
   private static ConnectionConfiguration updateConnectionConfiguration;
@@ -68,8 +68,8 @@
 
   @BeforeClass
   public static void beforeClass() throws Exception {
-PipelineOptionsFactory.register(IOTestPipelineOptions.class);
-options = 
TestPipeline.testingPipelineOptions().as(IOTestPipelineOptions.class);
+PipelineOptionsFactory.register(ElasticsearchPipelineOptions.class);
+options = 
TestPipeline.testingPipelineOptions().as(ElasticsearchPipelineOptions.class);
 readConnectionConfiguration =
 ElasticsearchIOITCommon.getConnectionConfiguration(
 options, ElasticsearchIOITCommon.IndexMode.READ);
diff --git 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
index c8bfa5658cb..b38b1696141 100644
--- 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOIT.java
+++ 

[beam] 01/01: Merge pull request #5876 from lgajowy/divide-elasticsearch-pipeline-options

[echauchot]

This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 824ea5332785b3df5a6453f376a975e4c66d99ab
Merge: bfe9b17 c01a12b
Author: Etienne Chauchot 
AuthorDate: Wed Jul 4 10:55:18 2018 +0200

Merge pull request #5876 from lgajowy/divide-elasticsearch-pipeline-options

Merge pull request #5876: [BEAM-4137] Divide elasticsearch pipeline options

 .../beam/sdk/io/common/IOTestPipelineOptions.java  | 25 +
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java|  8 +++---
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java|  8 +++---
 .../io/elasticsearch/ElasticsearchIOITCommon.java  | 31 +-
 4 files changed, 34 insertions(+), 38 deletions(-)

[beam] branch master updated (bfe9b17 -> 824ea53)

[echauchot]

This is an automated email from the ASF dual-hosted git repository.

echauchot pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bfe9b17  Merge pull request #5827 from Use GCP Credentials in Docker
 add d369ca6  [BEAM-4137] Divide separate PipelineOptions for elasticsearch 
tests
 add c01a12b  [BEAM-4137] Remove unused options from IOTestPipelineOptions
 new 824ea53  Merge pull request #5876 from 
lgajowy/divide-elasticsearch-pipeline-options

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/common/IOTestPipelineOptions.java  | 25 +
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java|  8 +++---
 .../sdk/io/elasticsearch/ElasticsearchIOIT.java|  8 +++---
 .../io/elasticsearch/ElasticsearchIOITCommon.java  | 31 +-
 4 files changed, 34 insertions(+), 38 deletions(-)

[jira] [Commented] (BEAM-4706) BigQueryTornadoesIT cannot be run using integrationTest and performanceTest tasks

[Lukasz Gajowy (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532438#comment-16532438
 ] 

Lukasz Gajowy commented on BEAM-4706:
-

Now it is possible to run this test using designated gradle tasks. Example 
runs: 

using perfkit high-level task:
{code}
./gradlew performanceTest -DpkbLocation="" 
-DintegrationTestPipelineOptions='["--tempLocation=gs:///", 
"--project=", "--tempRoot=gs:/"]' 
-DitModule=examples/java 
-DintegrationTest=org.apache.beam.examples.cookbook.BigQueryTornadoesIT 
-DintegrationTestRunner=dataflow
{code}
using low-level integration test task (all pipeline options have to be set 
manually):
 
{code}
./gradlew integrationTest -p examples/java/ 
-DintegrationTestPipelineOptions='["--tempLocation=gs://, 
"--project=", "--tempRoot=gs://", 
"--runner=TestDataflowRunner"]' --tests 
org.apache.beam.examples.cookbook.BigQueryTornadoesIT 
-DintegrationTestRunner=dataflow
{code}
 
it can be also run using direct runner (this is described in 
BigQueryTornadoesIT class' docs now). 

FYI: IOIT documentation is being improved in [a separate 
PR|https://github.com/apache/beam-site/pull/465]. It' doesn't cover 
BigQueryTornadoes though. I wrote the examples above to be used at least until 
appropriate docs appear.

 

> BigQueryTornadoesIT cannot be run using integrationTest and performanceTest 
> tasks
> -
>
> Key: BEAM-4706
> URL: https://issues.apache.org/jira/browse/BEAM-4706
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Kasia Kucharczyk
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> It seems that we cannot run this test using tasks from BuildModulePlugin 
> designed to run such tests. Those are not included in build.gradle of 
> examples module. There's a possibility that some other tests are not able to 
> run due to this reason (this also has to be checked).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4137) Split IOTestPipelineOptions to multiple, test-specific files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4137?focusedWorklogId=118992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118992
 ]

ASF GitHub Bot logged work on BEAM-4137:


Author: ASF GitHub Bot
Created on: 04/Jul/18 08:52
Start Date: 04/Jul/18 08:52
Worklog Time Spent: 10m 
  Work Description: echauchot commented on issue #5876: [BEAM-4137] Divide 
elasticsearch pipeline options
URL: https://github.com/apache/beam/pull/5876#issuecomment-402411489
 
 
   @lgajowy if you have run the integration tests locally, then it is fine for 
me. Indeed, we will need to create missing jenkins jobs for ITs. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118992)
Time Spent: 9h 40m  (was: 9.5h)

> Split IOTestPipelineOptions to multiple, test-specific files
> 
>
> Key: BEAM-4137
> URL: https://issues.apache.org/jira/browse/BEAM-4137
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Minor
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>
> Currently we have one big IOTestPipelineOptions interface that is used in 
> many IOITs. It contains test specific options that should rather be located 
> next to testing classes, not in a generic file. Let's split this. 
> Additionally, besides separation of concerns,  this will allow adding 
> test-specific @Default and @Required annotations and validate the options 
> better. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4137) Split IOTestPipelineOptions to multiple, test-specific files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4137?focusedWorklogId=118991=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118991
 ]

ASF GitHub Bot logged work on BEAM-4137:


Author: ASF GitHub Bot
Created on: 04/Jul/18 08:48
Start Date: 04/Jul/18 08:48
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #5876: 
[BEAM-4137] Divide elasticsearch pipeline options
URL: https://github.com/apache/beam/pull/5876#discussion_r200052744
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
 ##
 @@ -21,31 +21,8 @@
 import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.testing.TestPipelineOptions;
 
-/**
- * This shared set of options is used so that the full suite of IO tests can 
be run in one pass - if
- * a test tries to read TestPipelineOptions, it must be able to understand all 
the options that were
- * passed on the command line.
- */
+/** Pipeline options common for IO integration tests. */
 public interface IOTestPipelineOptions extends TestPipelineOptions {
-  /* Elasticsearch */
-  @Description("Server name for Elasticsearch server (host name/ip address)")
-  @Default.String("elasticsearch-server-name")
-  String getElasticsearchServer();
-
-  void setElasticsearchServer(String value);
-
-  @Description("Http port for elasticsearch server")
-  @Default.Integer(9200)
-  Integer getElasticsearchHttpPort();
-
-  void setElasticsearchHttpPort(Integer value);
-
-  /* Solr */
 
 Review comment:
   I did not know it was dead code. You were right to remove it. Yes a separate 
commit is enough of course.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118991)
Time Spent: 9.5h  (was: 9h 20m)

> Split IOTestPipelineOptions to multiple, test-specific files
> 
>
> Key: BEAM-4137
> URL: https://issues.apache.org/jira/browse/BEAM-4137
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Lukasz Gajowy
>Assignee: Lukasz Gajowy
>Priority: Minor
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>
> Currently we have one big IOTestPipelineOptions interface that is used in 
> many IOITs. It contains test specific options that should rather be located 
> next to testing classes, not in a generic file. Let's split this. 
> Additionally, besides separation of concerns,  this will allow adding 
> test-specific @Default and @Required annotations and validate the options 
> better. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4003) Futurize and fix python 2 compatibility for runners subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4003?focusedWorklogId=118990=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118990
 ]

ASF GitHub Bot logged work on BEAM-4003:


Author: ASF GitHub Bot
Created on: 04/Jul/18 08:34
Start Date: 04/Jul/18 08:34
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5373: 
[BEAM-4003] Futurize runners subpackage
URL: https://github.com/apache/beam/pull/5373#discussion_r200048708
 
 

 ##
 File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py
 ##
 @@ -52,6 +58,8 @@
 from apache_beam.utils import proto_utils
 from apache_beam.utils.plugin import BeamPlugin
 
+standard_library.install_aliases()
 
 Review comment:
   Introduced by futurize step because of `import urllib` 
([documentation](http://python-future.org/standard_library_imports.html)). 
However, since we import urllib from future, this is indeed not required anymore


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118990)
Time Spent: 5.5h  (was: 5h 20m)

> Futurize and fix python 2 compatibility for runners subpackage
> --
>
> Key: BEAM-4003
> URL: https://issues.apache.org/jira/browse/BEAM-4003
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4137) Split IOTestPipelineOptions to multiple, test-specific files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4137?focusedWorklogId=118986=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118986
 ]

ASF GitHub Bot logged work on BEAM-4137:


Author: ASF GitHub Bot
Created on: 04/Jul/18 08:12
Start Date: 04/Jul/18 08:12
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #5876: [BEAM-4137] Divide 
elasticsearch pipeline options
URL: https://github.com/apache/beam/pull/5876#issuecomment-402400371
 
 
   @echauchot phrase triggering is impossible now on the current codebase. It 
would be possible only when a Jenkins job exists that runs the elasticsearch 
tests. There is no such job yet - this is another task [that JB is 
on](https://issues.apache.org/jira/browse/BEAM-4398). I just refactored here 
simply to finish BEAM-4137. However, FWIW, I tested this locally and it reads 
the options well. Please let me know if such test is enough. 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118986)
Time Spent: 9h 10m  (was: 9h)

> Split IOTestPipelineOptions to multiple, test-specific files
> 
>
> Key: BEAM-4137
> URL: https://issues.apache.org/jira/browse/BEAM-4137
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Łukasz Gajowy
>Priority: Minor
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>
> Currently we have one big IOTestPipelineOptions interface that is used in 
> many IOITs. It contains test specific options that should rather be located 
> next to testing classes, not in a generic file. Let's split this. 
> Additionally, besides separation of concerns,  this will allow adding 
> test-specific @Default and @Required annotations and validate the options 
> better. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4137) Split IOTestPipelineOptions to multiple, test-specific files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4137?focusedWorklogId=118987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118987
 ]

ASF GitHub Bot logged work on BEAM-4137:


Author: ASF GitHub Bot
Created on: 04/Jul/18 08:12
Start Date: 04/Jul/18 08:12
Worklog Time Spent: 10m 
  Work Description: lgajowy edited a comment on issue #5876: [BEAM-4137] 
Divide elasticsearch pipeline options
URL: https://github.com/apache/beam/pull/5876#issuecomment-402400371
 
 
   @echauchot phrase triggering is impossible now on the current codebase. It 
would be possible only when a Jenkins job exists that runs the elasticsearch 
tests. There is no such job yet - this is another task [that is yet to be 
done](https://issues.apache.org/jira/browse/BEAM-4398). I just refactored here 
simply to finish BEAM-4137. However, FWIW, I tested this locally and it reads 
the options well. Please let me know if such test is enough. 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118987)
Time Spent: 9h 20m  (was: 9h 10m)

> Split IOTestPipelineOptions to multiple, test-specific files
> 
>
> Key: BEAM-4137
> URL: https://issues.apache.org/jira/browse/BEAM-4137
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Łukasz Gajowy
>Priority: Minor
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> Currently we have one big IOTestPipelineOptions interface that is used in 
> many IOITs. It contains test specific options that should rather be located 
> next to testing classes, not in a generic file. Let's split this. 
> Additionally, besides separation of concerns,  this will allow adding 
> test-specific @Default and @Required annotations and validate the options 
> better. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4137) Split IOTestPipelineOptions to multiple, test-specific files

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4137?focusedWorklogId=118985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118985
 ]

ASF GitHub Bot logged work on BEAM-4137:


Author: ASF GitHub Bot
Created on: 04/Jul/18 08:05
Start Date: 04/Jul/18 08:05
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on a change in pull request #5876: 
[BEAM-4137] Divide elasticsearch pipeline options
URL: https://github.com/apache/beam/pull/5876#discussion_r200040448
 
 

 ##
 File path: 
sdks/java/io/common/src/test/java/org/apache/beam/sdk/io/common/IOTestPipelineOptions.java
 ##
 @@ -21,31 +21,8 @@
 import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.testing.TestPipelineOptions;
 
-/**
- * This shared set of options is used so that the full suite of IO tests can 
be run in one pass - if
- * a test tries to read TestPipelineOptions, it must be able to understand all 
the options that were
- * passed on the command line.
- */
+/** Pipeline options common for IO integration tests. */
 public interface IOTestPipelineOptions extends TestPipelineOptions {
-  /* Elasticsearch */
-  @Description("Server name for Elasticsearch server (host name/ip address)")
-  @Default.String("elasticsearch-server-name")
-  String getElasticsearchServer();
-
-  void setElasticsearchServer(String value);
-
-  @Description("Http port for elasticsearch server")
-  @Default.Integer(9200)
-  Integer getElasticsearchHttpPort();
-
-  void setElasticsearchHttpPort(Integer value);
-
-  /* Solr */
 
 Review comment:
   As for solr - I didn't find any usages of those options - there are no Solr 
tests nor any code that use it. It's a dead code so I deleted it. I isolated 
the removal in a separate commit. Isn't this enough? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118985)
Time Spent: 9h  (was: 8h 50m)

> Split IOTestPipelineOptions to multiple, test-specific files
> 
>
> Key: BEAM-4137
> URL: https://issues.apache.org/jira/browse/BEAM-4137
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Łukasz Gajowy
>Priority: Minor
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> Currently we have one big IOTestPipelineOptions interface that is used in 
> many IOITs. It contains test specific options that should rather be located 
> next to testing classes, not in a generic file. Let's split this. 
> Additionally, besides separation of concerns,  this will allow adding 
> test-specific @Default and @Required annotations and validate the options 
> better. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=118981=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118981
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 07:35
Start Date: 04/Jul/18 07:35
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5715: 
[BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200032184
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsource.py
 ##
 @@ -49,6 +48,13 @@
 
 __all__ = ['FileBasedSource']
 
+try:
+  unicode   # pylint: disable=unicode-builtin
 
 Review comment:
   I used the try/except method to be consistent with the methodology 
implemented in #5053 and outlined in [the 
doc](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit#heading=h.ir03vqq4rotn)
 by @RobbeSneyders. I believe there where some issues in the coder package for 
the typechecks using `past.builtins`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118981)
Time Spent: 2h 40m  (was: 2.5h)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4000) Futurize and fix python 2 compatibility for io subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4000?focusedWorklogId=118980=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118980
 ]

ASF GitHub Bot logged work on BEAM-4000:


Author: ASF GitHub Bot
Created on: 04/Jul/18 07:34
Start Date: 04/Jul/18 07:34
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5715: 
[BEAM-4000] Futurize io subpackage
URL: https://github.com/apache/beam/pull/5715#discussion_r200032184
 
 

 ##
 File path: sdks/python/apache_beam/io/filebasedsource.py
 ##
 @@ -49,6 +48,13 @@
 
 __all__ = ['FileBasedSource']
 
+try:
+  unicode   # pylint: disable=unicode-builtin
 
 Review comment:
   I used the try/except method to be consistent with the methodology 
implemented in #5053 and outlined in [the 
doc](https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE/edit#heading=h.ir03vqq4rotn)
 by @RobbeSneyders. I believe there where some issues in the coder package for 
the typechecks using `past.utils`


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118980)
Time Spent: 2.5h  (was: 2h 20m)

> Futurize and fix python 2 compatibility for io subpackage
> -
>
> Key: BEAM-4000
> URL: https://issues.apache.org/jira/browse/BEAM-4000
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=118971=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118971
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 04/Jul/18 06:57
Start Date: 04/Jul/18 06:57
Worklog Time Spent: 10m 
  Work Description: superbobry edited a comment on issue #5869: [BEAM-1251] 
Replace NameError-driven dispatch with ``past``
URL: https://github.com/apache/beam/pull/5869#issuecomment-402380346
 
 
   Will do. I can fix `buffer` if you are short on time, just let me know.
   
   @cclauss also, can we merged futurization PRs first?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118971)
Time Spent: 11.5h  (was: 11h 20m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Assignee: Robbe
>Priority: Trivial
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1251) Python 3 Support

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1251?focusedWorklogId=118970=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-118970
 ]

ASF GitHub Bot logged work on BEAM-1251:


Author: ASF GitHub Bot
Created on: 04/Jul/18 06:47
Start Date: 04/Jul/18 06:47
Worklog Time Spent: 10m 
  Work Description: superbobry commented on issue #5869: [BEAM-1251] 
Replace NameError-driven dispatch with ``past``
URL: https://github.com/apache/beam/pull/5869#issuecomment-402380346
 
 
   Will do. I can fix `buffer` if you are short on time, just let me know.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 118970)
Time Spent: 11h 20m  (was: 11h 10m)

> Python 3 Support
> 
>
> Key: BEAM-1251
> URL: https://issues.apache.org/jira/browse/BEAM-1251
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Eyad Sibai
>Assignee: Robbe
>Priority: Trivial
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>
> I have been trying to use google datalab with python3. As I see there are 
> several packages that does not support python3 yet which google datalab 
> depends on. This is one of them.
> https://github.com/GoogleCloudPlatform/DataflowPythonSDK/issues/6



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1018

[Apache Jenkins Server]

See 


Changes:

[robertwb] Use local GCP credentials, if any, in local docker environment.

--
[...truncated 19.15 MB...]
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:37.951Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Read information schema
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:37.979Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Read
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:38.006Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:38.033Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:38.059Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Extract
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:38.087Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/WithKeys/AddKeys/Map
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:38.114Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey+SpannerIO.Write/Write
 mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/Combine.GroupedValues/Partial
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:38.144Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)/GroupByKey/Reify
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:38.176Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 0/ParDo(CollectWindows) 
into SpannerIO.Write/To mutation group
Jul 04, 2018 6:37:42 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:37:38.208Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Extract keys into SpannerIO.Write/Write mutations 

Build failed in Jenkins: beam_PostCommit_Python_Verify #5509

[Apache Jenkins Server]

See 


Changes:

[robertwb] Use local GCP credentials, if any, in local docker environment.

--
[...truncated 1.12 MB...]
test_match_type_variables 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_key_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_invalid_value_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_composite_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_check_valid_simple_type 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_type_checks_not_dict 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_value_type_must_be_valid_composite_param 
(apache_beam.typehints.typehints_test.DictHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.GeneratorHintTestCase) 
... ok
test_generator_argument_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_generator_return_hint_invalid_yield_type 
(apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 

Build failed in Jenkins: beam_PreCommit_Java_Cron #63

[Apache Jenkins Server]

See 


Changes:

[robertwb] Use local GCP credentials, if any, in local docker environment.

--
[...truncated 16.95 MB...]
INFO: 2018-07-04T06:17:56.606Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:56.824Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:56.848Z: Unzipping flatten s13 for input 
s12.org.apache.beam.sdk.values.PCollection.:349#1d275f544daf228c
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:56.878Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map, through flatten 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections,
 into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:56.909Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:56.933Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:56.966Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:56.997Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:57.028Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:57.060Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:57.091Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:57.112Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:57.135Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 04, 2018 6:17:59 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-04T06:17:57.162Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into