[jira] [Work logged] (BEAM-4862) varint overflow -62135596800 exception with Cloud Spanner Timestamp 0001-01-01T00:00:00Z

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4862?focusedWorklogId=128381=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128381
 ]

ASF GitHub Bot logged work on BEAM-4862:


Author: ASF GitHub Bot
Created on: 28/Jul/18 01:51
Start Date: 28/Jul/18 01:51
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6077: BEAM-4862 Fixes 
bug in Spanner's MutationGroupEncoder by converting timestamps into Long and 
not Int.
URL: https://github.com/apache/beam/pull/6077#issuecomment-408574596
 
 
   Format of the PR title should be "[BEAM-4862] Fixes..."


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128381)
Time Spent: 1h  (was: 50m)
Remaining Estimate: 23h  (was: 23h 10m)

> varint overflow -62135596800 exception with Cloud Spanner Timestamp 
> 0001-01-01T00:00:00Z
> 
>
> Key: BEAM-4862
> URL: https://issues.apache.org/jira/browse/BEAM-4862
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Eric Beach
>Assignee: Chamikara Jayalath
>Priority: Minor
>   Original Estimate: 24h
>  Time Spent: 1h
>  Remaining Estimate: 23h
>
> tl;dr - If you try to write a Timestamp of value "0001-01-01T00:00:00Z" as a 
> Spanner Mutation, you get an overflow error.
>  
> The crux of the issue appears to be that 0001-01-01T00:00:00Z, which is a 
> valid Timestamp per 
> [https://cloud.google.com/spanner/docs/data-types#timestamp-type], is too 
> large for an integer. See the two lines of code below. 
> [https://github.com/apache/beam/blob/release-2.5.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoder.java#L453]
> [https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java#L58]
>  
>  
> Stack Trade
> {{Caused by: java.io.IOException: varint overflow -62135596800 at 
> org.apache.beam.sdk.util.VarInt.decodeInt(VarInt.java:65) at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodePrimitive(MutationGroupEncoder.java:453)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeModification(MutationGroupEncoder.java:326)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeMutation(MutationGroupEncoder.java:280)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode(MutationGroupEncoder.java:264)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement(SpannerIO.java:1030)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn$DoFnInvoker.invokeProcessElement(Unknown
>  Source) at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
>  at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:146)
>  at 
> com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:181)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:124)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:53)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:115)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:113)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
>  at 

[jira] [Work logged] (BEAM-4862) varint overflow -62135596800 exception with Cloud Spanner Timestamp 0001-01-01T00:00:00Z

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4862?focusedWorklogId=128380=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128380
 ]

ASF GitHub Bot logged work on BEAM-4862:


Author: ASF GitHub Bot
Created on: 28/Jul/18 01:50
Start Date: 28/Jul/18 01:50
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#6077: BEAM-4862 Fixes bug in Spanner's MutationGroupEncoder by converting 
timestamps into Long and not Int.
URL: https://github.com/apache/beam/pull/6077#discussion_r205929233
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoderTest.java
 ##
 @@ -528,6 +528,32 @@ public void dateKeys() throws Exception {
 verifyEncodedOrdering(schema, "test", keys);
   }
 
+  @Test
+  // Test for Jira issue BEAM-4862
 
 Review comment:
   +1


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128380)
Time Spent: 50m  (was: 40m)
Remaining Estimate: 23h 10m  (was: 23h 20m)

> varint overflow -62135596800 exception with Cloud Spanner Timestamp 
> 0001-01-01T00:00:00Z
> 
>
> Key: BEAM-4862
> URL: https://issues.apache.org/jira/browse/BEAM-4862
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Eric Beach
>Assignee: Chamikara Jayalath
>Priority: Minor
>   Original Estimate: 24h
>  Time Spent: 50m
>  Remaining Estimate: 23h 10m
>
> tl;dr - If you try to write a Timestamp of value "0001-01-01T00:00:00Z" as a 
> Spanner Mutation, you get an overflow error.
>  
> The crux of the issue appears to be that 0001-01-01T00:00:00Z, which is a 
> valid Timestamp per 
> [https://cloud.google.com/spanner/docs/data-types#timestamp-type], is too 
> large for an integer. See the two lines of code below. 
> [https://github.com/apache/beam/blob/release-2.5.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoder.java#L453]
> [https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java#L58]
>  
>  
> Stack Trade
> {{Caused by: java.io.IOException: varint overflow -62135596800 at 
> org.apache.beam.sdk.util.VarInt.decodeInt(VarInt.java:65) at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodePrimitive(MutationGroupEncoder.java:453)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeModification(MutationGroupEncoder.java:326)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeMutation(MutationGroupEncoder.java:280)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode(MutationGroupEncoder.java:264)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement(SpannerIO.java:1030)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn$DoFnInvoker.invokeProcessElement(Unknown
>  Source) at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
>  at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:146)
>  at 
> com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:181)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:124)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:53)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:115)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:113)
>  at 
> 

[jira] [Work logged] (BEAM-4862) varint overflow -62135596800 exception with Cloud Spanner Timestamp 0001-01-01T00:00:00Z

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4862?focusedWorklogId=128379=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128379
 ]

ASF GitHub Bot logged work on BEAM-4862:


Author: ASF GitHub Bot
Created on: 28/Jul/18 01:50
Start Date: 28/Jul/18 01:50
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#6077: BEAM-4862 Fixes bug in Spanner's MutationGroupEncoder by converting 
timestamps into Long and not Int.
URL: https://github.com/apache/beam/pull/6077#discussion_r205929228
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoderTest.java
 ##
 @@ -528,6 +528,32 @@ public void dateKeys() throws Exception {
 verifyEncodedOrdering(schema, "test", keys);
   }
 
+  @Test
+  // Test for Jira issue BEAM-4862
+  public void decodeTimestampMutationGroup() {
+SpannerSchema spannerSchemaTimestamp =
+SpannerSchema.builder().addColumn("timestampTest", "timestamp", 
"TIMESTAMP").build();
+Timestamp timestamp1 = Timestamp.now();
+Mutation mutation1 =
+
Mutation.newInsertOrUpdateBuilder("timestampTest").set("timestamp").to(timestamp1).build();
 
 Review comment:
   Are all these scenarios failing currently ? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128379)
Time Spent: 40m  (was: 0.5h)
Remaining Estimate: 23h 20m  (was: 23.5h)

> varint overflow -62135596800 exception with Cloud Spanner Timestamp 
> 0001-01-01T00:00:00Z
> 
>
> Key: BEAM-4862
> URL: https://issues.apache.org/jira/browse/BEAM-4862
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Eric Beach
>Assignee: Chamikara Jayalath
>Priority: Minor
>   Original Estimate: 24h
>  Time Spent: 40m
>  Remaining Estimate: 23h 20m
>
> tl;dr - If you try to write a Timestamp of value "0001-01-01T00:00:00Z" as a 
> Spanner Mutation, you get an overflow error.
>  
> The crux of the issue appears to be that 0001-01-01T00:00:00Z, which is a 
> valid Timestamp per 
> [https://cloud.google.com/spanner/docs/data-types#timestamp-type], is too 
> large for an integer. See the two lines of code below. 
> [https://github.com/apache/beam/blob/release-2.5.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoder.java#L453]
> [https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java#L58]
>  
>  
> Stack Trade
> {{Caused by: java.io.IOException: varint overflow -62135596800 at 
> org.apache.beam.sdk.util.VarInt.decodeInt(VarInt.java:65) at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodePrimitive(MutationGroupEncoder.java:453)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeModification(MutationGroupEncoder.java:326)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeMutation(MutationGroupEncoder.java:280)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode(MutationGroupEncoder.java:264)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement(SpannerIO.java:1030)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn$DoFnInvoker.invokeProcessElement(Unknown
>  Source) at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
>  at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:146)
>  at 
> com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:181)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:124)
>  at 
> 

[jira] [Work logged] (BEAM-5040) BigQueryIO retries infinitely in WriteTable and WriteRename

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5040?focusedWorklogId=128378=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128378
 ]

ASF GitHub Bot logged work on BEAM-5040:


Author: ASF GitHub Bot
Created on: 28/Jul/18 01:29
Start Date: 28/Jul/18 01:29
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6080: [BEAM-5040] Fix 
retry bug for BigQuery jobs.
URL: https://github.com/apache/beam/pull/6080#issuecomment-408573241
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128378)
Time Spent: 20m  (was: 10m)

> BigQueryIO retries infinitely in WriteTable and WriteRename
> ---
>
> Key: BEAM-5040
> URL: https://issues.apache.org/jira/browse/BEAM-5040
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> BigQueryIO retries infinitely in WriteTable and WriteRename
> Several failure scenarios with the current code:
>  # It's possible for a load job to return failure even though it actually 
> succeeded (e.g. the reply might have timed out). In this case, BigQueryIO 
> will retry the job which will fail again (because the job id has already been 
> used), leading to indefinite retries. Correct behavior is to stop retrying as 
> the load job has succeeded.
>  # It's possible for a load job to be accepted by BigQuery, but then to fail 
> on the BigQuery side. In this case a retry with the same job id will fail as 
> that job id has already been used. BigQueryIO will sometimes detect this, but 
> if the worker has restarted it will instead issue a load with the old job id 
> and go into a retry loop. Correct behavior is to generate a new deterministic 
> job id and retry using that new job id.
>  # In many cases of worker restart, BigQueryIO ends up in infinite retry 
> loops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1110

[Apache Jenkins Server]

See 


Changes:

[aaltay] [BEAM-4747] mkdirs if they don't exist in localfilesystem (#5903)

--
[...truncated 19.73 MB...]

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_OUT
Submitted job: 2018-07-27_17_50_23-5808058874969833577

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testReportFailures 
STANDARD_ERROR
Jul 28, 2018 12:50:24 AM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To cancel the job using the 'gcloud' tool, run:
> gcloud dataflow jobs --project=apache-beam-testing cancel 
--region=us-central1 2018-07-27_17_50_23-5808058874969833577
Jul 28, 2018 12:50:24 AM 
org.apache.beam.runners.dataflow.TestDataflowRunner run
INFO: Running Dataflow job 2018-07-27_17_50_23-5808058874969833577 with 0 
expected assertions.
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:23.797Z: Autoscaling is enabled for job 
2018-07-27_17_50_23-5808058874969833577. The number of workers will be between 
1 and 1000.
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:23.837Z: Autoscaling was automatically enabled for 
job 2018-07-27_17_50_23-5808058874969833577.
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:26.356Z: Checking required Cloud APIs are enabled.
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:26.480Z: Checking permissions granted to controller 
Service Account.
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:30.697Z: Worker configuration: n1-standard-1 in 
us-central1-b.
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.119Z: Expanding CoGroupByKey operations into 
optimizable parts.
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.353Z: Expanding GroupByKey operations into 
optimizable parts.
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.390Z: Lifting ValueCombiningMappingFns into 
MergeBucketsMappingFns
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.626Z: Fusing adjacent ParDo, Read, Write, and 
Flatten operations
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.668Z: Elided trivial flatten 
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.700Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map into SpannerIO.Write/Write 
mutations to Cloud Spanner/Create seed/Read(CreateSource)
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.741Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Read information schema into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/Wait/Map
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.785Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/ParDo(UseWindowHashAsKeyAndWindowAsSortKey)
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.820Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/ParDo(IsmRecordForSingularValuePerWindow) 
into SpannerIO.Write/Write mutations to Cloud Spanner/Schema 
View/Combine.GloballyAsSingletonView/BatchViewOverrides.GroupByWindowHashAsKeyAndWindowAsSortKey/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Read
Jul 28, 2018 12:50:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:50:31.855Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Schema 

[jira] [Work logged] (BEAM-5040) BigQueryIO retries infinitely in WriteTable and WriteRename

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5040?focusedWorklogId=128375=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128375
 ]

ASF GitHub Bot logged work on BEAM-5040:


Author: ASF GitHub Bot
Created on: 28/Jul/18 00:52
Start Date: 28/Jul/18 00:52
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6080: [BEAM-5040] Fix 
retry bug for BigQuery jobs.
URL: https://github.com/apache/beam/pull/6080#issuecomment-408570747
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128375)
Time Spent: 10m
Remaining Estimate: 0h

> BigQueryIO retries infinitely in WriteTable and WriteRename
> ---
>
> Key: BEAM-5040
> URL: https://issues.apache.org/jira/browse/BEAM-5040
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> BigQueryIO retries infinitely in WriteTable and WriteRename
> Several failure scenarios with the current code:
>  # It's possible for a load job to return failure even though it actually 
> succeeded (e.g. the reply might have timed out). In this case, BigQueryIO 
> will retry the job which will fail again (because the job id has already been 
> used), leading to indefinite retries. Correct behavior is to stop retrying as 
> the load job has succeeded.
>  # It's possible for a load job to be accepted by BigQuery, but then to fail 
> on the BigQuery side. In this case a retry with the same job id will fail as 
> that job id has already been used. BigQueryIO will sometimes detect this, but 
> if the worker has restarted it will instead issue a load with the old job id 
> and go into a retry loop. Correct behavior is to generate a new deterministic 
> job id and retry using that new job id.
>  # In many cases of worker restart, BigQueryIO ends up in infinite retry 
> loops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #784

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT #797

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_Verify #5601

[Apache Jenkins Server]

See 


Changes:

[aaltay] [BEAM-4747] mkdirs if they don't exist in localfilesystem (#5903)

--
[...truncated 1.08 MB...]
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_simple_type_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_functions_as_regular_generator 
(apache_beam.typehints.typehints_test.TestGeneratorWrapper) ... ok
test_compatibility (apache_beam.typehints.typehints_test.TupleHintTestCase) ... 
ok

Jenkins build is back to normal : beam_PerformanceTests_ParquetIOIT #290

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_ParquetIOIT_HDFS #185

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #477

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #790

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT_HDFS #467

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #466

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #567

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #818

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_JDBC #898

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #562

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PreCommit_Java_Cron #155

[Apache Jenkins Server]

See 


Changes:

[aaltay] [BEAM-4747] mkdirs if they don't exist in localfilesystem (#5903)

--
[...truncated 18.24 MB...]
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.250Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.296Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
 with random key
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.344Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.385Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.421Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.459Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Finalize 
into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Values/Values/Map
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.498Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.538Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
 into WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add 
void key/AddKeys/Map
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.586Z: Fusing consumer 
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial
 into WordCount.CountWords/Count.PerElement/Init/Map
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.629Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.674Z: Fusing consumer Window.Into()/Window.Assign 
into ParDo(AddTimestamp)
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.705Z: Fusing consumer 
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues
 into 
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read
Jul 28, 2018 12:17:35 AM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-28T00:17:29.744Z: Fusing consumer 
WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify 
into 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #466

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_Spark #2005

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3098) Upgrade Java grpc version

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3098?focusedWorklogId=128357=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128357
 ]

ASF GitHub Bot logged work on BEAM-3098:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:48
Start Date: 27/Jul/18 23:48
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #5767: [BEAM-3098] 
Upgrades gRPC and protobuf dependencies
URL: https://github.com/apache/beam/pull/5767#issuecomment-408565084
 
 
   Closing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128357)
Time Spent: 1h 20m  (was: 1h 10m)

> Upgrade Java grpc version
> -
>
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3098) Upgrade Java grpc version

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3098?focusedWorklogId=128358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128358
 ]

ASF GitHub Bot logged work on BEAM-3098:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:48
Start Date: 27/Jul/18 23:48
Worklog Time Spent: 10m 
  Work Description: chamikaramj closed pull request #5767: [BEAM-3098] 
Upgrades gRPC and protobuf dependencies
URL: https://github.com/apache/beam/pull/5767
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 807d00e8e1c..4c41ec710c9 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -307,10 +307,10 @@ class BeamModulePlugin implements Plugin {
 def bigtable_proto_version = "1.0.0-pre3"
 def google_clients_version = "1.23.0"
 def google_auth_version = "0.7.1"
-def grpc_version = "1.2.0"
-def protobuf_version = "3.2.0"
+def grpc_version = "1.12.0"
+def protobuf_version = "3.5.1"
 def guava_version = "20.0"
-def netty_version = "4.1.8.Final"
+def netty_version = "4.1.22.Final"
 def grpc_google_common_protos = "0.1.9"
 def hamcrest_version = "1.3"
 def hadoop_version = "2.7.3"
@@ -371,7 +371,7 @@ class BeamModulePlugin implements Plugin {
 google_cloud_core   : 
"com.google.cloud:google-cloud-core:1.0.2",
 google_cloud_core_grpc  : 
"com.google.cloud:google-cloud-core-grpc:$grpc_version",
 google_cloud_dataflow_java_proto_library_all: 
"com.google.cloud.dataflow:google-cloud-dataflow-java-proto-library-all:0.5.160304",
-google_cloud_spanner: 
"com.google.cloud:google-cloud-spanner:0.20.0b-beta",
+google_cloud_spanner: 
"com.google.cloud:google-cloud-spanner:0.53.0-beta",
 google_http_client  : 
"com.google.http-client:google-http-client:$google_clients_version",
 google_http_client_jackson  : 
"com.google.http-client:google-http-client-jackson:$google_clients_version",
 google_http_client_jackson2 : 
"com.google.http-client:google-http-client-jackson2:$google_clients_version",
@@ -411,7 +411,7 @@ class BeamModulePlugin implements Plugin {
 malhar_library  : 
"org.apache.apex:malhar-library:$apex_malhar_version",
 mockito_core: 
"org.mockito:mockito-core:1.9.5",
 netty_handler   : 
"io.netty:netty-handler:$netty_version",
-netty_tcnative_boringssl_static : 
"io.netty:netty-tcnative-boringssl-static:1.1.33.Fork26",
+netty_tcnative_boringssl_static : 
"io.netty:netty-tcnative-boringssl-static:2.0.7.Final",
 netty_transport_native_epoll: 
"io.netty:netty-transport-native-epoll:$netty_version",
 postgres: 
"org.postgresql:postgresql:$postgres_version",
 protobuf_java   : 
"com.google.protobuf:protobuf-java:$protobuf_version",
diff --git a/pom.xml b/pom.xml
index 93b230ffb10..6b4d7b5c7a7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -127,7 +127,7 @@
 1.0.2
 
0.5.160304
 20.0
-1.2.0
+1.12.0
 
0.1.9
 
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4747) Python LocalFileSystem directory-creation semantics

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4747?focusedWorklogId=128355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128355
 ]

ASF GitHub Bot logged work on BEAM-4747:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:46
Start Date: 27/Jul/18 23:46
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5903: [BEAM-4747] mkdirs if 
they don't exist in localfilesystem
URL: https://github.com/apache/beam/pull/5903
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/io/localfilesystem.py 
b/sdks/python/apache_beam/io/localfilesystem.py
index 23a1e8a846c..e373ad0b18a 100644
--- a/sdks/python/apache_beam/io/localfilesystem.py
+++ b/sdks/python/apache_beam/io/localfilesystem.py
@@ -144,6 +144,9 @@ def create(self, path, mime_type='application/octet-stream',
 
 Returns: file handle with a close function for the user to use
 """
+parent = os.path.dirname(path)
+if not os.path.exists(parent):
+  os.makedirs(parent)
 return self._path_open(path, 'wb', mime_type, compression_type)
 
   def open(self, path, mime_type='application/octet-stream',
@@ -185,6 +188,10 @@ def _copy_path(source, destination):
 if os.path.isdir(source):
   shutil.copytree(source, destination)
 else:
+  parent = os.path.dirname(destination)
+  if not os.path.exists(parent):
+os.makedirs(parent)
+
   shutil.copy2(source, destination)
   except OSError as err:
 raise IOError(err)
@@ -217,6 +224,10 @@ def rename(self, source_file_names, 
destination_file_names):
 def _rename_file(source, destination):
   """Rename a single file object"""
   try:
+parent = os.path.dirname(destination)
+if not os.path.exists(parent):
+  os.makedirs(parent)
+
 os.rename(source, destination)
   except OSError as err:
 raise IOError(err)
diff --git a/sdks/python/apache_beam/io/localfilesystem_test.py 
b/sdks/python/apache_beam/io/localfilesystem_test.py
index d6d8eb4c217..5d032db59a1 100644
--- a/sdks/python/apache_beam/io/localfilesystem_test.py
+++ b/sdks/python/apache_beam/io/localfilesystem_test.py
@@ -196,8 +196,8 @@ def test_copy_error(self):
  [(path1, path2)])
 
   def test_copy_directory(self):
-path_t1 = os.path.join(self.tmpdir, 't1')
-path_t2 = os.path.join(self.tmpdir, 't2')
+path_t1 = os.path.join(self.tmpdir, 't1/11')
+path_t2 = os.path.join(self.tmpdir, 't2/22')
 self.fs.mkdirs(path_t1)
 self.fs.mkdirs(path_t2)
 
@@ -209,6 +209,19 @@ def test_copy_directory(self):
 self.fs.copy([path_t1], [path_t2])
 self.assertTrue(filecmp.cmp(path1, path2))
 
+  def test_create_mkdirs_open(self):
+path = os.path.join(self.tmpdir, 't1/t2/t3')
+with self.fs.create(path) as f:
+  f.write("yay")
+
+with self.fs.open(path) as f:
+  self.assertEqual(f.read(), "yay")
+
+  def test_open_error(self):
+path = os.path.join(self.tmpdir, 't1')
+with self.assertRaisesRegexp(IOError, r'No such file or directory'):
+  self.fs.open(path)
+
   def test_rename(self):
 path1 = os.path.join(self.tmpdir, 'f1')
 path2 = os.path.join(self.tmpdir, 'f2')
@@ -244,6 +257,22 @@ def test_rename_directory(self):
 self.assertTrue(self.fs.exists(path2))
 self.assertFalse(self.fs.exists(path1))
 
+  def test_rename_mkdirs(self):
+path_t1 = os.path.join(self.tmpdir, 't1')
+path_t2 = os.path.join(self.tmpdir, 't2/t3/t4')
+self.fs.mkdirs(path_t1)
+
+path1 = os.path.join(path_t1, 'f1')
+path2 = os.path.join(path_t2, 'f1')
+with open(path1, 'a') as f:
+  f.write('Hello')
+
+self.fs.rename([path_t1], [path_t2])
+self.assertTrue(self.fs.exists(path_t2))
+self.assertFalse(self.fs.exists(path_t1))
+self.assertTrue(self.fs.exists(path2))
+self.assertFalse(self.fs.exists(path1))
+
   def test_exists(self):
 path1 = os.path.join(self.tmpdir, 'f1')
 path2 = os.path.join(self.tmpdir, 'f2')


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128355)
Time Spent: 50m  (was: 40m)

> Python LocalFileSystem directory-creation semantics
> ---
>
> Key: BEAM-4747
> URL: 

[beam] branch master updated: [BEAM-4747] mkdirs if they don't exist in localfilesystem (#5903)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new d903dcf  [BEAM-4747] mkdirs if they don't exist in localfilesystem 
(#5903)
d903dcf is described below

commit d903dcfc7ff7300355688b08b779da880f15fe9d
Author: Ryan Williams 
AuthorDate: Fri Jul 27 19:46:19 2018 -0400

[BEAM-4747] mkdirs if they don't exist in localfilesystem (#5903)

* mkdirs if they don't exist in localfilesystem
* make localfilesystem create ancestor directories for output paths
---
 sdks/python/apache_beam/io/localfilesystem.py  | 11 
 sdks/python/apache_beam/io/localfilesystem_test.py | 33 --
 2 files changed, 42 insertions(+), 2 deletions(-)

diff --git a/sdks/python/apache_beam/io/localfilesystem.py 
b/sdks/python/apache_beam/io/localfilesystem.py
index 23a1e8a..e373ad0 100644
--- a/sdks/python/apache_beam/io/localfilesystem.py
+++ b/sdks/python/apache_beam/io/localfilesystem.py
@@ -144,6 +144,9 @@ class LocalFileSystem(FileSystem):
 
 Returns: file handle with a close function for the user to use
 """
+parent = os.path.dirname(path)
+if not os.path.exists(parent):
+  os.makedirs(parent)
 return self._path_open(path, 'wb', mime_type, compression_type)
 
   def open(self, path, mime_type='application/octet-stream',
@@ -185,6 +188,10 @@ class LocalFileSystem(FileSystem):
 if os.path.isdir(source):
   shutil.copytree(source, destination)
 else:
+  parent = os.path.dirname(destination)
+  if not os.path.exists(parent):
+os.makedirs(parent)
+
   shutil.copy2(source, destination)
   except OSError as err:
 raise IOError(err)
@@ -217,6 +224,10 @@ class LocalFileSystem(FileSystem):
 def _rename_file(source, destination):
   """Rename a single file object"""
   try:
+parent = os.path.dirname(destination)
+if not os.path.exists(parent):
+  os.makedirs(parent)
+
 os.rename(source, destination)
   except OSError as err:
 raise IOError(err)
diff --git a/sdks/python/apache_beam/io/localfilesystem_test.py 
b/sdks/python/apache_beam/io/localfilesystem_test.py
index d6d8eb4..5d032db 100644
--- a/sdks/python/apache_beam/io/localfilesystem_test.py
+++ b/sdks/python/apache_beam/io/localfilesystem_test.py
@@ -196,8 +196,8 @@ class LocalFileSystemTest(unittest.TestCase):
  [(path1, path2)])
 
   def test_copy_directory(self):
-path_t1 = os.path.join(self.tmpdir, 't1')
-path_t2 = os.path.join(self.tmpdir, 't2')
+path_t1 = os.path.join(self.tmpdir, 't1/11')
+path_t2 = os.path.join(self.tmpdir, 't2/22')
 self.fs.mkdirs(path_t1)
 self.fs.mkdirs(path_t2)
 
@@ -209,6 +209,19 @@ class LocalFileSystemTest(unittest.TestCase):
 self.fs.copy([path_t1], [path_t2])
 self.assertTrue(filecmp.cmp(path1, path2))
 
+  def test_create_mkdirs_open(self):
+path = os.path.join(self.tmpdir, 't1/t2/t3')
+with self.fs.create(path) as f:
+  f.write("yay")
+
+with self.fs.open(path) as f:
+  self.assertEqual(f.read(), "yay")
+
+  def test_open_error(self):
+path = os.path.join(self.tmpdir, 't1')
+with self.assertRaisesRegexp(IOError, r'No such file or directory'):
+  self.fs.open(path)
+
   def test_rename(self):
 path1 = os.path.join(self.tmpdir, 'f1')
 path2 = os.path.join(self.tmpdir, 'f2')
@@ -244,6 +257,22 @@ class LocalFileSystemTest(unittest.TestCase):
 self.assertTrue(self.fs.exists(path2))
 self.assertFalse(self.fs.exists(path1))
 
+  def test_rename_mkdirs(self):
+path_t1 = os.path.join(self.tmpdir, 't1')
+path_t2 = os.path.join(self.tmpdir, 't2/t3/t4')
+self.fs.mkdirs(path_t1)
+
+path1 = os.path.join(path_t1, 'f1')
+path2 = os.path.join(path_t2, 'f1')
+with open(path1, 'a') as f:
+  f.write('Hello')
+
+self.fs.rename([path_t1], [path_t2])
+self.assertTrue(self.fs.exists(path_t2))
+self.assertFalse(self.fs.exists(path_t1))
+self.assertTrue(self.fs.exists(path2))
+self.assertFalse(self.fs.exists(path1))
+
   def test_exists(self):
 path1 = os.path.join(self.tmpdir, 'f1')
 path2 = os.path.join(self.tmpdir, 'f2')

[jira] [Work logged] (BEAM-5039) python postcommit broken in call to WriteToPubSub

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5039?focusedWorklogId=128345=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128345
 ]

ASF GitHub Bot logged work on BEAM-5039:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:29
Start Date: 27/Jul/18 23:29
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #6085: [BEAM-5039] Fix 
streaming_wordcount_it_test
URL: https://github.com/apache/beam/pull/6085#issuecomment-408562906
 
 
   R: @charlesccychen 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128345)
Time Spent: 40m  (was: 0.5h)

> python postcommit broken in call to WriteToPubSub
> -
>
> Key: BEAM-5039
> URL: https://issues.apache.org/jira/browse/BEAM-5039
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> ERROR: test_streaming_wordcount_it 
> (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py",
>  line 105, in test_streaming_wordcount_it
> self.test_pipeline.get_full_options_as_args(**extra_opts))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/streaming_wordcount.py",
>  line 90, in run
> output | beam.io.WriteToPubSub(known_args.output_topic)
> TypeError: __init__() takes at least 3 arguments (2 given)
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/5597/consoleText



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-4788) Test Auto JIRA subtask456

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou closed BEAM-4788.
---
Resolution: Fixed

> Test Auto JIRA subtask456
> -
>
> Key: BEAM-4788
> URL: https://issues.apache.org/jira/browse/BEAM-4788
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
> Fix For: Not applicable
>
>
> updated couple feilds of this issue by using jira api



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5039) python postcommit broken in call to WriteToPubSub

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5039?focusedWorklogId=128344=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128344
 ]

ASF GitHub Bot logged work on BEAM-5039:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:27
Start Date: 27/Jul/18 23:27
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #6085: [BEAM-5039] Make 
with_attributes kwarg optional.
URL: https://github.com/apache/beam/pull/6085#issuecomment-408562684
 
 
   run python postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128344)
Time Spent: 0.5h  (was: 20m)

> python postcommit broken in call to WriteToPubSub
> -
>
> Key: BEAM-5039
> URL: https://issues.apache.org/jira/browse/BEAM-5039
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> ERROR: test_streaming_wordcount_it 
> (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT)
> --
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py",
>  line 105, in test_streaming_wordcount_it
> self.test_pipeline.get_full_options_as_args(**extra_opts))
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/streaming_wordcount.py",
>  line 90, in run
> output | beam.io.WriteToPubSub(known_args.output_topic)
> TypeError: __init__() takes at least 3 arguments (2 given)
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/5597/consoleText



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5040) BigQueryIO retries infinitely in WriteTable and WriteRename

[Reuven Lax (JIRA)]

Reuven Lax created BEAM-5040:


 Summary: BigQueryIO retries infinitely in WriteTable and 
WriteRename
 Key: BEAM-5040
 URL: https://issues.apache.org/jira/browse/BEAM-5040
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp
Affects Versions: 2.5.0
Reporter: Reuven Lax
Assignee: Reuven Lax


BigQueryIO retries infinitely in WriteTable and WriteRename

Several failure scenarios with the current code:
 # It's possible for a load job to return failure even though it actually 
succeeded (e.g. the reply might have timed out). In this case, BigQueryIO will 
retry the job which will fail again (because the job id has already been used), 
leading to indefinite retries. Correct behavior is to stop retrying as the load 
job has succeeded.
 # It's possible for a load job to be accepted by BigQuery, but then to fail on 
the BigQuery side. In this case a retry with the same job id will fail as that 
job id has already been used. BigQueryIO will sometimes detect this, but if the 
worker has restarted it will instead issue a load with the old job id and go 
into a retry loop. Correct behavior is to generate a new deterministic job id 
and retry using that new job id.
 # In many cases of worker restart, BigQueryIO ends up in infinite retry loops.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128334=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128334
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408560506
 
 
   Failed tests seem not relevant to this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128334)
Time Spent: 10h 10m  (was: 10h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 10h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128340
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408560576
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128340)
Time Spent: 11h 10m  (was: 11h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 11h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128342=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128342
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408550492
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128342)
Time Spent: 11.5h  (was: 11h 20m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 11.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128337=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128337
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408560576
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128337)
Time Spent: 10h 40m  (was: 10.5h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 10h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128335
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408560530
 
 
   run java precommit
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128335)
Time Spent: 10h 20m  (was: 10h 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 10h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128339=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128339
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408560569
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128339)
Time Spent: 11h  (was: 10h 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 11h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128338
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408560530
 
 
   run java precommit
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128338)
Time Spent: 10h 50m  (was: 10h 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 10h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128341=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128341
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-40801
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128341)
Time Spent: 11h 20m  (was: 11h 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 11h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128336
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:10
Start Date: 27/Jul/18 23:10
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408560569
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128336)
Time Spent: 10.5h  (was: 10h 20m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3098) Upgrade Java grpc version

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3098?focusedWorklogId=128333=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128333
 ]

ASF GitHub Bot logged work on BEAM-3098:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:09
Start Date: 27/Jul/18 23:09
Worklog Time Spent: 10m 
  Work Description: garrettjonesgoogle commented on issue #5767: 
[BEAM-3098] Upgrades gRPC and protobuf dependencies
URL: https://github.com/apache/beam/pull/5767#issuecomment-408560399
 
 
   This PR is obsolete - the upgrades were completed in 
https://github.com/apache/beam/pull/5988 .


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128333)
Time Spent: 1h 10m  (was: 1h)

> Upgrade Java grpc version
> -
>
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4833) Add support for users specifying a requirements.txt for their Python portable container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4833?focusedWorklogId=128332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128332
 ]

ASF GitHub Bot logged work on BEAM-4833:


Author: ASF GitHub Bot
Created on: 27/Jul/18 23:00
Start Date: 27/Jul/18 23:00
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6005: [BEAM-4833] Add 
support for user req.txt for portable python
URL: https://github.com/apache/beam/pull/6005#issuecomment-408559095
 
 
   and @charlesccychen 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128332)
Time Spent: 1h 20m  (was: 1h 10m)

> Add support for users specifying a requirements.txt for their Python portable 
> container
> ---
>
> Key: BEAM-4833
> URL: https://issues.apache.org/jira/browse/BEAM-4833
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: holdenk
>Assignee: holdenk
>Priority: Minor
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> It's pretty common that Python scripts require extra dependencies, even the 
> tensorflow model analysis TFMA example requires a different version of TF 
> than the one we install by default. While users can roll their own container 
> or edit the Dockerfile, it would probably be useful to provide an easier path 
> to integrating their dependencies.
> While we support automatically installing the dependencies at runtime on the 
> workers, this can be very slow, especially for things like tensorflow, arrow, 
> or other numeric heavy code.
> Another alternative could be a simple script to augment the existing base 
> image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4864) run_validatescontainer.sh in Python sdk has hard-coded bucket of sadness

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4864?focusedWorklogId=128331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128331
 ]

ASF GitHub Bot logged work on BEAM-4864:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:58
Start Date: 27/Jul/18 22:58
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6061: [BEAM-4864] Make 
run_validatescontainer.sh not quite as hard coded.
URL: https://github.com/apache/beam/pull/6061#issuecomment-408558928
 
 
   cc @alanmyrvold the original author (I think) of the script


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128331)
Time Spent: 0.5h  (was: 20m)

> run_validatescontainer.sh in Python sdk has hard-coded bucket of sadness
> 
>
> Key: BEAM-4864
> URL: https://issues.apache.org/jira/browse/BEAM-4864
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: holdenk
>Assignee: holdenk
>Priority: Trivial
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The run_validatescontainer.sh script looks amazing! However I could not 
> validate my container, and this made me sad. We can make it configurable and 
> then people can validate their container changes more easily :)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5012) Beam Dependency Update Request: org.springframework:spring-expression 5.0.7.RELEASE

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5012?focusedWorklogId=128330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128330
 ]

ASF GitHub Bot logged work on BEAM-5012:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:57
Start Date: 27/Jul/18 22:57
Worklog Time Spent: 10m 
  Work Description: yifanzou opened a new pull request #6091: [BEAM-5012] 
upgrade spring-expression
URL: https://github.com/apache/beam/pull/6091
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128330)
Time Spent: 10m
Remaining Estimate: 0h

> Beam Dependency Update Request: org.springframework:spring-expression 
> 5.0.7.RELEASE
> ---
>
> Key: BEAM-5012
> URL: https://issues.apache.org/jira/browse/BEAM-5012
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 

[jira] [Assigned] (BEAM-5012) Beam Dependency Update Request: org.springframework:spring-expression 5.0.7.RELEASE

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou reassigned BEAM-5012:
---

Assignee: yifan zou

> Beam Dependency Update Request: org.springframework:spring-expression 
> 5.0.7.RELEASE
> ---
>
> Key: BEAM-5012
> URL: https://issues.apache.org/jira/browse/BEAM-5012
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>
> 2018-07-25 20:34:53.257037
> Please review and upgrade the org.springframework:spring-expression 
> to the latest version 5.0.7.RELEASE 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128329
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:54
Start Date: 27/Jul/18 22:54
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408558312
 
 
   @apilloud @akedin I addressed comments. Can you take another look?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128329)
Time Spent: 10h  (was: 9h 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 10h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3098) Upgrade Java grpc version

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3098?focusedWorklogId=128328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128328
 ]

ASF GitHub Bot logged work on BEAM-3098:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:50
Start Date: 27/Jul/18 22:50
Worklog Time Spent: 10m 
  Work Description: nithinsujir removed a comment on issue #5767: 
[BEAM-3098] Upgrades gRPC and protobuf dependencies
URL: https://github.com/apache/beam/pull/5767#issuecomment-408557494
 
 
   @chamikaramj Can you point me to the spanner test failure logs?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128328)
Time Spent: 1h  (was: 50m)

> Upgrade Java grpc version
> -
>
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3098) Upgrade Java grpc version

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3098?focusedWorklogId=128327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128327
 ]

ASF GitHub Bot logged work on BEAM-3098:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:49
Start Date: 27/Jul/18 22:49
Worklog Time Spent: 10m 
  Work Description: nithinsujir commented on issue #5767: [BEAM-3098] 
Upgrades gRPC and protobuf dependencies
URL: https://github.com/apache/beam/pull/5767#issuecomment-408557667
 
 
   @chamikaramj Can you point me to the spanner test failure logs please?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128327)
Time Spent: 50m  (was: 40m)

> Upgrade Java grpc version
> -
>
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3098) Upgrade Java grpc version

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3098?focusedWorklogId=128326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128326
 ]

ASF GitHub Bot logged work on BEAM-3098:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:48
Start Date: 27/Jul/18 22:48
Worklog Time Spent: 10m 
  Work Description: nithinsujir commented on issue #5767: [BEAM-3098] 
Upgrades gRPC and protobuf dependencies
URL: https://github.com/apache/beam/pull/5767#issuecomment-408557494
 
 
   @chamikaramj Can you point me to the spanner test failure logs?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128326)
Time Spent: 40m  (was: 0.5h)

> Upgrade Java grpc version
> -
>
> Key: BEAM-3098
> URL: https://issues.apache.org/jira/browse/BEAM-3098
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Solomon Duskis
>Assignee: Garrett Jones
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Beam Java currently depends on grpc 1.2, which was released in March.  It 
> would be great if the dependency could be update to something newer, like 
> grpc 1.7.0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4833) Add support for users specifying a requirements.txt for their Python portable container

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4833?focusedWorklogId=128325=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128325
 ]

ASF GitHub Bot logged work on BEAM-4833:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:46
Start Date: 27/Jul/18 22:46
Worklog Time Spent: 10m 
  Work Description: holdenk commented on issue #6005: [BEAM-4833] Add 
support for user req.txt for portable python
URL: https://github.com/apache/beam/pull/6005#issuecomment-408557171
 
 
   Gentle ping @aaltay ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128325)
Time Spent: 1h 10m  (was: 1h)

> Add support for users specifying a requirements.txt for their Python portable 
> container
> ---
>
> Key: BEAM-4833
> URL: https://issues.apache.org/jira/browse/BEAM-4833
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: holdenk
>Assignee: holdenk
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> It's pretty common that Python scripts require extra dependencies, even the 
> tensorflow model analysis TFMA example requires a different version of TF 
> than the one we install by default. While users can roll their own container 
> or edit the Dockerfile, it would probably be useful to provide an easier path 
> to integrating their dependencies.
> While we support automatically installing the dependencies at runtime on the 
> workers, this can be very slow, especially for things like tensorflow, arrow, 
> or other numeric heavy code.
> Another alternative could be a simple script to augment the existing base 
> image.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128323
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:35
Start Date: 27/Jul/18 22:35
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-40801
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128323)
Time Spent: 9h 50m  (was: 9h 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 9h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128321
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:35
Start Date: 27/Jul/18 22:35
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408555464
 
 
   org.apache.beam.sdk.io.mqtt.MqttIOTest.testRead flakes


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128321)
Time Spent: 9h 40m  (was: 9.5h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 9h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=128320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128320
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:34
Start Date: 27/Jul/18 22:34
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r205914987
 
 

 ##
 File path: sdks/python/apache_beam/transforms/window.py
 ##
 @@ -348,6 +388,9 @@ def __eq__(self, other):
 if type(self) == type(other) == FixedWindows:
   return self.size == other.size and self.offset == other.offset
 
+  def __hash__(self):
+return hash((type(self), self.size, self.offset))
+
   def __ne__(self, other):
 
 Review comment:
   Thanks! I have just pushed the commit with these changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128320)
Time Spent: 12h 50m  (was: 12h 40m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 12h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3906) Get Python Wheel Validation Automated

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3906?focusedWorklogId=128319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128319
 ]

ASF GitHub Bot logged work on BEAM-3906:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:34
Start Date: 27/Jul/18 22:34
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4943: [BEAM-3906] Automate 
Validation Aganist Python Wheel
URL: https://github.com/apache/beam/pull/4943#issuecomment-408555184
 
 
   @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128319)
Time Spent: 27.5h  (was: 27h 20m)

> Get Python Wheel Validation Automated
> -
>
> Key: BEAM-3906
> URL: https://issues.apache.org/jira/browse/BEAM-3906
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-python, testing
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 27.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=128318=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128318
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:32
Start Date: 27/Jul/18 22:32
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r205914607
 
 

 ##
 File path: sdks/python/apache_beam/transforms/window.py
 ##
 @@ -348,6 +388,9 @@ def __eq__(self, other):
 if type(self) == type(other) == FixedWindows:
   return self.size == other.size and self.offset == other.offset
 
+  def __hash__(self):
+return hash((type(self), self.size, self.offset))
+
   def __ne__(self, other):
 
 Review comment:
   Agree with you, looking at the test it seems that we are doing the right 
thing. I think there will not be performance impact here, but I'll do one final 
A/B test with and without the PR to be safe once we finalize it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128318)
Time Spent: 12h 40m  (was: 12.5h)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 12h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=128317=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128317
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:25
Start Date: 27/Jul/18 22:25
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r205913757
 
 

 ##
 File path: sdks/python/apache_beam/transforms/window.py
 ##
 @@ -348,6 +388,9 @@ def __eq__(self, other):
 if type(self) == type(other) == FixedWindows:
   return self.size == other.size and self.offset == other.offset
 
+  def __hash__(self):
+return hash((type(self), self.size, self.offset))
+
   def __ne__(self, other):
 
 Review comment:
   Adding 
   ```
def __ne__(self, other):
  return not self == other
   ```
   for `IntervalWindow` causes `test_global_window` to fail on the last 
assertion (comparing IntervalWindow (max-range) to GlobalWindow). To resolve 
this failed assertion I need to add type in `__eq__`:
   
   ```
 def __hash__(self):
   return hash((self.start, self.end, type(self)))
   
 def __eq__(self, other):
   return (self.start == other.start
   and self.end == other.end)
   and type(self) == type(other))
   
 def __ne__(self, other):
   return not self == other
   ```
   I think this makes sense and will be necessary for Python3 compatibility, 
however I'm not sure if this will have performance impact here?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128317)
Time Spent: 12.5h  (was: 12h 20m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 12.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4225) Integrate Nexmark with perfkit dashboards

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4225?focusedWorklogId=128316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128316
 ]

ASF GitHub Bot logged work on BEAM-4225:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:25
Start Date: 27/Jul/18 22:25
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on issue #4976: [BEAM-4225] Add 
Nexmark PostCommit runs for spark, flink and direct runner and export to 
Bigquery
URL: https://github.com/apache/beam/pull/4976#issuecomment-408553746
 
 
   Hi everyone,
   After reading this thread, I'm still confused: what is the specific reason 
behind disabling phrase triggering?
   I think that in our current state of world we need to have an option to run 
post-commits for PR. This feature is still utilized to verify that specific 
post-commit is fixed or to test big changes.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128316)
Time Spent: 11h  (was: 10h 50m)

> Integrate Nexmark with perfkit dashboards
> -
>
> Key: BEAM-4225
> URL: https://issues.apache.org/jira/browse/BEAM-4225
> Project: Beam
>  Issue Type: Improvement
>  Components: examples-nexmark
>Reporter: Etienne Chauchot
>Assignee: Etienne Chauchot
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 11h
>  Remaining Estimate: 0h
>
> The aim is to run Nexmark as post-commits and export the results to the 
> performance dashboards:
> see the threads:
> [https://lists.apache.org/thread.html/9f8fe1c6df7d8bfe2697332e69722ca4edd2810adc6a914cdf32da29@%3Cdev.beam.apache.org%3E]
> https://lists.apache.org/thread.html/701196efd6e74b7715785d43019a4a73e8a093997f59662fdadf8f2a@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.6.0 updated: Changed hardcoded version number into env variable. Added more comments.

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch release-2.6.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.6.0 by this push:
 new c5d0dbf  Changed hardcoded version number into env variable. Added 
more comments.
c5d0dbf is described below

commit c5d0dbfd045ec3798c532bec6556d5225ce27b76
Author: Boyuan Zhang 
AuthorDate: Wed Jul 25 17:26:05 2018 -0700

Changed hardcoded version number into env variable. Added more comments.

(cherry picked from commit 28f193b in master)
---
 sdks/java/javadoc/build.gradle | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/sdks/java/javadoc/build.gradle b/sdks/java/javadoc/build.gradle
index e6f7c4d..240aef4 100644
--- a/sdks/java/javadoc/build.gradle
+++ b/sdks/java/javadoc/build.gradle
@@ -16,6 +16,13 @@
  * limitations under the License.
  */
 
+/*
+ * Aggregate Javadoc is not published to Maven Central. Each Java module that 
is published
+ * to Maven Cental packages its own Javadoc. To generate aggregated javadocs, 
run:
+ *   ./gradlew :beam-sdks-java-javadoc:aggregateJavadoc
+ * Generated files will be located under 
beam/sdks/java/javadoc/build/docs/javadoc and are
+ * used as part of the beam-site source tree.
+ */
 description = "Apache Beam :: SDKs :: Java :: Aggregated Javadoc"
 apply plugin: 'java'
 
@@ -91,7 +98,7 @@ task aggregateJavadoc(type: Javadoc) {
 
   options.with {
 failOnError false
-title "Apache Beam 2.5.0"
+title "Apache Beam " + project.version
 overview 'overview.html'
 addStringOption('Xdoclint:all', '-quiet')
 addStringOption('Xdoclint:-missing', '-quiet')

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128315=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128315
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:06
Start Date: 27/Jul/18 22:06
Worklog Time Spent: 10m 
  Work Description: amaliujia removed a comment on issue #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408522973
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128315)
Time Spent: 9.5h  (was: 9h 20m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 9.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128314=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128314
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 22:06
Start Date: 27/Jul/18 22:06
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408550492
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128314)
Time Spent: 9h 20m  (was: 9h 10m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 9h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4862) varint overflow -62135596800 exception with Cloud Spanner Timestamp 0001-01-01T00:00:00Z

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4862?focusedWorklogId=128306=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128306
 ]

ASF GitHub Bot logged work on BEAM-4862:


Author: ASF GitHub Bot
Created on: 27/Jul/18 21:34
Start Date: 27/Jul/18 21:34
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #6077: 
BEAM-4862 Fixes bug in Spanner's MutationGroupEncoder by converting timestamps 
into Long and not Int.
URL: https://github.com/apache/beam/pull/6077#discussion_r205905009
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoderTest.java
 ##
 @@ -528,6 +528,32 @@ public void dateKeys() throws Exception {
 verifyEncodedOrdering(schema, "test", keys);
   }
 
+  @Test
+  // Test for Jira issue BEAM-4862
 
 Review comment:
   I believe this comment becomes irrelevant once the issue is fixed. If we 
want to keep it, it would be better to put it into the code, because then it 
will comment on the reason why the code was implemented one way, not the other.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128306)

> varint overflow -62135596800 exception with Cloud Spanner Timestamp 
> 0001-01-01T00:00:00Z
> 
>
> Key: BEAM-4862
> URL: https://issues.apache.org/jira/browse/BEAM-4862
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Eric Beach
>Assignee: Chamikara Jayalath
>Priority: Minor
>   Original Estimate: 24h
>  Time Spent: 0.5h
>  Remaining Estimate: 23.5h
>
> tl;dr - If you try to write a Timestamp of value "0001-01-01T00:00:00Z" as a 
> Spanner Mutation, you get an overflow error.
>  
> The crux of the issue appears to be that 0001-01-01T00:00:00Z, which is a 
> valid Timestamp per 
> [https://cloud.google.com/spanner/docs/data-types#timestamp-type], is too 
> large for an integer. See the two lines of code below. 
> [https://github.com/apache/beam/blob/release-2.5.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoder.java#L453]
> [https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java#L58]
>  
>  
> Stack Trade
> {{Caused by: java.io.IOException: varint overflow -62135596800 at 
> org.apache.beam.sdk.util.VarInt.decodeInt(VarInt.java:65) at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodePrimitive(MutationGroupEncoder.java:453)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeModification(MutationGroupEncoder.java:326)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeMutation(MutationGroupEncoder.java:280)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode(MutationGroupEncoder.java:264)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement(SpannerIO.java:1030)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn$DoFnInvoker.invokeProcessElement(Unknown
>  Source) at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
>  at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:146)
>  at 
> com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:181)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:124)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:53)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:115)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
>  at 
> 

[jira] [Work logged] (BEAM-4862) varint overflow -62135596800 exception with Cloud Spanner Timestamp 0001-01-01T00:00:00Z

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4862?focusedWorklogId=128305=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128305
 ]

ASF GitHub Bot logged work on BEAM-4862:


Author: ASF GitHub Bot
Created on: 27/Jul/18 21:34
Start Date: 27/Jul/18 21:34
Worklog Time Spent: 10m 
  Work Description: Ardagan commented on a change in pull request #6077: 
BEAM-4862 Fixes bug in Spanner's MutationGroupEncoder by converting timestamps 
into Long and not Int.
URL: https://github.com/apache/beam/pull/6077#discussion_r205904597
 
 

 ##
 File path: 
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoderTest.java
 ##
 @@ -528,6 +528,32 @@ public void dateKeys() throws Exception {
 verifyEncodedOrdering(schema, "test", keys);
   }
 
+  @Test
+  // Test for Jira issue BEAM-4862
+  public void decodeTimestampMutationGroup() {
 
 Review comment:
   Please, split this test into two:
   decodeTimestampCanDecodeTimestampMinValue and 
decodeTimestampCanDecodeTimestampMaxValue.
   
   These names will give detailed description on what this test does. Splitting 
will allow to easily understand failing setup by reading test name only.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128305)
Time Spent: 0.5h  (was: 20m)
Remaining Estimate: 23.5h  (was: 23h 40m)

> varint overflow -62135596800 exception with Cloud Spanner Timestamp 
> 0001-01-01T00:00:00Z
> 
>
> Key: BEAM-4862
> URL: https://issues.apache.org/jira/browse/BEAM-4862
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.5.0
>Reporter: Eric Beach
>Assignee: Chamikara Jayalath
>Priority: Minor
>   Original Estimate: 24h
>  Time Spent: 0.5h
>  Remaining Estimate: 23.5h
>
> tl;dr - If you try to write a Timestamp of value "0001-01-01T00:00:00Z" as a 
> Spanner Mutation, you get an overflow error.
>  
> The crux of the issue appears to be that 0001-01-01T00:00:00Z, which is a 
> valid Timestamp per 
> [https://cloud.google.com/spanner/docs/data-types#timestamp-type], is too 
> large for an integer. See the two lines of code below. 
> [https://github.com/apache/beam/blob/release-2.5.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/MutationGroupEncoder.java#L453]
> [https://github.com/apache/beam/blob/279a05604b83a54e8e5a79e13d8761f94841f326/sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java#L58]
>  
>  
> Stack Trade
> {{Caused by: java.io.IOException: varint overflow -62135596800 at 
> org.apache.beam.sdk.util.VarInt.decodeInt(VarInt.java:65) at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodePrimitive(MutationGroupEncoder.java:453)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeModification(MutationGroupEncoder.java:326)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeMutation(MutationGroupEncoder.java:280)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode(MutationGroupEncoder.java:264)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement(SpannerIO.java:1030)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn$DoFnInvoker.invokeProcessElement(Unknown
>  Source) at 
> org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
>  at 
> org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:146)
>  at 
> com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
>  at 
> com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:181)
>  at 
> com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:124)
>  at 
> com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:53)
>  at 
> 

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 6d7fd1a3fe225a73e08c32f6ec15e78868252913
Author: Mergebot 
AuthorDate: Fri Jul 27 21:29:05 2018 +

Prepare repository for deployment.
---
 content/contribute/become-a-committer/index.html   |   6 +
 content/contribute/committer-guide/index.html  |   6 +
 content/contribute/dependencies/index.html |   6 +
 content/contribute/design-documents/index.html |   6 +
 content/contribute/docker-images/index.html|   6 +
 content/contribute/eclipse/index.html  |   6 +
 content/contribute/index.html  |   6 +
 content/contribute/intellij/index.html |   6 +
 content/contribute/portability/index.html  |   6 +
 .../index.html | 151 ++---
 .../index.html | 122 +
 .../{eclipse => postcommits-policies}/index.html   | 151 ++---
 .../contribute/ptransform-style-guide/index.html   |   6 +
 content/contribute/release-guide/index.html|   6 +
 content/contribute/runner-guide/index.html |   6 +
 content/contribute/testing/index.html  | 105 ++
 .../contribute/website-contributions/index.html|   6 +
 17 files changed, 389 insertions(+), 218 deletions(-)

diff --git a/content/contribute/become-a-committer/index.html 
b/content/contribute/become-a-committer/index.html
index 2a9bbc0..1ccffc9 100644
--- a/content/contribute/become-a-committer/index.html
+++ b/content/contribute/become-a-committer/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/committer-guide/index.html 
b/content/contribute/committer-guide/index.html
index 72770da..58e5ed5 100644
--- a/content/contribute/committer-guide/index.html
+++ b/content/contribute/committer-guide/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/dependencies/index.html 
b/content/contribute/dependencies/index.html
index 57c2c13..a4a306f 100644
--- a/content/contribute/dependencies/index.html
+++ b/content/contribute/dependencies/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/design-documents/index.html 
b/content/contribute/design-documents/index.html
index eee9c3b..93e2fab 100644
--- a/content/contribute/design-documents/index.html
+++ b/content/contribute/design-documents/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/docker-images/index.html 
b/content/contribute/docker-images/index.html
index ec15c74..dc3c803 100644
--- a/content/contribute/docker-images/index.html
+++ b/content/contribute/docker-images/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/eclipse/index.html 
b/content/contribute/eclipse/index.html
index 5545811..6e26d29 100644
--- a/content/contribute/eclipse/index.html
+++ b/content/contribute/eclipse/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/index.html b/content/contribute/index.html
index 580062e..7a5ad79 100644
--- a/content/contribute/index.html
+++ b/content/contribute/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/intellij/index.html 
b/content/contribute/intellij/index.html
index 449b20f..f35bbfa 100644
--- a/content/contribute/intellij/index.html
+++ b/content/contribute/intellij/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/portability/index.html 
b/content/contribute/portability/index.html
index f0f247d..09d0e57 100644
--- a/content/contribute/portability/index.html
+++ b/content/contribute/portability/index.html
@@ -160,6 +160,12 @@
   
 
 
+  Policies
+  
+Post-commit tests 
policies
+  
+
+
   Committers
 
   
diff --git a/content/contribute/website-contributions/index.html 
b/content/contribute/postcommits-guides/index.html
similarity index 79%
copy from content/contribute/website-contributions/index.html
copy to content/contribute/postcommits-guides/index.html
index 7c52811..be19269 100644
--- a/content/contribute/website-contributions/index.html
+++ b/content/contribute/postcommits-guides/index.html
@@ -28,7 +28,7 @@
   
   
   
-  Beam Website Contributions
+  Post-commit tests processes guides
   
   

[beam-site] branch mergebot updated (63b7f4a -> 67a2c16)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 63b7f4a  This closes #511
 new 67a2c16  This closes #511

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (63b7f4a)
\
 N -- N -- N   refs/heads/mergebot (67a2c16)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

[beam-site] 01/01: This closes #511

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 67a2c16eefcd6ed76f26a946702e91e619acd9fb
Merge: 8a7108d 491c456
Author: Mergebot 
AuthorDate: Fri Jul 27 21:26:02 2018 +

This closes #511

 src/_includes/section-menu/contribute.html |  6 ++
 src/contribute/postcommits-guides.md   | 74 
 src/contribute/postcommits-policies-details.md | 93 ++
 src/contribute/postcommits-policies.md | 82 +++
 src/contribute/testing.md  | 81 ++
 5 files changed, 336 insertions(+)

[jira] [Commented] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

[Mikhail Gryzykhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16560340#comment-16560340
 ] 

Mikhail Gryzykhin commented on BEAM-4686:
-

I've synced with Jason.

Seems that the easiest way would be to add retries.

I'll also check if there is any other source of keys available.

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: flake
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4686) Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin reassigned BEAM-4686:
---

Assignee: Mikhail Gryzykhin

> Build Failure: gpg: keyserver receive failed: unknown pubkey algorithm
> --
>
> Key: BEAM-4686
> URL: https://issues.apache.org/jira/browse/BEAM-4686
> Project: Beam
>  Issue Type: Bug
>  Components: website
>Affects Versions: 2.6.0
>Reporter: Andrew Pilloud
>Assignee: Mikhail Gryzykhin
>Priority: Major
>  Labels: flake
>
> I see this on a large percentage of builds:
>  
> https://builds.apache.org/job/beam_PreCommit_Website_Stage/1145/console
> {code:java}
> [beam_PreCommit_Website_Stage] $ /bin/bash -xe 
> /tmp/jenkins6730031359263725887.sh
> + gpg --keyserver hkp://keys.gnupg.net --recv-keys 
> 409B6B1796C275462A1703113804BB82D39DC0E3 
> 7D2BAF1CF37B13E2069D6956105BD0E739499BDB
> gpg: requesting key D39DC0E3 from hkp server keys.gnupg.net
> gpg: requesting key 39499BDB from hkp server keys.gnupg.net
> gpgkeys: key 409B6B1796C275462A1703113804BB82D39DC0E3 can't be retrieved
> gpgkeys: key 7D2BAF1CF37B13E2069D6956105BD0E739499BDB can't be retrieved
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
> gpg: keyserver communications error: keyserver helper general error
> gpg: keyserver communications error: unknown pubkey algorithm
> gpg: keyserver receive failed: unknown pubkey algorithm
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch mergebot updated (5580b87 -> 63b7f4a)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 5580b87  This closes #511
 new 63b7f4a  This closes #511

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (5580b87)
\
 N -- N -- N   refs/heads/mergebot (63b7f4a)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

[beam-site] 01/01: This closes #511

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 63b7f4a35ea4b0f2c1197c8d480fa5d4a7a52ae5
Merge: 8a7108d 491c456
Author: Mergebot 
AuthorDate: Fri Jul 27 21:04:01 2018 +

This closes #511

 src/_includes/section-menu/contribute.html |  6 ++
 src/contribute/postcommits-guides.md   | 74 
 src/contribute/postcommits-policies-details.md | 93 ++
 src/contribute/postcommits-policies.md | 82 +++
 src/contribute/testing.md  | 81 ++
 5 files changed, 336 insertions(+)

[jira] [Work logged] (BEAM-5017) Beam Dependency Update Request: org.elasticsearch.plugin:transport-netty4-client 6.3.2

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5017?focusedWorklogId=128289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128289
 ]

ASF GitHub Bot logged work on BEAM-5017:


Author: ASF GitHub Bot
Created on: 27/Jul/18 20:54
Start Date: 27/Jul/18 20:54
Worklog Time Spent: 10m 
  Work Description: yifanzou closed pull request #6087: Do Not Merge. 
[BEAM-5017] upgrade elasticsearch dependnecies to 6.3.2
URL: https://github.com/apache/beam/pull/6087
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle 
b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
index c24add29437..a3011abfa46 100644
--- a/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
+++ b/sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle
@@ -31,7 +31,7 @@ test {
 
 def jna_version = "4.1.0"
 def log4j_version = "2.6.2"
-def elastic_search_version = "5.6.3"
+def elastic_search_version = "6.3.2"
 
 configurations.all {
   resolutionStrategy {
@@ -48,7 +48,10 @@ dependencies {
   testCompile 
"org.elasticsearch.plugin:transport-netty4-client:$elastic_search_version"
   testCompile 
"com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.5.0"
   testCompile "org.elasticsearch:elasticsearch:$elastic_search_version"
-
+  testCompile ('junit:junit:4.12') {
+exclude module : 'hamcrest'
+exclude module : 'hamcrest-core'
+  }
   testCompile project(path: ":beam-sdks-java-core", configuration: "shadow")
   testCompile project(path: ":beam-sdks-java-io-elasticsearch", configuration: 
"shadow")
   testCompile project(path: ":beam-sdks-java-io-common", configuration: 
"shadowTest")


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128289)
Time Spent: 20m  (was: 10m)

> Beam Dependency Update Request: 
> org.elasticsearch.plugin:transport-netty4-client 6.3.2
> --
>
> Key: BEAM-5017
> URL: https://issues.apache.org/jira/browse/BEAM-5017
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> 2018-07-25 20:35:24.396372
> Please review and upgrade the 
> org.elasticsearch.plugin:transport-netty4-client to the latest version 6.3.2 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5019) Beam Dependency Update Request: org.tukaani:xz 1.8

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou reassigned BEAM-5019:
---

Assignee: yifan zou

> Beam Dependency Update Request: org.tukaani:xz 1.8
> --
>
> Key: BEAM-5019
> URL: https://issues.apache.org/jira/browse/BEAM-5019
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>
> 2018-07-25 20:35:35.370731
> Please review and upgrade the org.tukaani:xz to the latest version 
> 1.8 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5017) Beam Dependency Update Request: org.elasticsearch.plugin:transport-netty4-client 6.3.2

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou reassigned BEAM-5017:
---

Assignee: (was: yifan zou)

> Beam Dependency Update Request: 
> org.elasticsearch.plugin:transport-netty4-client 6.3.2
> --
>
> Key: BEAM-5017
> URL: https://issues.apache.org/jira/browse/BEAM-5017
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 2018-07-25 20:35:24.396372
> Please review and upgrade the 
> org.elasticsearch.plugin:transport-netty4-client to the latest version 6.3.2 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128273
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 19:55
Start Date: 27/Jul/18 19:55
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6006: [BEAM-4808][SQL] 
add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#issuecomment-408522973
 
 
   run java postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128273)
Time Spent: 9h 10m  (was: 9h)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 9h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128272=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128272
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 19:54
Start Date: 27/Jul/18 19:54
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#discussion_r205882691
 
 

 ##
 File path: 
sdks/java/extensions/sql/jdbc/src/test/java/org/apache/beam/sdk/extensions/sql/jdbc/BeamSqlLineIT.java
 ##
 @@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql.jdbc;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static 
org.apache.beam.sdk.extensions.sql.jdbc.BeamSqlLineTestingUtils.buildArgs;
+import static 
org.apache.beam.sdk.extensions.sql.jdbc.BeamSqlLineTestingUtils.toLines;
+import static org.hamcrest.CoreMatchers.everyItem;
+import static org.junit.Assert.assertThat;
+import static org.junit.Assert.assertTrue;
+
+import com.alibaba.fastjson.JSONObject;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import java.io.ByteArrayOutputStream;
+import java.io.Serializable;
+import java.text.ParseException;
+import java.text.SimpleDateFormat;
+import java.util.Arrays;
+import java.util.List;
+import java.util.TimeZone;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage;
+import org.apache.beam.sdk.io.gcp.pubsub.TestPubsub;
+import org.hamcrest.collection.IsIn;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+
+/** BeamSqlLine integration tests. */
+public class BeamSqlLineIT implements Serializable {
+
+  @Rule public transient TestPubsub eventsTopic = TestPubsub.create();
+
+  private static final Integer numberOfThreads = 4;
+  private static String createPubsubTableStatement;
+  private static String readFromPubsub;
+  private static String filterForSouthManhattan;
+  private static String slidingWindowQuery;
+  private static String fixedWindowQuery;
+  private static PubsubMessageJSONStringConstructor constructor;
+  private static ExecutorService pool;
+  private static final String publicTopic = 
"projects/pubsub-public-data/topics/taxirides-realtime";
+
+  @BeforeClass
+  public static void setUp() {
+pool = Executors.newFixedThreadPool(numberOfThreads);
 
 Review comment:
   I instead create a ExecutorService to help run thread.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128272)
Time Spent: 9h  (was: 8h 50m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 9h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4808) Add an integration test for BeamSqlLine

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4808?focusedWorklogId=128271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128271
 ]

ASF GitHub Bot logged work on BEAM-4808:


Author: ASF GitHub Bot
Created on: 27/Jul/18 19:53
Start Date: 27/Jul/18 19:53
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6006: 
[BEAM-4808][SQL] add e2e test for BeamSqlLine.
URL: https://github.com/apache/beam/pull/6006#discussion_r205882308
 
 

 ##
 File path: sdks/java/extensions/sql/jdbc/build.gradle
 ##
 @@ -71,7 +68,35 @@ task shadowJarTest(type: Test, dependsOn: 
":beam-sdks-java-extensions-sql-jdbc:s
   useJUnit { }
 }
 
+task endToEndTest(type: Test) {
+  group = "Verification"
+  def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing'
+  def gcsTempRoot = project.findProperty('gcsTempRoot') ?: 
'gs://temp-storage-for-end-to-end-tests/'
+
+  // Disable Gradle cache (it should not be used because the IT's won't run).
+  outputs.upToDateWhen { false }
+
+  def pipelineOptions = [
+  "--project=${gcpProject}",
+  "--tempLocation=${gcsTempRoot}",
+  "--blockOnRun=false"]
+
+  systemProperty "beamTestPipelineOptions", JsonOutput.toJson(pipelineOptions)
 
 Review comment:
   It actually works for TestPubsub where TestPipeline/TestPipelineOptions is 
invoked. So this configuration should be kept.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128271)
Time Spent: 8h 50m  (was: 8h 40m)

> Add an integration test for BeamSqlLine
> ---
>
> Key: BEAM-4808
> URL: https://issues.apache.org/jira/browse/BEAM-4808
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 8h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-2977) improve unbounded prose in wordcount example

[Melissa Pashniak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Melissa Pashniak resolved BEAM-2977.

   Resolution: Fixed
Fix Version/s: Not applicable

> improve unbounded prose in wordcount example
> 
>
> Key: BEAM-2977
> URL: https://issues.apache.org/jira/browse/BEAM-2977
> Project: Beam
>  Issue Type: Improvement
>  Components: website
>Reporter: Melissa Pashniak
>Assignee: Melissa Pashniak
>Priority: Minor
> Fix For: Not applicable
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> request from tgroh: the way I think of unbounded is that it doesn't make 
> sense to ask "do I have all of the data", just "at what point do I have all 
> of the data up to"
> context:
>  ### Unbounded and bounded pipeline input modes
> ...
>  If your input is continuously updating, then it's considered 'unbounded'.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5017) Beam Dependency Update Request: org.elasticsearch.plugin:transport-netty4-client 6.3.2

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5017?focusedWorklogId=128268=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128268
 ]

ASF GitHub Bot logged work on BEAM-5017:


Author: ASF GitHub Bot
Created on: 27/Jul/18 19:30
Start Date: 27/Jul/18 19:30
Worklog Time Spent: 10m 
  Work Description: yifanzou opened a new pull request #6087: Do Not Merge. 
[BEAM-5017] upgrade elasticsearch dependnecies to 6.3.2
URL: https://github.com/apache/beam/pull/6087
 
 
   Test Ongoing
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128268)
Time Spent: 10m
Remaining Estimate: 0h

> Beam Dependency Update Request: 
> org.elasticsearch.plugin:transport-netty4-client 6.3.2
> --
>
> Key: BEAM-5017
> URL: https://issues.apache.org/jira/browse/BEAM-5017
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 2018-07-25 

[jira] [Assigned] (BEAM-5017) Beam Dependency Update Request: org.elasticsearch.plugin:transport-netty4-client 6.3.2

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou reassigned BEAM-5017:
---

Assignee: yifan zou

> Beam Dependency Update Request: 
> org.elasticsearch.plugin:transport-netty4-client 6.3.2
> --
>
> Key: BEAM-5017
> URL: https://issues.apache.org/jira/browse/BEAM-5017
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Assignee: yifan zou
>Priority: Major
>
> 2018-07-25 20:35:24.396372
> Please review and upgrade the 
> org.elasticsearch.plugin:transport-netty4-client to the latest version 6.3.2 
>  
> cc: 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1109

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
[...truncated 19.94 MB...]
INFO: 2018-07-27T18:49:28.316Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow)
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.368Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as 
view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Extract
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.409Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample keys/GroupByKey/Reify
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.455Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Extract into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.506Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey/Reify into 
SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/GroupByKey+SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues/Partial
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.551Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/GroupByKey+SpannerIO.Write/Write 
mutations to Cloud Spanner/Sample keys/Combine.GroupedValues/Partial into 
SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.598Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Write
 into SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait 
view 
0/Sample.Any/Combine.globally(SampleAny)/Combine.perKey(SampleAny)/GroupByKey/Reify
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.642Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForSize/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.678Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.715Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Keys sample as view/GBKaSVForKeys/Write into 
SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample as 
view/ParMultiDo(ToIsmRecordForMapLike)
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.764Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Flatten.Iterables/FlattenIterables/FlatMap into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map
Jul 27, 2018 6:49:31 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:49:28.814Z: Fusing consumer SpannerIO.Write/Write 
mutations to Cloud Spanner/Wait.OnSignal/To wait view 
0/Sample.Any/Combine.globally(SampleAny)/Values/Values/Map into 
SpannerIO.Write/Write mutations to Cloud Spanner/Wait.OnSignal/To wait view 

[jira] [Work logged] (BEAM-4006) Futurize and fix python 2 compatibility for transforms subpackage

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4006?focusedWorklogId=128254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128254
 ]

ASF GitHub Bot logged work on BEAM-4006:


Author: ASF GitHub Bot
Created on: 27/Jul/18 18:46
Start Date: 27/Jul/18 18:46
Worklog Time Spent: 10m 
  Work Description: Fematich commented on a change in pull request #5729: 
[BEAM-4006] Futurize transforms subpackage
URL: https://github.com/apache/beam/pull/5729#discussion_r205866972
 
 

 ##
 File path: sdks/python/apache_beam/transforms/window.py
 ##
 @@ -191,15 +192,31 @@ def __init__(self, end):
   def max_timestamp(self):
 return self.end.predecessor()
 
-  def __cmp__(self, other):
-# Order first by endpoint, then arbitrarily.
-return cmp(self.end, other.end) or cmp(hash(self), hash(other))
-
   def __eq__(self, other):
 raise NotImplementedError
 
+  def __ne__(self, other):
+return (cmp(self.end, other.end)
+or cmp(hash(self), hash(other))) != 0
+
+  def __lt__(self, other):
+return (cmp(self.end, other.end)
+or cmp(hash(self), hash(other))) < 0
+
+  def __le__(self, other):
+return (cmp(self.end, other.end)
+or cmp(hash(self), hash(other))) <= 0
+
+  def __gt__(self, other):
+return (cmp(self.end, other.end)
+or cmp(hash(self), hash(other))) > 0
+
+  def __ge__(self, other):
+return (cmp(self.end, other.end)
+or cmp(hash(self), hash(other))) >= 0
+
   def __hash__(self):
-return hash(self.end)
+raise NotImplementedError
 
 Review comment:
   OK thanks, and interesting! So the `NotImplementedError` is only relevant 
for the consistency of the `__eq__` but doesn't have an impact for child 
classes. Good to know :-)!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128254)
Time Spent: 12h 20m  (was: 12h 10m)

> Futurize and fix python 2 compatibility for transforms subpackage
> -
>
> Key: BEAM-4006
> URL: https://issues.apache.org/jira/browse/BEAM-4006
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>  Time Spent: 12h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #5600

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
[...truncated 1.08 MB...]
test_repr (apache_beam.typehints.typehints_test.GeneratorHintTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.IterableHintTestCase) 
... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_tuple_compatibility 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_must_be_iterable 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_invalid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_composite_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_type_check_violation_valid_simple_type 
(apache_beam.typehints.typehints_test.IterableHintTestCase) ... ok
test_enforce_kv_type_constraint 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_be_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_param_must_have_length_2 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_getitem_proxy_to_tuple 
(apache_beam.typehints.typehints_test.KVHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_invalid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_composite_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_enforce_list_type_constraint_valid_simple_type 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_constraint_compatibility 
(apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_list_repr (apache_beam.typehints.typehints_test.ListHintTestCase) ... ok
test_getitem_proxy_to_union 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_getitem_sequence_not_allowed 
(apache_beam.typehints.typehints_test.OptionalHintTestCase) ... ok
test_any_return_type_hint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_primitive_type_or_type_constraint 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_must_be_single_return_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_no_kwargs_accepted 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_composite_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_simple_type 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_type_check_violation 
(apache_beam.typehints.typehints_test.ReturnsDecoratorTestCase) ... ok
test_compatibility (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_getitem_invalid_composite_type_param 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_repr (apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_invalid_elem_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_must_be_set 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_composite_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_type_check_valid_elem_simple_type 
(apache_beam.typehints.typehints_test.SetHintTestCase) ... ok
test_any_argument_type_hint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_basic_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_composite_type_assertion 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_invalid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_must_be_primitive_type_or_constraint 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_mix_positional_and_keyword_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_only_positional_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_valid_simple_type_arguments 
(apache_beam.typehints.typehints_test.TakesDecoratorTestCase) ... ok
test_functions_as_regular_generator 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #783

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam6 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8991979640092653176.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins7741295642741186852.sh
+ rm -rf .beam_env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2032406382069599269.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2434343209333789461.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4538308420119320641.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2998329773356895951.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1751055385649105439.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6347502862117888712.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Compressed_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8264543442962929892.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.8.2)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.2.8.2)
Requirement already satisfied: grpcio<2,>=1.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.13.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(2.1.0)

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #796

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam14 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4659338469391254742.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3353695659466200407.sh
+ rm -rf .beam_env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4312093088267153867.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8318384506631148921.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins5757234724840696742.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins8063789274185483290.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins4723520008996168590.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins7042273575748624492.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_AvroIOIT] $ /bin/bash -xe 
/tmp/jenkins3368275543866736459.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.8.2)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.2.8.2)
Requirement already satisfied: grpcio<2,>=1.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.13.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(2.1.0)
Requirement already satisfied: httplib2<=0.11.3,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.11.3)
Requirement already satisfied: 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT #289

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins5639514355070264322.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins8386495882052158872.sh
+ rm -rf .beam_env
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins9071780644221283159.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins2161149691469585106.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins4214785282998535076.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins3010346802809583994.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins3077328498723448393.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins4548075318380120404.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_ParquetIOIT] $ /bin/bash -xe 
/tmp/jenkins72508652315988413.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.8.2)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.2.8.2)
Requirement already satisfied: grpcio<2,>=1.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.13.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(2.1.0)
Requirement already satisfied: httplib2<=0.11.3,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 

Build failed in Jenkins: beam_PerformanceTests_ParquetIOIT_HDFS #184

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3376596480716844240.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6560451113704588239.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1211891518051407024.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-parquetioit-hdfs-184
namespace "beam-performancetests-parquetioit-hdfs-184" created
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7399426748507400610.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-parquetioit-hdfs-184
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1509827535909232641.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1788198488530086197.sh
+ rm -rf .beam_env
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3699497550473891688.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8550628019189469267.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3765923688726226066.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_ParquetIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1099503653585704829.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #789

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1449430373951848767.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6734317095377198806.sh
+ rm -rf .beam_env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins4636049035581565571.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins9064972645699507888.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins2019705029970007521.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins92195388058310737.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins5745810028395905340.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins6827450499198472918.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TFRecordIOIT] $ /bin/bash -xe 
/tmp/jenkins1950785426227438934.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.8.2)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.2.8.2)
Requirement already satisfied: grpcio<2,>=1.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.13.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(2.1.0)
Requirement already satisfied: httplib2<=0.11.3,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #817

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam13 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2020595053046679620.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2086477025324258252.sh
+ rm -rf .beam_env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8322023823672119872.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins4643770377068859076.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1035430998016530214.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins5895860682689285248.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins6468250469026539055.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins2334370093496297091.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins1213798554871785108.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.8.2)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.2.8.2)
Requirement already satisfied: grpcio<2,>=1.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.13.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(2.1.0)
Requirement already satisfied: httplib2<=0.11.3,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.11.3)
Requirement already satisfied: 

Build failed in Jenkins: beam_PreCommit_Java_Cron #154

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
[...truncated 17.24 MB...]
INFO: 2018-07-27T18:16:51.715Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:51.762Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:51.798Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:51.835Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/ExpandIterable
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:51.881Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Drop 
key/Values/Map
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:51.917Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle.ViaRandomKey/Pair
 with random key into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Gather 
bundles
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:51.970Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:52.018Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:52.060Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Write
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Reify
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:52.105Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/GroupByWindow
 into 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/GroupByKey/Read
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:52.144Z: Unzipping flatten s13-u58 for input 
s14.org.apache.beam.sdk.values.PCollection.:370#43d0369d0f0559b7-c56
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:52.190Z: Fusing unzipped copy of 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Reshuffle/Window.Into()/Window.Assign,
 through flatten s13-u58, into producer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/GatherTempFileResults/Add void 
key/AddKeys/Map
Jul 27, 2018 6:16:58 PM 
org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
INFO: 2018-07-27T18:16:52.236Z: Fusing consumer 
WriteOneFilePerWindow/TextIO.Write/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair
 with random key into 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #466

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8950600045485835217.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8696457604136756451.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5696912227944613758.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-compressed-textioit-hdfs-466
namespace "beam-performancetests-compressed-textioit-hdfs-466" created
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6393312126918406909.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-compressed-textioit-hdfs-466
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2095126688223360339.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins330359335910770356.sh
+ rm -rf .beam_env
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1789239944685657551.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5791131170054191493.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7283752168079108127.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Compressed_TextIOIT_HDFS] $ 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #465

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam10 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8119115562378276170.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5864367650063582533.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2083779187023871578.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-xmlioit-hdfs-465
namespace "beam-performancetests-xmlioit-hdfs-465" created
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins699842225404245486.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-xmlioit-hdfs-465
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6014619183923155562.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins9191812166878471145.sh
+ rm -rf .beam_env
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5784783425611898232.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5793378248259422420.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins479330985089344.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6397153793847291733.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)

Build failed in Jenkins: beam_PerformanceTests_Python #1258

[Apache Jenkins Server]

See 


Changes:

[ehudm] Python PubSub updates:

[ehudm] Fix lint errors and clarify PubsubMessage docs.

[ehudm] Skip unit test if pubsub is not installed.

[ehudm] Fix docstring indentation.

[ehudm] Minor fixes.

[lcwik] [BEAM-4860] Ensure that shaded artifacts that are produced don't leak

[pablo] [BEAM-4818] Move the visteg plugin to the root project.

[lcwik] [BEAM-4262] Bumping bigtable versions (#6078)

[github] [BEAM-4658] Update pipeline representation in runner support libraries

[pablo] Changed hardcoded version number into env variable. Added more comments.

[aaltay] [BEAM-4076] Fix schemas on Dataflow and FnApi (#6072)

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk aa2b585a5356ce3b607dd3d17a3c6ab2ce65353b # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8055711672218079793.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3631238739723495981.sh
+ rm -rf .beam_env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1463782467337498506.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7762123783080025661.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4373957814379593676.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2690147038434867229.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7250500320601567036.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7762139428107416882.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5165480218991547042.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.8.2)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Requirement already 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #476

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam11 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6688904602124359529.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins55163208866818501.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5518922879038035172.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-textioit-hdfs-476
namespace "beam-performancetests-textioit-hdfs-476" created
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins853707827625091034.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-textioit-hdfs-476
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5837084009709425841.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2678295567562447283.sh
+ rm -rf .beam_env
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7384753640183589382.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8764681770493235567.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4758277083928930937.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2813421562253566.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #566

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam8 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins568936217841374858.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins509016382234182.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins1649839204878856454.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-hadoopinputformat-566
namespace "beam-performancetests-hadoopinputformat-566" created
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins3229922319350969077.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-hadoopinputformat-566
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins3275660470482778.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins4522874050252334230.sh
+ rm -rf .beam_env
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins5421762309539908149.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins7220303765922816972.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins1862410808271256030.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_HadoopInputFormat] $ /bin/bash -xe 
/tmp/jenkins5216581838501439314.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 

Build failed in Jenkins: beam_PerformanceTests_Spark #2004

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins2865729309441400071.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8485485912482542910.sh
+ rm -rf .beam_env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins626568125936203751.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins6710272760267067130.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3493477126103153991.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins3353314334441187539.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins5394603990052921755.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins7557321415792237654.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Spark] $ /bin/bash -xe /tmp/jenkins8984065146620547329.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.8.2)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.2.8.2)
Requirement already satisfied: grpcio<2,>=1.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.13.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(2.1.0)
Requirement already satisfied: httplib2<=0.11.3,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.11.3)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #561

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam4 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins1104745000540452231.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins1192572838163468563.sh
+ rm -rf .beam_env
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins280048458515817372.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins8050151679500263556.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins1680674334192085622.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins4439455652296095762.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins6451514023197967487.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in 
./.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins122184588855451907.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins2068569538208075600.sh
+ .beam_env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.8.2)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.7.0.dev0) (1.7)
Requirement already satisfied: dill<=0.2.8.2,>=0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.2.8.2)
Requirement already satisfied: grpcio<2,>=1.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(1.13.0)
Requirement already satisfied: hdfs<3.0.0,>=2.1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(2.1.0)
Requirement already satisfied: httplib2<=0.11.3,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.7.0.dev0) 
(0.11.3)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 

Build failed in Jenkins: beam_PerformanceTests_JDBC #897

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam11 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7766766223190446940.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8967371655788532322.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins981611974511345412.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-jdbc-897
namespace "beam-performancetests-jdbc-897" created
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4345201627093015774.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-jdbc-897
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins2198580675982934259.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins3469901215101432010.sh
+ rm -rf .beam_env
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8751606217644572154.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7207593771497314146.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6005637382318383537.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins3267987719563274993.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 
(18.0)
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6238117651742452734.sh
+ .perfkit_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.perfkit_env/lib/python2.7/site-packages 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #465

[Apache Jenkins Server]

See 


Changes:

[qinyeli] Introducing version into CacheManager

[lukasz.gajowy] [BEAM-5037] Fix deserialization problem due to transient field

[lukasz.gajowy] Remove redundant static field

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 7ea9c82457c96835215bf799836738e256bac1da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 7ea9c82457c96835215bf799836738e256bac1da
Commit message: "Merge pull request #6083: [BEAM-5037] fix synthetic options"
 > git rev-list --no-walk 6afa12fefced45b79aecb0c3ca6040757acd12cd # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3940146926545499005.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running [gcloud.container.clusters.get-credentials] with arguments: 
[--verbosity: "debug", --zone: "us-central1-a", NAME: "io-datastores"]
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6307441862907665188.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7064120224538361653.sh
+ kubectl 
--kubeconfig=
 create namespace beam-performancetests-avroioit-hdfs-465
namespace "beam-performancetests-avroioit-hdfs-465" created
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5630533251402438350.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=beam-performancetests-avroioit-hdfs-465
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins677030762659162401.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5813199959752511550.sh
+ rm -rf .beam_env
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4381510410169794002.sh
+ rm -rf .perfkit_env
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1690937880488154402.sh
+ virtualenv .beam_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7270431417400082865.sh
+ virtualenv .perfkit_env --system-site-packages
New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_AvroIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5166519510159931307.sh
+ .beam_env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.beam_env/lib/python2.7/site-packages (40.0.0)
Requirement already up-to-date: pip in ./.beam_env/lib/python2.7/site-packages 

[beam-site] 01/01: This closes #511

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 5580b8775ba34eb312fadf29cd9c9400eb8ac7fc
Merge: 8a7108d 491c456
Author: Mergebot 
AuthorDate: Fri Jul 27 17:55:51 2018 +

This closes #511

 src/_includes/section-menu/contribute.html |  6 ++
 src/contribute/postcommits-guides.md   | 74 
 src/contribute/postcommits-policies-details.md | 93 ++
 src/contribute/postcommits-policies.md | 82 +++
 src/contribute/testing.md  | 81 ++
 5 files changed, 336 insertions(+)

[beam-site] branch mergebot updated (7756e66 -> 5580b87)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 7756e66  This closes #511
 new 5580b87  This closes #511

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (7756e66)
\
 N -- N -- N   refs/heads/mergebot (5580b87)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:

[jira] [Work logged] (BEAM-5037) HashFunction is not intialized in SyntheticOptions

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5037?focusedWorklogId=128230=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128230
 ]

ASF GitHub Bot logged work on BEAM-5037:


Author: ASF GitHub Bot
Created on: 27/Jul/18 17:53
Start Date: 27/Jul/18 17:53
Worklog Time Spent: 10m 
  Work Description: lgajowy closed pull request #6083: [BEAM-5037] fix 
synthetic options
URL: https://github.com/apache/beam/pull/6083
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedIO.java
 
b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedIO.java
index d9f652ace9d..71b146d43fc 100644
--- 
a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedIO.java
+++ 
b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticBoundedIO.java
@@ -186,7 +186,7 @@ public SyntheticSourceReader createReader(PipelineOptions 
pipelineOptions) {
   for (int i = 0; i < relativeSizes.length; ++i) {
 relativeSizes[i] =
 sourceOptions.bundleSizeDistribution.sample(
-sourceOptions.hashFunction.hashInt(i).asLong());
+sourceOptions.hashFunction().hashInt(i).asLong());
   }
 
   // Generate offset ranges proportional to the relative sizes.
@@ -307,7 +307,7 @@ public Record genRecord(long position) {
   // close to each other. To make seeds fed into the Random objects 
unrelated,
   // we use a hashing function to map the position to its corresponding 
hashcode,
   // and use the hashcode as a seed to feed into the Random object.
-  long hashCodeOfPosition = hashFunction.hashLong(position).asLong();
+  long hashCodeOfPosition = hashFunction().hashLong(position).asLong();
   return new Record(genKvPair(hashCodeOfPosition), 
nextDelay(hashCodeOfPosition));
 }
 
@@ -402,7 +402,7 @@ protected boolean advanceImpl() {
   currentKvPair = record.kv;
   // TODO: add a separate distribution for the sleep time of reading the 
first record
   // (e.g.,"open" the files).
-  long hashCodeOfVal = 
options.hashFunction.hashBytes(currentKvPair.getValue()).asLong();
+  long hashCodeOfVal = 
options.hashFunction().hashBytes(currentKvPair.getValue()).asLong();
   Random random = new Random(hashCodeOfVal);
   SyntheticUtils.delay(
   record.sleepMsec, options.cpuUtilizationInMixedDelay, 
options.delayType, random);
diff --git 
a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticOptions.java
 
b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticOptions.java
index 9e91c67c683..7cb9b94a38d 100644
--- 
a/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticOptions.java
+++ 
b/sdks/java/io/synthetic/src/main/java/org/apache/beam/sdk/io/synthetic/SyntheticOptions.java
@@ -61,9 +61,6 @@
 MIXED,
   }
 
-  /** Mapper for (de)serializing JSON. */
-  private static final ObjectMapper MAPPER = new ObjectMapper();
-
   /**
* Wrapper over a distribution. Unfortunately commons-math does not provide 
a common interface
* over both RealDistribution and IntegerDistribution, and we sometimes need 
one and sometimes the
@@ -160,7 +157,7 @@ public Object getDistribution() {
* The hash function is used to generate seeds that are fed into the random 
number generators and
* the sleep time distributions.
*/
-  @JsonIgnore public transient HashFunction hashFunction;
+  @JsonIgnore private transient HashFunction hashFunction;
 
   /**
* SyntheticOptions supports several delay distributions including uniform, 
normal, exponential,
@@ -217,7 +214,15 @@ public Object getDistribution() {
   @JsonDeserialize
   public void setSeed(int seed) {
 this.seed = seed;
-this.hashFunction = Hashing.murmur3_128(seed);
+  }
+
+  public HashFunction hashFunction() {
+// due to field's transiency initialize when null.
+if (hashFunction == null) {
+  this.hashFunction = Hashing.murmur3_128(seed);
+}
+
+return hashFunction;
   }
 
   static class SamplerDeserializer extends JsonDeserializer {
@@ -298,7 +303,6 @@ public void validate() {
 hotKeyFraction >= 0,
 "hotKeyFraction should be a non-negative number, but found %s",
 hotKeyFraction);
-checkArgument(hashFunction != null, "hashFunction hasn't been 
initialized.");
 if (hotKeyFraction > 0) {
   int intBytes = Integer.SIZE / 8;
   checkArgument(
@@ -314,7 +318,7 @@ public void validate() {
   @Override
   public String toString() {

[jira] [Work logged] (BEAM-5018) Beam Dependency Update Request: org.tukaani

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5018?focusedWorklogId=128229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-128229
 ]

ASF GitHub Bot logged work on BEAM-5018:


Author: ASF GitHub Bot
Created on: 27/Jul/18 17:53
Start Date: 27/Jul/18 17:53
Worklog Time Spent: 10m 
  Work Description: yifanzou opened a new pull request #6086: Do Not Merge. 
[BEAM-5018] Upgrade org.tukaani:xz to 1.8
URL: https://github.com/apache/beam/pull/6086
 
 
   Test Ongoing
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 128229)
Time Spent: 10m
Remaining Estimate: 0h

> Beam Dependency Update Request: org.tukaani
> ---
>
> Key: BEAM-5018
> URL: https://issues.apache.org/jira/browse/BEAM-5018
> Project: Beam
>  Issue Type: Bug
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> 2018-07-25 20:35:34.272505
> Please review and upgrade the org.tukaani to the latest version None 
>  
> cc: 



--
This message was sent