[jira] [Commented] (BEAM-5396) Flink portable runner savepoint / upgrade support

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618442#comment-16618442
 ] 

Ankur Goenka commented on BEAM-5396:


Makes sense. Its a runner specific functionality so its a good idea to track it 
in feature matrix.

> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Priority: Major
>
> The Flink runner overall and the new portable implementation specifically 
> need to support Flink savepoints for production use. Specifically, it should 
> be possible to upgrade a stateful portable Beam pipeline that runs on Flink, 
> which involves taking a savepoint and then starting the new version of the 
> pipeline from that savepoint. The potential issues with pipeline evolution 
> and migration are similar to those when using the Flink DataStream API 
> (schema / name changes etc.).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1073

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #6007

[Apache Jenkins Server]

See 

[beam] branch charlesccychen-patch-1 deleted (was deaaa76)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch charlesccychen-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git.


 was deaaa76  [BEAM-5414] Latest grpcio-tools incompatible with latest 
protobuf 3.6.1

The revisions that were on this branch are still contained in
other references; therefore, this change does not discard any commits
from the repository.

[jira] [Work logged] (BEAM-5414) grpcio-tools 1.15.0 proto generation breaks compatibility with latest protobuf 3.6.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5414?focusedWorklogId=145189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145189
 ]

ASF GitHub Bot logged work on BEAM-5414:


Author: ASF GitHub Bot
Created on: 18/Sep/18 03:16
Start Date: 18/Sep/18 03:16
Worklog Time Spent: 10m 
  Work Description: charlesccychen closed pull request #6419: [BEAM-5414] 
Latest grpcio-tools incompatible with latest protobuf 3.6.1
URL: https://github.com/apache/beam/pull/6419
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145189)
Time Spent: 0.5h  (was: 20m)

> grpcio-tools 1.15.0 proto generation breaks compatibility with latest 
> protobuf 3.6.1
> 
>
> Key: BEAM-5414
> URL: https://issues.apache.org/jira/browse/BEAM-5414
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, the latest grpcio-tools 1.15.0 proto generation breaks 
> compatibility with the latest protobuf 3.6.1.  This means that any proto 
> generation needs to happen with grpcio-tools 1.14.2 until this is fixed.
> CC: [~angoenka]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (738bf00 -> d6dc360)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 738bf00  Merge pull request #6316: [BEAM-4461] Add Unnest transform.
 add deaaa76  [BEAM-5414] Latest grpcio-tools incompatible with latest 
protobuf 3.6.1
 new d6dc360  Merge pull request #6419 from apache/charlesccychen-patch-1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/gen_protos.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

[beam] 01/01: Merge pull request #6419 from apache/charlesccychen-patch-1

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d6dc3602c808ce1961f46098deb898e5c7480ad4
Merge: 738bf00 deaaa76
Author: Charles Chen 
AuthorDate: Mon Sep 17 20:16:31 2018 -0700

Merge pull request #6419 from apache/charlesccychen-patch-1

[BEAM-5414] Latest grpcio-tools incompatible with latest protobuf 3.6.1

 sdks/python/gen_protos.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=145187=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145187
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 18/Sep/18 03:13
Start Date: 18/Sep/18 03:13
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #6298: [BEAM-4461] 
Introduce Group transform.
URL: https://github.com/apache/beam/pull/6298#issuecomment-422239865
 
 
   @akedin  @apilloud most of the files are now reverted, as they are submitted 
in other PRs. There's a little more I can split off into separate PRs, if you 
think necessary.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145187)
Time Spent: 10.5h  (was: 10h 20m)

> Create a library of useful transforms that use schemas
> --
>
> Key: BEAM-4461
> URL: https://issues.apache.org/jira/browse/BEAM-4461
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>Priority: Major
>  Time Spent: 10.5h
>  Remaining Estimate: 0h
>
> e.g. JoinBy(fields). Project, Filter, etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5396) Flink portable runner savepoint / upgrade support

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618418#comment-16618418
 ] 

Thomas Weise commented on BEAM-5396:


The operability features are table stakes for serious projects. They include 
not only a story for compatibility and upgrades, but also metrics, logging etc. 
I think that we should track these in the feature matrix.

 

> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Priority: Major
>
> The Flink runner overall and the new portable implementation specifically 
> need to support Flink savepoints for production use. Specifically, it should 
> be possible to upgrade a stateful portable Beam pipeline that runs on Flink, 
> which involves taking a savepoint and then starting the new version of the 
> pipeline from that savepoint. The potential issues with pipeline evolution 
> and migration are similar to those when using the Flink DataStream API 
> (schema / name changes etc.).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5375) KafkaIO reader should handle runtime exceptions kafka client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5375?focusedWorklogId=145184=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145184
 ]

ASF GitHub Bot logged work on BEAM-5375:


Author: ASF GitHub Bot
Created on: 18/Sep/18 03:07
Start Date: 18/Sep/18 03:07
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6421: [cherrypick 
d34d3af068] [BEAM-5375] KafkaIO : Handle runtime exceptions while fetching from 
Kafka better.  (#6391)
URL: https://github.com/apache/beam/pull/6421#issuecomment-422238947
 
 
   Thanks, this LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145184)
Time Spent: 1h 40m  (was: 1.5h)

> KafkaIO reader should handle runtime exceptions kafka client
> 
>
> Key: BEAM-5375
> URL: https://issues.apache.org/jira/browse/BEAM-5375
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka
>Affects Versions: 2.7.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> KafkaIO reader might stop reading from Kafka without any explicit error 
> message if KafkaConsumer throws a runtime exception while polling for 
> messages. One of the Dataflow customers encountered this issue (see [user@ 
> thread|[https://lists.apache.org/thread.html/c0cf8f45f567a0623592e2d8340f5288e3e774b59bca985aec410a81@%3Cuser.beam.apache.org%3E])]
> 'consumerPollThread()' in KafkaIO deliberately avoided catching runtime 
> exceptions. It shoud handle it.. stuff happens at runtime. 
> It should result in 'IOException' from start()/advance(). The runners will 
> handle properly reporting and closing the readers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5375) KafkaIO reader should handle runtime exceptions kafka client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5375?focusedWorklogId=145185=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145185
 ]

ASF GitHub Bot logged work on BEAM-5375:


Author: ASF GitHub Bot
Created on: 18/Sep/18 03:07
Start Date: 18/Sep/18 03:07
Worklog Time Spent: 10m 
  Work Description: charlesccychen closed pull request #6421: [cherrypick 
d34d3af068] [BEAM-5375] KafkaIO : Handle runtime exceptions while fetching from 
Kafka better.  (#6391)
URL: https://github.com/apache/beam/pull/6421
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
index de241a87676..68ea8127753 100644
--- 
a/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
+++ 
b/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java
@@ -172,7 +172,6 @@ public boolean start() throws IOException {
 offsetFetcherThread.scheduleAtFixedRate(
 this::updateLatestOffsets, 0, OFFSET_UPDATE_INTERVAL_SECONDS, 
TimeUnit.SECONDS);
 
-nextBatch();
 return advance();
   }
 
@@ -376,6 +375,7 @@ public long getSplitBacklogBytes() {
   // like 100 milliseconds does not work well. This along with large receive 
buffer for
   // consumer achieved best throughput in tests (see 
`defaultConsumerProperties`).
   private final ExecutorService consumerPollThread = 
Executors.newSingleThreadExecutor();
+  private AtomicReference consumerPollException = new 
AtomicReference<>();
   private final SynchronousQueue> 
availableRecordsQueue =
   new SynchronousQueue<>();
   private AtomicReference finalizedCheckpointMark = new 
AtomicReference<>();
@@ -391,7 +391,7 @@ public long getSplitBacklogBytes() {
   Executors.newSingleThreadScheduledExecutor();
   private static final int OFFSET_UPDATE_INTERVAL_SECONDS = 1;
 
-  static final long UNINITIALIZED_OFFSET = -1;
+  private static final long UNINITIALIZED_OFFSET = -1;
 
   //Add SpEL instance to cover the interface difference of Kafka client
   private transient ConsumerSpEL consumerSpEL;
@@ -570,28 +570,33 @@ Instant updateAndGetWatermark() {
   private void consumerPollLoop() {
 // Read in a loop and enqueue the batch of records, if any, to 
availableRecordsQueue.
 
-ConsumerRecords records = ConsumerRecords.empty();
-while (!closed.get()) {
-  try {
-if (records.isEmpty()) {
-  records = consumer.poll(KAFKA_POLL_TIMEOUT.getMillis());
-} else if (availableRecordsQueue.offer(
-records, RECORDS_ENQUEUE_POLL_TIMEOUT.getMillis(), 
TimeUnit.MILLISECONDS)) {
-  records = ConsumerRecords.empty();
-}
-KafkaCheckpointMark checkpointMark = 
finalizedCheckpointMark.getAndSet(null);
-if (checkpointMark != null) {
-  commitCheckpointMark(checkpointMark);
+try {
+  ConsumerRecords records = ConsumerRecords.empty();
+  while (!closed.get()) {
+try {
+  if (records.isEmpty()) {
+records = consumer.poll(KAFKA_POLL_TIMEOUT.getMillis());
+  } else if (availableRecordsQueue.offer(
+  records, RECORDS_ENQUEUE_POLL_TIMEOUT.getMillis(), 
TimeUnit.MILLISECONDS)) {
+records = ConsumerRecords.empty();
+  }
+  KafkaCheckpointMark checkpointMark = 
finalizedCheckpointMark.getAndSet(null);
+  if (checkpointMark != null) {
+commitCheckpointMark(checkpointMark);
+  }
+} catch (InterruptedException e) {
+  LOG.warn("{}: consumer thread is interrupted", this, e); // not 
expected
+  break;
+} catch (WakeupException e) {
+  break;
 }
-  } catch (InterruptedException e) {
-LOG.warn("{}: consumer thread is interrupted", this, e); // not 
expected
-break;
-  } catch (WakeupException e) {
-break;
   }
+  LOG.info("{}: Returning from consumer pool loop", this);
+} catch (Exception e) { // mostly an unrecoverable KafkaException.
+  LOG.error("{}: Exception while reading from Kafka", this, e);
+  consumerPollException.set(e);
+  throw e;
 }
-
-LOG.info("{}: Returning from consumer pool loop", this);
   }
 
   private void commitCheckpointMark(KafkaCheckpointMark checkpointMark) {
@@ -622,7 +627,7 @@ void finalizeCheckpointMarkAsync(KafkaCheckpointMark 
checkpointMark) {
 checkpointMarkCommitsEnqueued.inc();
   }
 
-  private void nextBatch() {
+  private void nextBatch() throws IOException {
 curBatch = 

[beam] branch release-2.7.0 updated (f29e74a -> e14c385)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch release-2.7.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f29e74a  Merge pull request #6412 from lukecwik/release270
 add d34d3af  [BEAM-5375] KafkaIO : Handle runtime exceptions while 
fetching from Kafka better.  (#6391)
 new e14c385  Merge pull request #6421 from rangadi/cherrypick

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/kafka/KafkaUnboundedReader.java| 55 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  | 40 
 2 files changed, 72 insertions(+), 23 deletions(-)

[beam] 01/01: Merge pull request #6421 from rangadi/cherrypick

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch release-2.7.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e14c385ef1d1cbdd7c5d79dc6e135f1aa494c39c
Merge: f29e74a d34d3af
Author: Charles Chen 
AuthorDate: Mon Sep 17 20:07:03 2018 -0700

Merge pull request #6421 from rangadi/cherrypick

[cherrypick d34d3af068] [BEAM-5375] KafkaIO : Handle runtime exceptions 
while fetching from Kafka better.  (#6391)

 .../beam/sdk/io/kafka/KafkaUnboundedReader.java| 55 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  | 40 
 2 files changed, 72 insertions(+), 23 deletions(-)

Build failed in Jenkins: beam_PostCommit_Py_VR_Dataflow #1072

[Apache Jenkins Server]

See 


Changes:

[relax] Add Unnest transform.

[relax] Apply spotless.

[relax] Add javadoc.

[relax] Address CR comments.

[relax] Address code-review comments.

[relax] Fix duplicate field names in SQL.

[relax] Fix comment.

[relax] Fix typo.

--
[...truncated 74.47 KB...]
test_flatten_multiple_pcollections_having_multiple_consumers 
(apache_beam.transforms.ptransform_test.PTransformTest) ... ERROR
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.3.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.3.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.3.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.3.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.3.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.3.0.zip
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-40.3.0.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.2.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Successfully downloaded 

Jenkins build is back to normal : beam_PostCommit_Python_PVR_Flink_Gradle #46

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5342) Migrate google-api-client libraries to 1.24.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5342?focusedWorklogId=145182=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145182
 ]

ASF GitHub Bot logged work on BEAM-5342:


Author: ASF GitHub Bot
Created on: 18/Sep/18 02:36
Start Date: 18/Sep/18 02:36
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6365: [BEAM-5342] 
Upgrades Google API Client libraries to 1.24.1
URL: https://github.com/apache/beam/pull/6365#issuecomment-422234250
 
 
   R: @huygaa11 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145182)
Time Spent: 40m  (was: 0.5h)

> Migrate google-api-client libraries to 1.24.1
> -
>
> Key: BEAM-5342
> URL: https://issues.apache.org/jira/browse/BEAM-5342
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp, runner-dataflow
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> We currently use 1.23 libraries which is about an year old. We should migrate 
> to more recent 1.24.1 which fixes several known issues.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4461) Create a library of useful transforms that use schemas

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4461?focusedWorklogId=145181=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145181
 ]

ASF GitHub Bot logged work on BEAM-4461:


Author: ASF GitHub Bot
Created on: 18/Sep/18 02:34
Start Date: 18/Sep/18 02:34
Worklog Time Spent: 10m 
  Work Description: reuvenlax closed pull request #6316: [BEAM-4461] Add 
Unnest transform.
URL: https://github.com/apache/beam/pull/6316
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
index 43d7a0f813d..86a0f4653d5 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/Schema.java
@@ -167,6 +167,10 @@ public Schema(List fields) {
 this.fields = fields;
 int index = 0;
 for (Field field : fields) {
+  if (fieldIndices.get(field.getName()) != null) {
+throw new IllegalArgumentException(
+"Duplicate field " + field.getName() + " added to schema");
+  }
   fieldIndices.put(field.getName(), index++);
 }
 this.hashCode = Objects.hash(fieldIndices, fields);
@@ -528,17 +532,18 @@ public int hashCode() {
 
 public abstract Builder toBuilder();
 
+/** Builder for {@link Field}. */
 @AutoValue.Builder
-abstract static class Builder {
-  abstract Builder setName(String name);
+public abstract static class Builder {
+  public abstract Builder setName(String name);
 
-  abstract Builder setDescription(String description);
+  public abstract Builder setDescription(String description);
 
-  abstract Builder setType(FieldType fieldType);
+  public abstract Builder setType(FieldType fieldType);
 
-  abstract Builder setNullable(Boolean nullable);
+  public abstract Builder setNullable(Boolean nullable);
 
-  abstract Field build();
+  public abstract Field build();
 }
 
 /** Return's a field with the give name and type. */
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
new file mode 100644
index 000..9c3381ef4e1
--- /dev/null
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.schemas.transforms;
+
+import com.google.auto.value.AutoValue;
+import com.google.common.collect.Lists;
+import java.util.List;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.annotations.Experimental.Kind;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.Schema.Field;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+/**
+ * A {@link PTransform} to unnest nested rows.
+ *
+ * For example, consider a Row with the following nestedschema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location: LatLong
+ *
+ * LatLong Schema: latitude: DOUBLE longitude: DOUBLE
+ *
+ * After unnesting, all of the rows will be converted to rows satisfying 
the following schema:
+ *
+ * UserEvent Schema: userid: INT64 timestamp: DATETIME location.latitude: 
DOUBLE
+ * location.longitude: DOUBLE
+ *
+ * By default nested names are concatenated to generated the unnested name, 
however {@link
+ * Unnest.Inner#withFieldNameFunction} can be used to specify a custom naming 
policy.
+ *
+ * Note that currently array and map values are not unnested.
+ */

[beam] 01/01: Merge pull request #6316: [BEAM-4461] Add Unnest transform.

[reuvenlax]

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 738bf0097c4b573528e955ebdf4001d9eec4f755
Merge: 2940915 ebce0d7
Author: reuvenlax 
AuthorDate: Mon Sep 17 19:34:20 2018 -0700

Merge pull request #6316: [BEAM-4461] Add Unnest transform.

 .../java/org/apache/beam/sdk/schemas/Schema.java   |  17 ++-
 .../apache/beam/sdk/schemas/transforms/Unnest.java | 170 +
 .../org/apache/beam/sdk/transforms/Create.java |  13 ++
 .../beam/sdk/schemas/transforms/UnnestTest.java| 161 +++
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  34 +++--
 .../sql/impl/transform/BeamJoinTransforms.java |  61 
 6 files changed, 409 insertions(+), 47 deletions(-)

[beam] branch master updated (2940915 -> 738bf00)

[reuvenlax]

This is an automated email from the ASF dual-hosted git repository.

reuvenlax pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2940915  Merge pull request #6382 from apilloud/sqlagg
 add dca825d  Add Unnest transform.
 add 6f5e4a2  Apply spotless.
 add b4f9a7b  Add javadoc.
 add 4912cd9  Address CR comments.
 add 1ecbdd5  Address code-review comments.
 add c8acedf  Fix duplicate field names in SQL.
 add 73b980a8 Fix comment.
 add ebce0d7  Fix typo.
 new 738bf00  Merge pull request #6316: [BEAM-4461] Add Unnest transform.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/schemas/Schema.java   |  17 ++-
 .../apache/beam/sdk/schemas/transforms/Unnest.java | 170 +
 .../org/apache/beam/sdk/transforms/Create.java |  13 ++
 .../beam/sdk/schemas/transforms/UnnestTest.java| 161 +++
 .../sdk/extensions/sql/impl/rel/BeamJoinRel.java   |  34 +++--
 .../sql/impl/transform/BeamJoinTransforms.java |  61 
 6 files changed, 409 insertions(+), 47 deletions(-)
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/Unnest.java
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/UnnestTest.java

Build failed in Jenkins: beam_PostCommit_Python_Verify #6006

[Apache Jenkins Server]

See 


--
[...truncated 1.09 MB...]
test_deterministic_coder (apache_beam.coders.coders_test_common.CodersTest) ... 
ok
test_dill_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_fast_primitives_coder (apache_beam.coders.coders_test_common.CodersTest) 
... ok
test_float_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_global_window_coder (apache_beam.coders.coders_test_common.CodersTest) ... 
ok
test_interval_window_coder (apache_beam.coders.coders_test_common.CodersTest) 
... ok
test_iterable_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_iterable_coder_unknown_length 
(apache_beam.coders.coders_test_common.CodersTest) ... ok
test_length_prefix_coder (apache_beam.coders.coders_test_common.CodersTest) ... 
ok
test_nested_observables (apache_beam.coders.coders_test_common.CodersTest) ... 
ok
test_pickle_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_proto_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_singleton_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_timestamp_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_tuple_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_tuple_sequence_coder (apache_beam.coders.coders_test_common.CodersTest) 
... ok
test_utf8_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_varint_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_windowed_value_coder (apache_beam.coders.coders_test_common.CodersTest) 
... ok
test_windowedvalue_coder_paneinfo 
(apache_beam.coders.coders_test_common.CodersTest) ... ok
test_observable (apache_beam.coders.observable_test.ObservableMixinTest) ... 
:50:
 DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(3, self.observed_count)
ok
test_base64_pickle_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_bytes_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_custom_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_deterministic_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_dill_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_fast_primitives_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_float_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_global_window_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_interval_window_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_iterable_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_iterable_coder_unknown_length 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_length_prefix_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_nested_observables 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_pickle_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_proto_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_singleton_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_timestamp_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_tuple_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_tuple_sequence_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_utf8_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_varint_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_windowed_value_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_windowedvalue_coder_paneinfo 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_using_slow_impl (apache_beam.coders.slow_coders_test.SlowCoders) ... ok
test_standard_coders 
(apache_beam.coders.standard_coders_test.StandardCodersTest) ... 
:50:
 ResourceWarning: unclosed file <_io.TextIOWrapper 
name='
 mode='r' encoding='UTF-8'>
  for ix, spec in enumerate(yaml.load_all(open(test_yaml))):
ok
test_byte_counting (apache_beam.coders.stream_test.StreamTest) ... 
:131:
 DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(0, bc_s.get_count())
ok

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145169=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145169
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 18/Sep/18 01:12
Start Date: 18/Sep/18 01:12
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6420: [BEAM-5364] Check 
if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6420#issuecomment-40603
 
 
   Thanks. Waiting for tests.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145169)
Time Spent: 4h  (was: 3h 50m)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5408) (Java) Using Compression.GZIP with TFRecordIO

[Raghu Angadi (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618351#comment-16618351
 ] 

Raghu Angadi commented on BEAM-5408:


The root cause for this is described in BEAM-5412. Both could be combined into 
one.

> (Java) Using Compression.GZIP with TFRecordIO
> -
>
> Key: BEAM-5408
> URL: https://issues.apache.org/jira/browse/BEAM-5408
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.4.0
>Reporter: haden lee
>Assignee: Chamikara Jayalath
>Priority: Major
>
> In short, `TFRecrdIO.read()` does not seem to work if the entry being read is 
> longer than 8,192 (in terms of byte[] length).  `TFRecordIO.write()` seems to 
> be OK with this though (based on some experiments). Perhaps there is some 
> hard-coded value for this specific length somewhere in the SDK, and I'm 
> wondering if it can be increased or parameterized. 
> [I've posted this on 
> StackOverflow|https://stackoverflow.com/questions/52284639/beam-java-sdk-with-tfrecord-and-compression-gzip],
>  but I was advised to report it here.
> Here are the details:
> We're using Beam Java SDK (and Google Cloud Dataflow to run batch jobs) a 
> lot, and we noticed something weird (possibly a bug?) when we tried to use 
> `TFRecordIO` with `Compression.GZIP`. We were able to come up with some 
> sample code that can reproduce the errors we face.
> To be clear, we are using Beam Java SDK 2.4.
> Suppose we have `PCollection` which can be a PC of proto messages, 
> for instance, in byte[] format.
>  We usually write this to GCS (Google Cloud Storage) using Base64 encoding 
> (newline delimited Strings) or using TFRecordIO (without compression). We 
> have had no issue reading the data from GCS in this manner for a very long 
> time (2.5+ years for the former and ~1.5 years for the latter).
> Recently, we tried `TFRecordIO` with `Compression.GZIP` option, and 
> *sometimes* we get an exception as the data is seen as invalid (while being 
> read). The data itself (the gzip files) is not corrupted, and we've tested 
> various things, and reached the following conclusion.
> When a `byte[]` that is being compressed under `TFRecordIO` is above certain 
> threshold (I'd say when at or above 8192), then 
> `TFRecordIO.read().withCompression(Compression.GZIP)` would not work.
>  Specifically, it will throw the following exception:
>  
> {code:java}
> // code placeholder
> Exception in thread "main" java.lang.IllegalStateException: Invalid data
> at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkState(Preconditions.java:444)
> at org.apache.beam.sdk.io.TFRecordIO$TFRecordCodec.read(TFRecordIO.java:642)
> at 
> org.apache.beam.sdk.io.TFRecordIO$TFRecordSource$TFRecordReader.readNextRecord(TFRecordIO.java:526)
> at 
> org.apache.beam.sdk.io.CompressedSource$CompressedReader.readNextRecord(CompressedSource.java:426)
> at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:473)
> at 
> org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:468)
> at 
> org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)
> at 
> org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$BoundedReadEvaluator.processElement(BoundedReadEvaluatorFactory.java:141)
> at 
> org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:161)
> at 
> org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:125)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>  
> This can be reproduced easily, so you can refer to the code at the end. You 
> will also see comments about the byte array length (as I tested with various 
> sizes, I concluded that 8192 is the magic number).
> So I'm wondering if this is a bug or known issue – I couldn't find anything 
> close to this on Apache Beam's Issue Tracker [here][1] but if there is 
> another forum/site I need to check, please let me know!
>  If this is indeed a bug, what would be the right channel to report this?
> —
>  The following code can reproduce the error we have.
> A successful run (with parameters 1, 39, 100) would show the following 
> message at the end:
> {code:java}
> // code placeholder
>  counter metrics from CountDoFn
> [counter] plain_base64_proto_array_len: 8126
> [counter] plain_base64_proto_in: 1
> [counter] plain_base64_proto_val_cnt: 

[jira] [Assigned] (BEAM-5412) TFRecordIO fails with records larger than 8K

[Raghu Angadi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raghu Angadi reassigned BEAM-5412:
--

Assignee: Chamikara Jayalath  (was: Eugene Kirpichov)

> TFRecordIO fails with records larger than 8K
> 
>
> Key: BEAM-5412
> URL: https://issues.apache.org/jira/browse/BEAM-5412
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-text
>Affects Versions: 2.4.0
>Reporter: Raghu Angadi
>Assignee: Chamikara Jayalath
>Priority: Major
>
> This was reported on 
> [Stackoverflow|https://stackoverflow.com/questions/52284639/beam-java-sdk-with-tfrecord-and-compression-gzip].
>  TFRecordIO reader assumes a single call to {{channel.read()}} returns as 
> much as can fit in the input buffer. {{read()}} can return fewer bytes than 
> requested. Assert failure : 
> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TFRecordIO.java#L642



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5375) KafkaIO reader should handle runtime exceptions kafka client

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5375?focusedWorklogId=145164=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145164
 ]

ASF GitHub Bot logged work on BEAM-5375:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:44
Start Date: 18/Sep/18 00:44
Worklog Time Spent: 10m 
  Work Description: rangadi opened a new pull request #6421: [cherrypick 
d34d3af068] [BEAM-5375] KafkaIO : Handle runtime exceptions while fetching from 
Kafka better.  (#6391)
URL: https://github.com/apache/beam/pull/6421
 
 
   Cherry-pick #6391 into release-2.7.0.
   
   +R: @charlesccychen 
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145164)
Time Spent: 1.5h  (was: 1h 20m)

> KafkaIO reader should handle runtime exceptions kafka client
> 
>
> Key: BEAM-5375
> URL: https://issues.apache.org/jira/browse/BEAM-5375
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kafka
>Affects Versions: 2.7.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> KafkaIO reader might stop reading from Kafka without any explicit error 
> message if KafkaConsumer throws a runtime exception while polling for 
> messages. One of the Dataflow customers encountered this issue (see [user@ 
> thread|[https://lists.apache.org/thread.html/c0cf8f45f567a0623592e2d8340f5288e3e774b59bca985aec410a81@%3Cuser.beam.apache.org%3E])]
> 'consumerPollThread()' in KafkaIO deliberately avoided catching runtime 
> exceptions. It shoud handle it.. stuff happens at runtime. 
> It should result in 'IOException' from start()/advance(). The runners will 
> handle properly reporting and closing the readers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5219) Expose OutboundMessage in PubSub client

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka resolved BEAM-5219.

   Resolution: Fixed
Fix Version/s: 2.8.0

> Expose OutboundMessage in PubSub client
> ---
>
> Key: BEAM-5219
> URL: https://issues.apache.org/jira/browse/BEAM-5219
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Minor
> Fix For: 2.8.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> publish method in org/apache/beam/sdk/io/gcp/pubsub/PubsubClient.java is 
> public but the argument OutboundMessage is not public which makes the api 
> unusable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5262) JobState support for Reference Runner

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka resolved BEAM-5262.

   Resolution: Fixed
Fix Version/s: 2.8.0

> JobState support for Reference Runner
> -
>
> Key: BEAM-5262
> URL: https://issues.apache.org/jira/browse/BEAM-5262
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Minor
> Fix For: 2.8.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Reference runner does not support getStateStream which is needed by portable 
> SDK



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #1063

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145163=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145163
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:40
Start Date: 18/Sep/18 00:40
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #6225: [BEAM-4176] 
move some artifact-staging logs from info to debug
URL: https://github.com/apache/beam/pull/6225#issuecomment-422215682
 
 
   Finally coming back to this; [I tried adding an SDK-side log as you 
suggested](https://github.com/ryan-williams/beam/commit/863f3706bac051adc07778ba7cc25c48d4795163#diff-968a6a22041ca278b1b1ed91603639d8R193),
 but it feels equivalently spammy to me at first glance:
   
   
[![](https://cl.ly/3bfb319aa212/Screen%20Shot%202018-09-17%20at%208.37.56%20PM.png)](https://cl.ly/3bfb319aa212/Screen%20Shot%202018-09-17%20at%208.37.56%20PM.png)
   
   do you think it's important to INFO-log every artifact? I'd have said these 
should be DEBUGs on both ends
   
   perhaps two INFOs saying "staging X artifacts" / "staged X artifacts"?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145163)
Time Spent: 19h 40m  (was: 19.5h)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot 
> 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 19h 40m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5396) Flink portable runner savepoint / upgrade support

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618330#comment-16618330
 ] 

Ankur Goenka commented on BEAM-5396:


Thanks for the reference link. Its very relevant.
This feature require quite a bit of effort. We can start thing of a design 
while we are improving portability and once we are stable, we can get to this 
feature.

> Flink portable runner savepoint / upgrade support
> -
>
> Key: BEAM-5396
> URL: https://issues.apache.org/jira/browse/BEAM-5396
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Priority: Major
>
> The Flink runner overall and the new portable implementation specifically 
> need to support Flink savepoints for production use. Specifically, it should 
> be possible to upgrade a stateful portable Beam pipeline that runs on Flink, 
> which involves taking a savepoint and then starting the new version of the 
> pipeline from that savepoint. The potential issues with pipeline evolution 
> and migration are similar to those when using the Flink DataStream API 
> (schema / name changes etc.).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145162=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145162
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:37
Start Date: 18/Sep/18 00:37
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #6328: [BEAM-4176] 
enumerate primitive transforms in portable construction
URL: https://github.com/apache/beam/pull/6328#issuecomment-422215195
 
 
   I didn't have docker daemon running :) a re-run hung in `FlattenTest` 
though, but I'm assuming that's not related to this change…


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145162)
Time Spent: 19.5h  (was: 19h 20m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot 
> 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 19.5h
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145154=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145154
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:30
Start Date: 18/Sep/18 00:30
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 commented on issue #6420: [BEAM-5364] Check 
if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6420#issuecomment-422214040
 
 
   @chamikaramj One more try.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145154)
Time Spent: 3h 50m  (was: 3h 40m)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145152=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145152
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:29
Start Date: 18/Sep/18 00:29
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 opened a new pull request #6420: 
[BEAM-5364] Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6420
 
 
   Check if validation is disabled when validating BigtableSource
   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145152)
Time Spent: 3h 40m  (was: 3.5h)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145151
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:28
Start Date: 18/Sep/18 00:28
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 closed pull request #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index ae8fe7d04d9..edad185323c 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -1080,6 +1080,11 @@ private long 
getEstimatedSizeBytesBasedOnSamples(List sam
 
 @Override
 public void validate() {
+  if (!config.getValidate()) {
+LOG.debug("Validation is disabled");
+return;
+  }
+
   ValueProvider tableId = config.getTableId();
   checkArgument(
   tableId != null && tableId.isAccessible() && 
!tableId.get().isEmpty(),
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
index 47727e5b8a1..cadb908be5a 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
@@ -87,11 +87,13 @@
 import org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.BigtableSource;
 import org.apache.beam.sdk.io.range.ByteKey;
 import org.apache.beam.sdk.io.range.ByteKeyRange;
+import org.apache.beam.sdk.options.Description;
 import org.apache.beam.sdk.options.PipelineOptionsFactory;
 import org.apache.beam.sdk.options.ValueProvider;
 import org.apache.beam.sdk.testing.ExpectedLogs;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.testing.TestPipeline.PipelineRunMissingException;
 import org.apache.beam.sdk.transforms.Create;
 import org.apache.beam.sdk.transforms.SerializableFunction;
 import org.apache.beam.sdk.transforms.display.DisplayData;
@@ -115,6 +117,27 @@
   @Rule public ExpectedException thrown = ExpectedException.none();
   @Rule public ExpectedLogs logged = ExpectedLogs.none(BigtableIO.class);
 
+  /** Read Options for testing. */
+  public interface ReadOptions extends GcpOptions {
+@Description("The project that contains the table to export.")
+ValueProvider getBigtableProject();
+
+@SuppressWarnings("unused")
+void setBigtableProject(ValueProvider projectId);
+
+@Description("The Bigtable instance id that contains the table to export.")
+ValueProvider getBigtableInstanceId();
+
+@SuppressWarnings("unused")
+void setBigtableInstanceId(ValueProvider instanceId);
+
+@Description("The Bigtable table id to export.")
+ValueProvider getBigtableTableId();
+
+@SuppressWarnings("unused")
+void setBigtableTableId(ValueProvider tableId);
+  }
+
   static final ValueProvider NOT_ACCESSIBLE_VALUE =
   new ValueProvider() {
 @Override
@@ -223,6 +246,39 @@ public void 
testReadValidationFailsMissingInstanceIdAndProjectId() {
 read.expand(null);
   }
 
+  @Test
+  public void testReadWithRuntimeParametersValidationFailed() {
+ReadOptions options = 
PipelineOptionsFactory.fromArgs().withValidation().as(ReadOptions.class);
+
+BigtableIO.Read read =
+BigtableIO.read()
+.withProjectId(options.getBigtableProject())
+.withInstanceId(options.getBigtableInstanceId())
+.withTableId(options.getBigtableTableId());
+
+thrown.expect(IllegalArgumentException.class);
+thrown.expectMessage("tableId was not supplied");
+
+p.apply(read);
+  }
+
+  @Test
+  public void testReadWithRuntimeParametersValidationDisabled() {
+ReadOptions options = 
PipelineOptionsFactory.fromArgs().withValidation().as(ReadOptions.class);
+
+BigtableIO.Read read =
+BigtableIO.read()
+.withoutValidation()
+.withProjectId(options.getBigtableProject())
+.withInstanceId(options.getBigtableInstanceId())
+   

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145148=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145148
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:23
Start Date: 18/Sep/18 00:23
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422212927
 
 
   sigh I messed up on rebase :(


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145148)
Time Spent: 3h 20m  (was: 3h 10m)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #1450

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-5376] Support nullability on all Row types

[apilloud] Simplify BeamAggregationRule

[apilloud] Fully explain BeamAggregationRel

[apilloud] Simplify project with Aggregation

[apilloud] Remove unneeded CalcWindow

[robbe.sneyders] Add already passing modules to python 3 test list

[thw] Optional Flink job server parameters and logging cleanup.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam15 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 2940915243cb7138512c6da6e00e27c770c6c284 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2940915243cb7138512c6da6e00e27c770c6c284
Commit message: "Merge pull request #6382 from apilloud/sqlagg"
 > git rev-list --no-walk 0a61899c151c02fb7d04aa4914afe51cd8bf38b5 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7201920467874094633.sh
+ rm -rf 

[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5878263347469562612.sh
+ rm -rf 

[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3716358002857192750.sh
+ virtualenv 

New python executable in 

Also creating executable in 

Installing setuptools, pkg_resources, pip, wheel...done.
Running virtualenv with interpreter /usr/bin/python2
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins152121935705757846.sh
+ 

 install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./env/.perfkit_env/lib/python2.7/site-packages (40.3.0)
Requirement already up-to-date: pip in 
./env/.perfkit_env/lib/python2.7/site-packages (18.0)
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5215993345671146587.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git 

Cloning into 
'
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5421615136299948645.sh
+ 

 install -r 

Collecting absl-py (from -r 

 (line 14))
Collecting jinja2>=2.7 (from -r 

 (line 15))
  Using cached 
https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./env/.perfkit_env/lib/python2.7/site-packages (from -r 

 (line 16)) (40.3.0)
Collecting colorlog[windows]==2.6.0 (from -r 

 (line 17))
  Using cached 
https://files.pythonhosted.org/packages/59/1a/46a1bf2044ad8b30b52fed0f389338c85747e093fe7f51a567f4cb525892/colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r 

 (line 18))
Collecting futures>=3.0.3 (from 

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145146=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145146
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:12
Start Date: 18/Sep/18 00:12
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422211286
 
 
   I can, but Beam prefers merge commits :) : 
https://beam.apache.org/contribute/committer-guide/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145146)
Time Spent: 3h 10m  (was: 3h)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145142=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145142
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:09
Start Date: 18/Sep/18 00:09
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422210746
 
 
   Squash comments on merge at the end?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145142)
Time Spent: 2h 50m  (was: 2h 40m)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #45

[Apache Jenkins Server]

See 


--
[...truncated 6.26 MB...]
[grpc-default-executor-1] INFO sdk_worker.__init__ - Creating insecure control 
channel.
[grpc-default-executor-1] INFO sdk_worker.__init__ - Control channel 
established.
[grpc-default-executor-1] INFO 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - 
Beam Fn Control client connected with id 1
[grpc-default-executor-0] INFO sdk_worker.__init__ - Initializing SDKHarness 
with 12 workers.
[grpc-default-executor-0] INFO sdk_worker.run - Got work 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 6
[grpc-default-executor-0] INFO sdk_worker.run - Got work 5
[grpc-default-executor-1] INFO sdk_worker.run - Got work 4
[grpc-default-executor-1] INFO sdk_worker.run - Got work 3
[grpc-default-executor-1] INFO sdk_worker.run - Got work 7
[grpc-default-executor-1] INFO sdk_worker.create_state_handler - Creating 
channel for localhost:45147
[grpc-default-executor-1] INFO sdk_worker.run - Got work 2
[grpc-default-executor-1] INFO sdk_worker.run - Got work 8
[grpc-default-executor-1] INFO 
org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn Data client 
connected.
[grpc-default-executor-1] INFO data_plane.create_data_channel - Creating 
channel for localhost:36533
[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (92493c2b020ff8d11252ae3d62a548ea) switched from 
RUNNING to FINISHED.
[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (92493c2b020ff8d11252ae3d62a548ea).
[Source: Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task Source: Collection Source 
-> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (92493c2b020ff8d11252ae3d62a548ea) [FINISHED]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task Source: 
Collection Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem 92493c2b020ff8d11252ae3d62a548ea.
[grpc-default-executor-0] INFO sdk_worker.run - Got work 9
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Collection 
Source -> 
31assert_that/Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (92493c2b020ff8d11252ae3d62a548ea) switched from 
RUNNING to FINISHED.
[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145144=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145144
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 18/Sep/18 00:09
Start Date: 18/Sep/18 00:09
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422210807
 
 
   Squash commits on merge at the end?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145144)
Time Spent: 3h  (was: 2h 50m)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PreCommit_Website_Cron #72

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-5376] Support nullability on all Row types

[apilloud] Simplify BeamAggregationRule

[apilloud] Fully explain BeamAggregationRel

[apilloud] Simplify project with Aggregation

[apilloud] Remove unneeded CalcWindow

[robbe.sneyders] Add already passing modules to python 3 test list

[thw] Optional Flink job server parameters and logging cleanup.

--
[...truncated 8.30 KB...]
:assemble (Thread[Task worker for ':buildSrc' Thread 2,5,main]) completed. Took 
0.0 secs.
:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 1.86 secs.
:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.0 secs.
:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 2,5,main]) 
completed. Took 0.036 secs.
:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
4,5,main]) completed. Took 0.001 secs.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 4,5,main]) completed. 
Took 0.0 secs.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:compileTestJava (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.002 secs.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.004 secs.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:processTestResources (Thread[Task worker for ':buildSrc' Thread 4,5,main]) 
completed. Took 0.002 secs.
:testClasses (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:testClasses (Thread[Task worker for ':buildSrc' Thread 4,5,main]) completed. 
Took 0.0 secs.
:test (Thread[Task worker for ':buildSrc' Thread 4,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:test (Thread[Task worker for ':buildSrc' Thread 4,5,main]) completed. Took 
0.003 secs.
:check (Thread[Task worker for ':buildSrc' Thread 

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145134
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:52
Start Date: 17/Sep/18 23:52
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422207974
 
 
   Thanks. Please fixup commits into a single commit.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145134)
Time Spent: 2h 40m  (was: 2.5h)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145132
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:45
Start Date: 17/Sep/18 23:45
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422206841
 
 
   Added a missing period in a comment.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145132)
Time Spent: 2.5h  (was: 2h 20m)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145130=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145130
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:41
Start Date: 17/Sep/18 23:41
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422206207
 
 
   I found the error.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145130)
Time Spent: 2h 20m  (was: 2h 10m)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145129=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145129
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:40
Start Date: 17/Sep/18 23:40
Worklog Time Spent: 10m 
  Work Description: kevinsi4508 commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422205988
 
 
   Not sure if the failure is related to the change in the PR:
   
   16:48:00 FAILURE: Build failed with an exception.
   16:48:00 
   16:48:00 * What went wrong:
   16:48:00 Execution failed for task 
':beam-sdks-java-io-google-cloud-platform:checkstyleTest'.
   16:48:00 > Checkstyle rule violations were found. See the report at: 
file:///home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_Commit/src/sdks/java/io/google-cloud-platform/build/reports/checkstyle/test.html
   16:48:00   Checkstyle files with violations: 1
   16:48:00   Checkstyle violations by severity: [error:1]
   16:48:00 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145129)
Time Spent: 2h 10m  (was: 2h)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5299) Define max global window as a shared value in protos like URN enums.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5299?focusedWorklogId=145128=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145128
 ]

ASF GitHub Bot logged work on BEAM-5299:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:39
Start Date: 17/Sep/18 23:39
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on a change in pull request 
#6381: [BEAM-5299] Define max timestamp for global window in proto
URL: https://github.com/apache/beam/pull/6381#discussion_r218260703
 
 

 ##
 File path: sdks/java/core/build.gradle
 ##
 @@ -51,6 +51,8 @@ test {
 }
 
 dependencies {
+  // Required to loads constants from the model, e.g. max timestamp for global 
window
+  shadow project(path: ":beam-model-pipeline", configuration: "shadow")
 
 Review comment:
   does it make sense for java/core to depend on model/pipeline?
   
   I'd have expected a dependency arrow in the other direction based on their 
names and the little I know about what each does.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145128)
Time Spent: 40m  (was: 0.5h)

> Define max global window as a shared value in protos like URN enums.
> 
>
> Key: BEAM-5299
> URL: https://issues.apache.org/jira/browse/BEAM-5299
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, sdk-go, sdk-java-core, sdk-py-core
>Reporter: Luke Cwik
>Assignee: Maximilian Michels
>Priority: Minor
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Instead of having each language define a max timestamp themselves, define the 
> max timestamps within proto to be shared across different languages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=145119=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145119
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:17
Start Date: 17/Sep/18 23:17
Worklog Time Spent: 10m 
  Work Description: apilloud closed pull request #6382: [BEAM-4449] Finish 
splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
index b65d85fb4fd..dec5217fa37 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
@@ -33,9 +33,6 @@
 import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlOperatorExpression;
 import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlPrimitive;
 import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlUdfExpression;
-import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlWindowEndExpression;
-import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlWindowExpression;
-import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.BeamSqlWindowStartExpression;
 import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.DateOperators;
 import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.StringOperators;
 import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.operator.arithmetic.BeamSqlDivideExpression;
@@ -504,21 +501,6 @@ private static BeamSqlExpression 
getBeamSqlExpression(RexNode rexNode) {
   ret = new BeamSqlIsNotNullExpression(subExps.get(0));
   break;
 
-case "HOP":
-case "TUMBLE":
-case "SESSION":
-  ret = new BeamSqlWindowExpression(subExps, 
node.type.getSqlTypeName());
-  break;
-case "HOP_START":
-case "TUMBLE_START":
-case "SESSION_START":
-  ret = new BeamSqlWindowStartExpression();
-  break;
-case "HOP_END":
-case "TUMBLE_END":
-case "SESSION_END":
-  ret = new BeamSqlWindowEndExpression();
-  break;
 default:
   // handle UDF
   if (((RexCall) rexNode).getOperator() instanceof 
SqlUserDefinedFunction) {
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlWindowEndExpression.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlWindowEndExpression.java
deleted file mode 100644
index 9caa4158cf5..000
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlWindowEndExpression.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements.  See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership.  The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License.  You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.beam.sdk.extensions.sql.impl.interpreter.operator;
-
-import 
org.apache.beam.sdk.extensions.sql.impl.interpreter.BeamSqlExpressionEnvironment;
-import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
-import org.apache.beam.sdk.transforms.windowing.IntervalWindow;
-import org.apache.beam.sdk.values.Row;
-import org.apache.calcite.sql.type.SqlTypeName;
-import org.joda.time.DateTime;
-
-/**
- * {@code BeamSqlExpression} for {@code HOP_END}, {@code TUMBLE_END}, {@code 
SESSION_END} operation.
- *
- * These operators returns the end timestamp of window.
- */
-public class BeamSqlWindowEndExpression extends 

[beam] branch master updated (f386050 -> 2940915)

[apilloud]

This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f386050  Merge pull request #6383 from apilloud/null
 add 3532ae7  Simplify BeamAggregationRule
 add 1b420e8  Fully explain BeamAggregationRel
 add 46e3c98  Simplify project with Aggregation
 add 4c8645c  Remove unneeded CalcWindow
 new 2940915  Merge pull request #6382 from apilloud/sqlagg

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sql/impl/interpreter/BeamSqlFnExecutor.java|  18 ---
 .../operator/BeamSqlWindowEndExpression.java   |  50 ---
 .../operator/BeamSqlWindowExpression.java  |  53 ---
 .../operator/BeamSqlWindowStartExpression.java |  51 ---
 .../sql/impl/rel/BeamAggregationRel.java   |  69 --
 .../sdk/extensions/sql/impl/rel/BeamCalcRel.java   |   6 +-
 .../sql/impl/rule/AggregateWindowFactory.java  | 152 -
 .../sql/impl/rule/AggregateWindowField.java|  51 ---
 .../sql/impl/rule/BeamAggregationRule.java | 130 --
 9 files changed, 179 insertions(+), 401 deletions(-)
 delete mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlWindowEndExpression.java
 delete mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlWindowExpression.java
 delete mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/operator/BeamSqlWindowStartExpression.java
 delete mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/AggregateWindowFactory.java
 delete mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/AggregateWindowField.java

[beam] 01/01: Merge pull request #6382 from apilloud/sqlagg

[apilloud]

This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2940915243cb7138512c6da6e00e27c770c6c284
Merge: f386050 4c8645c
Author: Andrew Pilloud 
AuthorDate: Mon Sep 17 16:17:27 2018 -0700

Merge pull request #6382 from apilloud/sqlagg

[BEAM-4449] Finish splitting Aggregate and Project

 .../sql/impl/interpreter/BeamSqlFnExecutor.java|  18 ---
 .../operator/BeamSqlWindowEndExpression.java   |  50 ---
 .../operator/BeamSqlWindowExpression.java  |  53 ---
 .../operator/BeamSqlWindowStartExpression.java |  51 ---
 .../sql/impl/rel/BeamAggregationRel.java   |  69 --
 .../sdk/extensions/sql/impl/rel/BeamCalcRel.java   |   6 +-
 .../sql/impl/rule/AggregateWindowFactory.java  | 152 -
 .../sql/impl/rule/AggregateWindowField.java|  51 ---
 .../sql/impl/rule/BeamAggregationRule.java | 130 --
 9 files changed, 179 insertions(+), 401 deletions(-)

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618264#comment-16618264
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

I don't think https://github.com/apache/beam/pull/6403 is a culprit, but we do 
have a flaky test that we need to investigate.

[~huygaa] [~chamikara] [~charleschen]: Do you know how to reliably identify a 
relevant Dataflow job ID for a failed integration test from a stream of Jenkins 
test logs? In this case, for: test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT).
 

> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=145117=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145117
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:13
Start Date: 17/Sep/18 23:13
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #6382: 
[BEAM-4449] Finish splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#discussion_r218256205
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
 ##
 @@ -54,29 +61,133 @@ public void onMatch(RelOptRuleCall call) {
 
   private static RelNode updateWindow(RelOptRuleCall call, Aggregate 
aggregate, Project project) {
 ImmutableBitSet groupByFields = aggregate.getGroupSet();
-List projectMapping = project.getProjects();
+ArrayList projects = new ArrayList(project.getProjects());
 
-Optional windowField = Optional.empty();
+WindowFn windowFn = null;
+int windowFieldIndex = -1;
 
 for (int groupFieldIndex : groupByFields.asList()) {
-  RexNode projNode = projectMapping.get(groupFieldIndex);
+  RexNode projNode = projects.get(groupFieldIndex);
   if (!(projNode instanceof RexCall)) {
 continue;
   }
 
-  windowField = AggregateWindowFactory.getWindowFieldAt((RexCall) 
projNode, groupFieldIndex);
+  RexCall rexCall = (RexCall) projNode;
+  WindowFn fn = createWindowFn(rexCall.getOperands(), rexCall.op.kind);
+  if (fn != null) {
+windowFn = fn;
+windowFieldIndex = groupFieldIndex;
+projects.set(groupFieldIndex, rexCall.getOperands().get(0));
+  }
 }
 
+final Project newProject =
+project.copy(project.getTraitSet(), project.getInput(), projects, 
project.getRowType());
+
 return new BeamAggregationRel(
 aggregate.getCluster(),
 aggregate.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
-convert(
-aggregate.getInput(),
-
aggregate.getInput().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
+convert(newProject, 
newProject.getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
 aggregate.indicator,
 aggregate.getGroupSet(),
 aggregate.getGroupSets(),
 aggregate.getAggCallList(),
-windowField);
+windowFn,
+windowFieldIndex);
+  }
+
+  /**
 
 Review comment:
   The two are somewhat intertwined at the moment due to how we apply the 
`_BEGIN` and `_END` functions. As a result splitting them appears to be 
non-trivial, so I've opened https://issues.apache.org/jira/browse/BEAM-5416


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145117)
Time Spent: 8h  (was: 7h 50m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 8h
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5416) Split Aggregate and Window rel nodes in SQL

[Andrew Pilloud (JIRA)]

Andrew Pilloud created BEAM-5416:


 Summary: Split Aggregate and Window rel nodes in SQL
 Key: BEAM-5416
 URL: https://issues.apache.org/jira/browse/BEAM-5416
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Andrew Pilloud


Currently BeamAggregationRel impliments both Aggregate and Window operations. 
We should split this into two rel nodes if possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5364) BigtableIO source tries to validate table ID even though validation is turned off

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5364?focusedWorklogId=145115=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145115
 ]

ASF GitHub Bot logged work on BEAM-5364:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:08
Start Date: 17/Sep/18 23:08
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6413: [BEAM-5364]  
Check if validation is disabled when validating BigtableSource
URL: https://github.com/apache/beam/pull/6413#issuecomment-422200420
 
 
   Failures seems to be related: 
https://builds.apache.org/job/beam_PreCommit_Java_Commit/1316/console


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145115)
Time Spent: 2h  (was: 1h 50m)

>  BigtableIO source tries to validate table ID even though validation is 
> turned off
> --
>
> Key: BEAM-5364
> URL: https://issues.apache.org/jira/browse/BEAM-5364
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1084|https://www.google.com/url?q=https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java%23L1084=D=AFQjCNEfHprTOvnwAwFSrXwUuLvc__JBWg]
> The validation can be turned off with following:
> BigtableIO.read()
>             .withoutValidation() // skip validation when constructing the 
> pipelline.
> A Dataflow template cannot be constructed due to this validation failure.
>  
> Error log when trying to construct a template:
> Exception in thread "main" java.lang.IllegalArgumentException: tableId was 
> not supplied
>         at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$BigtableSource.validate(BigtableIO.java:1084)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:95)
>         at org.apache.beam.sdk.io.Read$Bounded.expand(Read.java:85)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:471)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:44)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:167)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:423)
>         at 
> org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Read.expand(BigtableIO.java:179)
>         at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
>         at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
>         at org.apache.beam.sdk.values.PBegin.apply(PBegin.java:56)
>         at org.apache.beam.sdk.Pipeline.apply(Pipeline.java:182)
>         at 
> com.google.cloud.teleport.bigtable.BigtableToAvro.main(BigtableToAvro.java:89)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5414) grpcio-tools 1.15.0 proto generation breaks compatibility with latest protobuf 3.6.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5414?focusedWorklogId=145113=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145113
 ]

ASF GitHub Bot logged work on BEAM-5414:


Author: ASF GitHub Bot
Created on: 17/Sep/18 23:04
Start Date: 17/Sep/18 23:04
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on issue #6419: [BEAM-5414] 
Latest grpcio-tools incompatible with latest protobuf 3.6.1
URL: https://github.com/apache/beam/pull/6419#issuecomment-422199810
 
 
   LGTM. Thanks.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145113)
Time Spent: 20m  (was: 10m)

> grpcio-tools 1.15.0 proto generation breaks compatibility with latest 
> protobuf 3.6.1
> 
>
> Key: BEAM-5414
> URL: https://issues.apache.org/jira/browse/BEAM-5414
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently, the latest grpcio-tools 1.15.0 proto generation breaks 
> compatibility with the latest protobuf 3.6.1.  This means that any proto 
> generation needs to happen with grpcio-tools 1.14.2 until this is fixed.
> CC: [~angoenka]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-5351:

Fix Version/s: (was: 2.8.0)
   2.7.0

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.7.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-5351.
-
Resolution: Fixed

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.7.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5351:
---

Assignee: Luke Cwik  (was: Charles Chen)

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
> Fix For: 2.7.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618246#comment-16618246
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

Actually, after looking again at Jenkins log, I am no longer sure if 13:13:17 
WARNING:root:Timing out on waiting for job 
2018-09-17_13_12_39-8645011813028270201 after 30 seconds is relevant, there are 
a few more of those:


13:02:07 WARNING:root:Timing out on waiting for job 
2018-09-17_12_58_58-3598041514621079800 after 182 seconds
13:03:08 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_57-1488988588266282762 after 603 seconds
13:05:03 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_51-1885862207900782284 after 724 seconds


> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618246#comment-16618246
 ] 

Valentyn Tymofieiev edited comment on BEAM-5415 at 9/17/18 10:46 PM:
-

Actually, after looking again at Jenkins log, I am no longer sure if "13:13:17 
WARNING:root:Timing out on waiting for job 
2018-09-17_13_12_39-8645011813028270201 after 30 seconds" is relevant, there 
are a few more of those:


13:02:07 WARNING:root:Timing out on waiting for job 
2018-09-17_12_58_58-3598041514621079800 after 182 seconds
13:03:08 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_57-1488988588266282762 after 603 seconds
13:05:03 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_51-1885862207900782284 after 724 seconds



was (Author: tvalentyn):
Actually, after looking again at Jenkins log, I am no longer sure if 13:13:17 
WARNING:root:Timing out on waiting for job 
2018-09-17_13_12_39-8645011813028270201 after 30 seconds is relevant, there are 
a few more of those:


13:02:07 WARNING:root:Timing out on waiting for job 
2018-09-17_12_58_58-3598041514621079800 after 182 seconds
13:03:08 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_57-1488988588266282762 after 603 seconds
13:05:03 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_51-1885862207900782284 after 724 seconds


> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5395) BeamPython data plane streams data

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5395?focusedWorklogId=145105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145105
 ]

ASF GitHub Bot logged work on BEAM-5395:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:37
Start Date: 17/Sep/18 22:37
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6405: 
[BEAM-5395] Chunk data streams.
URL: https://github.com/apache/beam/pull/6405#discussion_r218249585
 
 

 ##
 File path: sdks/python/apache_beam/coders/slow_stream.py
 ##
 @@ -34,15 +34,18 @@ class OutputStream(object):
 
   def __init__(self):
 self.data = []
+self.byte_count = 0
 
 Review comment:
   Nit: Shall we rely on ` len(self.data)` instead of creating and managing a 
new field.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145105)
Time Spent: 0.5h  (was: 20m)

> BeamPython data plane streams data
> --
>
> Key: BEAM-5395
> URL: https://issues.apache.org/jira/browse/BEAM-5395
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently the default implementation is to buffer all data for the bundle. 
> Experiments were made splitting at arbitrary byte boundaries, but it appears 
> that Java requires messages to be split on element boundaries. For now we 
> should implement that in Python (even if this means not being able to split 
> up large elements among multiple messages). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3371) Add ability to stage directories with compiled classes to Spark

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3371?focusedWorklogId=145100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145100
 ]

ASF GitHub Bot logged work on BEAM-3371:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:30
Start Date: 17/Sep/18 22:30
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on issue #6244: [BEAM-3371] Enable 
running integration tests on Spark
URL: https://github.com/apache/beam/pull/6244#issuecomment-422192924
 
 
   Hi @aromanenko-dev - I applied the suggestions. PTAL. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145100)
Time Spent: 2h 10m  (was: 2h)

> Add ability to stage directories with compiled classes to Spark
> ---
>
> Key: BEAM-3371
> URL: https://issues.apache.org/jira/browse/BEAM-3371
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Lukasz Gajowy
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> This one is basically the same issue as
>  [this Flink's one|https://issues.apache.org/jira/browse/BEAM-3370], except 
> of two things:
> - a detection of files to stage has to be provided in Spark, which is already 
> being developed [here|https://issues.apache.org/jira/browse/BEAM-981]
> - the test execution is not interrupted by FileNotFoundException but by *the 
> effect* of the directory not being staged (absence of test classes on the 
> Spark's classpath, hence ClassNotFoundException).
> Again, this probably could be resolved analogously as in flink, while 
> BEAM-981 issue is resolved. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3371) Add ability to stage directories with compiled classes to Spark

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3371?focusedWorklogId=145099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145099
 ]

ASF GitHub Bot logged work on BEAM-3371:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:28
Start Date: 17/Sep/18 22:28
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on a change in pull request #6244: 
[BEAM-3371] Enable running integration tests on Spark
URL: https://github.com/apache/beam/pull/6244#discussion_r218247934
 
 

 ##
 File path: 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java
 ##
 @@ -95,73 +83,33 @@ public void translate(Pipeline pipeline) {
 pipeline.replaceAll(
 FlinkTransformOverrides.getDefaultOverrides(translationMode == 
TranslationMode.STREAMING));
 
-// Local flink configurations work in the same JVM and have no problems 
with improperly
-// formatted files on classpath (eg. directories with .class files or 
empty directories).
-// prepareFilesToStage() only when using remote flink cluster.
-List filesToStage;
-if (!options.getFlinkMaster().matches("\\[.*\\]")) {
-  filesToStage = prepareFilesToStage();
-} else {
-  filesToStage = options.getFilesToStage();
-}
+prepareFilesToStageForRemoteClusterExecution();
 
 FlinkPipelineTranslator translator;
 if (translationMode == TranslationMode.STREAMING) {
   this.flinkStreamEnv =
-  FlinkExecutionEnvironments.createStreamExecutionEnvironment(options, 
filesToStage);
+  FlinkExecutionEnvironments.createStreamExecutionEnvironment(
+  options, options.getFilesToStage());
   translator = new FlinkStreamingPipelineTranslator(flinkStreamEnv, 
options);
 } else {
   this.flinkBatchEnv =
-  FlinkExecutionEnvironments.createBatchExecutionEnvironment(options, 
filesToStage);
+  FlinkExecutionEnvironments.createBatchExecutionEnvironment(
+  options, options.getFilesToStage());
   translator = new FlinkBatchPipelineTranslator(flinkBatchEnv, options);
 }
 
 translator.translate(pipeline);
   }
 
-  private List prepareFilesToStage() {
-return options
-.getFilesToStage()
-.stream()
-.map(File::new)
-.filter(File::exists)
-.map(file -> file.isDirectory() ? packageDirectoriesToStage(file) : 
file.getAbsolutePath())
-.collect(Collectors.toList());
-  }
-
-  private String packageDirectoriesToStage(File directoryToStage) {
-String hash = calculateDirectoryContentHash(directoryToStage);
-String pathForJar = getUniqueJarPath(hash);
-zipDirectory(directoryToStage, pathForJar);
-return pathForJar;
-  }
-
-  private String calculateDirectoryContentHash(File directoryToStage) {
-Hasher hasher = Hashing.md5().newHasher();
-try (OutputStream hashStream = Funnels.asOutputStream(hasher)) {
-  ZipFiles.zipDirectory(directoryToStage, hashStream);
-  return Base64Variants.MODIFIED_FOR_URL.encode(hasher.hash().asBytes());
-} catch (IOException e) {
-  throw new RuntimeException(e);
-}
-  }
-
-  private String getUniqueJarPath(String contentHash) {
-String tempLocation = options.getTempLocation();
-
-checkArgument(
-!Strings.isNullOrEmpty(tempLocation),
-"Please provide \"tempLocation\" pipeline option. Flink runner needs 
it to store jars "
-+ "made of directories that were on classpath.");
-
-return String.format("%s%s.jar", tempLocation, contentHash);
-  }
-
-  private void zipDirectory(File directoryToStage, String uniqueDirectoryPath) 
{
-try {
-  ZipFiles.zipDirectory(directoryToStage, new 
FileOutputStream(uniqueDirectoryPath));
-} catch (IOException e) {
-  throw new RuntimeException(e);
+  /**
+   * Local configurations work in the same JVM and have no problems with 
improperly formatted files
+   * on classpath (eg. directories with .class files or empty directories). 
Prepare files for
+   * staging only when using remote cluster.
+   */
+  private void prepareFilesToStageForRemoteClusterExecution() {
+if (!options.getFlinkMaster().matches("\\[.*\\]")) {
 
 Review comment:
   So the idea here is that we don't want to `prepareFilesForStaging()` when 
using local (and auto, which in turn can decide to run [local]). We only need 
to prepare resources like this (remove nonexistent paths, package test 
resources into jars etc) when we actually stage them to a remote runner. 
   
   I figured that it might be more descriptive when the code will be more 
explicit and mention "[auto]" and "[local]" here, wdyt? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For 

[jira] [Work logged] (BEAM-3371) Add ability to stage directories with compiled classes to Spark

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3371?focusedWorklogId=145097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145097
 ]

ASF GitHub Bot logged work on BEAM-3371:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:25
Start Date: 17/Sep/18 22:25
Worklog Time Spent: 10m 
  Work Description: lgajowy commented on a change in pull request #6244: 
[BEAM-3371] Enable running integration tests on Spark
URL: https://github.com/apache/beam/pull/6244#discussion_r218247312
 
 

 ##
 File path: 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/PipelineResourcesTest.java
 ##
 @@ -68,4 +75,42 @@ public void detectClassPathResourceWithNonFileResources() 
throws Exception {
 
 PipelineResources.detectClassPathResourcesToStage(classLoader);
   }
+
+  @Test
+  public void testRemovingNonexistentFilesFromFilesToStage() throws 
IOException {
+String nonexistentFilePath = tmpFolder.getRoot().getPath() + 
"/nonexistent/file";
+String temporaryLocation = tmpFolder.newFolder().getAbsolutePath();
+
+List filesToStage = new ArrayList<>();
+filesToStage.add(nonexistentFilePath);
+
+List result = 
PipelineResources.prepareFilesForStaging(filesToStage, temporaryLocation);
+
+assertThat(result, is(empty()));
 
 Review comment:
   ok, good idea. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145097)
Time Spent: 1h 50m  (was: 1h 40m)

> Add ability to stage directories with compiled classes to Spark
> ---
>
> Key: BEAM-3371
> URL: https://issues.apache.org/jira/browse/BEAM-3371
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-spark
>Reporter: Lukasz Gajowy
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> This one is basically the same issue as
>  [this Flink's one|https://issues.apache.org/jira/browse/BEAM-3370], except 
> of two things:
> - a detection of files to stage has to be provided in Spark, which is already 
> being developed [here|https://issues.apache.org/jira/browse/BEAM-981]
> - the test execution is not interrupted by FileNotFoundException but by *the 
> effect* of the directory not being staged (absence of test classes on the 
> Spark's classpath, hence ClassNotFoundException).
> Again, this probably could be resolved analogously as in flink, while 
> BEAM-981 issue is resolved. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=145096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145096
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:21
Start Date: 17/Sep/18 22:21
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6373: 
[BEAM-5288] Enhance Environment proto to support different types of environments
URL: https://github.com/apache/beam/pull/6373#discussion_r218246416
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -992,13 +992,15 @@ message SideInput {
 // An environment for executing UDFs. By default, an SDK container URL, but
 // can also be a process forked by a command, or an externally managed process.
 message Environment {
+  // Deprecated
+  string url = 1;
 
 Review comment:
   Is it necessary to keep it here as the first field vs. adding it last (since 
the plan is to remove it in the next release)? Also, what is the reason to keep 
it around?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145096)
Time Spent: 3h 10m  (was: 3h)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle #1062

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Add already passing modules to python 3 test list

[thw] Optional Flink job server parameters and logging cleanup.

--
[...truncated 19.04 MB...]
INFO: Adding 
View.AsSingleton/Combine.GloballyAsSingletonView/CreateDataflowView as step s9
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding Create123/Read(CreateSource) as step s10
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding OutputSideInputs as step s11
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Window.Into()/Window.Assign as step 
s12
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous) as step 
s13
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map 
as step s14
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign as step 
s15
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/GroupByKey as step 
s16
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GatherAllOutputs/Values/Values/Map as 
step s17
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/RewindowActuals/Window.Assign as step 
s18
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/KeyForDummy/AddKeys/Map as step s19
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveActualsTriggering/Flatten.PCollections as step 
s20
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Create.Values/Read(CreateSource) as 
step s21
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/WindowIntoDummy/Window.Assign as step 
s22
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding 
PAssert$33/GroupGlobally/RemoveDummyTriggering/Flatten.PCollections as step s23
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/FlattenDummyAndContents as step s24
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/NeverTrigger/Flatten.PCollections as 
step s25
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/GroupDummyAndContents as step s26
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/Values/Values/Map as step s27
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GroupGlobally/ParDo(Concat) as step s28
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/GetPane/Map as step s29
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/RunChecks as step s30
Sep 17, 2018 10:18:46 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding PAssert$33/VerifyAssertions/ParDo(DefaultConclude) as step s31
Sep 17, 2018 10:18:46 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-validates-runner-tests//viewtest0testsingletonsideinput-jenkins-0917221836-cc2e599f/output/results/staging/
Sep 17, 2018 10:18:46 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <71190 bytes, hash 

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618224#comment-16618224
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

>From Dataflow worker logs: 
I  finish  
I  Stopping BeamFnMapTaskExecutor, grabbing final metric updates 
I  Stopping BeamFnMapTaskExecutor, grabbing final metric updates 
I  Setting node annotation to enable volume controller attach/detach 
I  Pipeline stage consuming pubsub took 2m55s and default ack deadline is 10s. 
Consider increasing ack deadline for subscription 
projects/apache-beam-testing/subscriptions/psit_subscription_inputa3ed93d9-f6b0-420b-b5ee-cb887e9b51de
 
I  Setting node annotation to enable volume controller attach/detach 
I  Dataflow cancel 844138762903-comp...@developer.gserviceaccount.com 
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"844138762903-comp...@developer.gserviceaccount.com"},"requestMetadata":{"callerIp":"35.225.248.94","requestAttributes":{"time":"2018-09-17T20:18:05.162196575Z","auth":{}},"destinationAttributes":{}},"s…
 Dataflow cancel  844138762903-comp...@developer.gserviceaccount.com 
E  Lost lease for work with id 6186087141253459847 with expiration time: 
1537215545000, now: 1537215486508 
E  Lost lease for work with id 6186087141253459847 with expiration time: 
1537215545000, now: 1537215486508 
I  E0917 20:18:06.508733   193 work_service_client.cc:692] Lost lease for work 
with id 6186087141253459847 with expiration time: 1537215545000, now: 
1537215486508 
W  Unable to update setup work item 4601443631075438240 error: 
INVALID_ARGUMENT: Http(400) Bad Request 


> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=145093=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145093
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:15
Start Date: 17/Sep/18 22:15
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #6382: 
[BEAM-4449] Finish splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#discussion_r218245226
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
 ##
 @@ -54,29 +62,133 @@ public void onMatch(RelOptRuleCall call) {
 
   private static RelNode updateWindow(RelOptRuleCall call, Aggregate 
aggregate, Project project) {
 ImmutableBitSet groupByFields = aggregate.getGroupSet();
-List projectMapping = project.getProjects();
+ArrayList projects = new ArrayList(project.getProjects());
 
-Optional windowField = Optional.empty();
+WindowFn windowFn = null;
+int windowFieldIndex = -1;
 
 for (int groupFieldIndex : groupByFields.asList()) {
-  RexNode projNode = projectMapping.get(groupFieldIndex);
+  RexNode projNode = projects.get(groupFieldIndex);
   if (!(projNode instanceof RexCall)) {
 continue;
   }
 
-  windowField = AggregateWindowFactory.getWindowFieldAt((RexCall) 
projNode, groupFieldIndex);
+  RexCall rexCall = (RexCall) projNode;
+  WindowFn fn = createWindowFn(rexCall.getOperands(), rexCall.op.kind);
+  if (fn != null) {
 
 Review comment:
   My understanding is that `Optional` is for passing null through lambda 
expressions and maps. In this simple case it makes things more complicated.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145093)
Time Spent: 7h 50m  (was: 7h 40m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 7h 50m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618218#comment-16618218
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

*13:13:17* WARNING:root:Timing out on waiting for job 
2018-09-17_13_12_39-8645011813028270201 after 30 seconds

> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618217#comment-16618217
 ] 

Valentyn Tymofieiev edited comment on BEAM-5415 at 9/17/18 10:14 PM:
-

13:41:54 test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... 
ERROR
13:41:54 
13:41:54 ==
13:41:54 ERROR: test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
13:41:54 --
13:41:54 Traceback (most recent call last):
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
 line 812, in run
13:41:54 test(orig)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 45, in __call__
13:41:54 return self.run(*arg, **kwarg)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 133, in run
13:41:54 self.runTest(result)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 151, in runTest
13:41:54 test(result)
13:41:54   File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
13:41:54 return self.run(*args, **kwds)
13:41:54   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
13:41:54 testMethod()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
 line 161, in test_leader_board_it
13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
 line 345, in run
13:41:54 'total_score': 'INTEGER',
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 414, in __exit__
13:41:54 self.run().wait_until_finish()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 394, in run
13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 407, in run
13:41:54 return self.runner.run_pipeline(self)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
 line 68, in run_pipeline
13:41:54 self.result.cancel()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1167, in cancel
13:41:54 self._update_job()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1078, in _update_job
13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
 line 184, in wrapper
13:41:54 return fun(*args, **kwargs)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
 line 629, in get_job
13:41:54 response = self._client.projects_locations_jobs.Get(request)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
 line 659, in Get
13:41:54 config, request, global_params=global_params)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 720, in _RunMethod
13:41:54 http, http_request, **opts)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
 line 346, in MakeRequest
13:41:54 check_response_func=check_response_func)
13:41:54   File 

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618217#comment-16618217
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

*13:41:54* test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... 
ERROR*13:41:54* *13:41:54* 
==*13:41:54*
 ERROR: test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)*13:41:54*
 
--*13:41:54*
 Traceback (most recent call last):*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
 line 812, in run*13:41:54* test(orig)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 45, in __call__*13:41:54* return self.run(*arg, **kwarg)*13:41:54*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 133, in run*13:41:54* self.runTest(result)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 151, in runTest*13:41:54* test(result)*13:41:54*   File 
"/usr/lib/python2.7/unittest/case.py", line 393, in __call__*13:41:54* 
return self.run(*args, **kwds)*13:41:54*   File 
"/usr/lib/python2.7/unittest/case.py", line 329, in run*13:41:54* 
testMethod()*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
 line 161, in test_leader_board_it*13:41:54* 
self.test_pipeline.get_full_options_as_args(**extra_opts))*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
 line 345, in run*13:41:54* 'total_score': 'INTEGER',*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 414, in __exit__*13:41:54* self.run().wait_until_finish()*13:41:54*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 394, in run*13:41:54* self.to_runner_api(), self.runner, 
self._options).run(False)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 407, in run*13:41:54* return self.runner.run_pipeline(self)*13:41:54* 
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
 line 68, in run_pipeline*13:41:54* self.result.cancel()*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1167, in cancel*13:41:54* self._update_job()*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1078, in _update_job*13:41:54* self._job = 
self._runner.dataflow_client.get_job(self.job_id())*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
 line 184, in wrapper*13:41:54* return fun(*args, **kwargs)*13:41:54*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
 line 629, in get_job*13:41:54* response = 
self._client.projects_locations_jobs.Get(request)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
 line 659, in Get*13:41:54* config, request, 
global_params=global_params)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 720, in _RunMethod*13:41:54* http, http_request, **opts)*13:41:54*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
 line 346, in MakeRequest*13:41:54* 
check_response_func=check_response_func)*13:41:54*   File 

[jira] [Work logged] (BEAM-5414) grpcio-tools 1.15.0 proto generation breaks compatibility with latest protobuf 3.6.1

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5414?focusedWorklogId=145088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145088
 ]

ASF GitHub Bot logged work on BEAM-5414:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:10
Start Date: 17/Sep/18 22:10
Worklog Time Spent: 10m 
  Work Description: charlesccychen opened a new pull request #6419: 
[BEAM-5414] Latest grpcio-tools incompatible with latest protobuf 3.6.1
URL: https://github.com/apache/beam/pull/6419
 
 
   R: @chamikaramj 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145088)
Time Spent: 10m
Remaining Estimate: 0h

> grpcio-tools 1.15.0 proto generation breaks compatibility with latest 
> protobuf 3.6.1
> 
>
> Key: BEAM-5414
> URL: https://issues.apache.org/jira/browse/BEAM-5414
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently, the latest grpcio-tools 1.15.0 proto generation breaks 
> compatibility with the latest protobuf 3.6.1.  This means that any proto 
> generation needs to happen with grpcio-tools 1.14.2 until this is fixed.
> CC: [~angoenka]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch charlesccychen-patch-1 created (now deaaa76)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch charlesccychen-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at deaaa76  [BEAM-5414] Latest grpcio-tools incompatible with latest 
protobuf 3.6.1

This branch includes the following new commits:

 new deaaa76  [BEAM-5414] Latest grpcio-tools incompatible with latest 
protobuf 3.6.1

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.

[beam] 01/01: [BEAM-5414] Latest grpcio-tools incompatible with latest protobuf 3.6.1

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch charlesccychen-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git

commit deaaa76e5518ae05e3b6f6da5549d44a6e72000f
Author: Charles Chen 
AuthorDate: Mon Sep 17 15:09:42 2018 -0700

[BEAM-5414] Latest grpcio-tools incompatible with latest protobuf 3.6.1
---
 sdks/python/gen_protos.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/sdks/python/gen_protos.py b/sdks/python/gen_protos.py
index 5314bed..3ade761 100644
--- a/sdks/python/gen_protos.py
+++ b/sdks/python/gen_protos.py
@@ -32,7 +32,8 @@ import warnings
 
 import pkg_resources
 
-GRPC_TOOLS = 'grpcio-tools>=1.3.5,<2'
+# TODO(BEAM-5414): latest grpcio-tools incompatible with latest protobuf 3.6.1.
+GRPC_TOOLS = 'grpcio-tools>=1.3.5,<=1.14.2'
 
 BEAM_PROTO_PATHS = [
 os.path.join('..', '..', 'model', 'pipeline', 'src', 'main', 'proto'),

[jira] [Closed] (BEAM-5258) Investigate if we can disable Row type flattening in Calcite

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-5258.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Investigate if we can disable Row type flattening in Calcite
> 
>
> Key: BEAM-5258
> URL: https://issues.apache.org/jira/browse/BEAM-5258
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>
> Either disable the flattening in PlannerImpl or Flattener could be a good 
> start.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=145086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145086
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:05
Start Date: 17/Sep/18 22:05
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6382: 
[BEAM-4449] Finish splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#discussion_r218243029
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
 ##
 @@ -504,21 +501,6 @@ private static BeamSqlExpression 
getBeamSqlExpression(RexNode rexNode) {
   ret = new BeamSqlIsNotNullExpression(subExps.get(0));
   break;
 
-case "HOP":
-case "TUMBLE":
-case "SESSION":
-  ret = new BeamSqlWindowExpression(subExps, 
node.type.getSqlTypeName());
 
 Review comment:
   SGTM


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145086)
Time Spent: 7h 40m  (was: 7.5h)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 7h 40m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #6004

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=145084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145084
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:02
Start Date: 17/Sep/18 22:02
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #6382: 
[BEAM-4449] Finish splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#discussion_r218241789
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
 ##
 @@ -54,29 +61,133 @@ public void onMatch(RelOptRuleCall call) {
 
   private static RelNode updateWindow(RelOptRuleCall call, Aggregate 
aggregate, Project project) {
 ImmutableBitSet groupByFields = aggregate.getGroupSet();
-List projectMapping = project.getProjects();
+ArrayList projects = new ArrayList(project.getProjects());
 
-Optional windowField = Optional.empty();
+WindowFn windowFn = null;
+int windowFieldIndex = -1;
 
 for (int groupFieldIndex : groupByFields.asList()) {
-  RexNode projNode = projectMapping.get(groupFieldIndex);
+  RexNode projNode = projects.get(groupFieldIndex);
   if (!(projNode instanceof RexCall)) {
 continue;
   }
 
-  windowField = AggregateWindowFactory.getWindowFieldAt((RexCall) 
projNode, groupFieldIndex);
+  RexCall rexCall = (RexCall) projNode;
+  WindowFn fn = createWindowFn(rexCall.getOperands(), rexCall.op.kind);
+  if (fn != null) {
+windowFn = fn;
+windowFieldIndex = groupFieldIndex;
+projects.set(groupFieldIndex, rexCall.getOperands().get(0));
+  }
 }
 
+final Project newProject =
+project.copy(project.getTraitSet(), project.getInput(), projects, 
project.getRowType());
+
 return new BeamAggregationRel(
 aggregate.getCluster(),
 aggregate.getTraitSet().replace(BeamLogicalConvention.INSTANCE),
-convert(
-aggregate.getInput(),
-
aggregate.getInput().getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
+convert(newProject, 
newProject.getTraitSet().replace(BeamLogicalConvention.INSTANCE)),
 aggregate.indicator,
 aggregate.getGroupSet(),
 aggregate.getGroupSets(),
 aggregate.getAggCallList(),
-windowField);
+windowFn,
+windowFieldIndex);
+  }
+
+  /**
 
 Review comment:
   Is it possible to extract the windowing logic into a separate RelNode?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145084)
Time Spent: 7.5h  (was: 7h 20m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Batkhuyag Batsaikhan (JIRA)]

Batkhuyag Batsaikhan created BEAM-5415:
--

 Summary: [beam_PostCommit_Python_Verify] test is failing
 Key: BEAM-5415
 URL: https://issues.apache.org/jira/browse/BEAM-5415
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Batkhuyag Batsaikhan
Assignee: Valentyn Tymofieiev


Failing job url: 
[https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]

Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=145083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145083
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 17/Sep/18 22:02
Start Date: 17/Sep/18 22:02
Worklog Time Spent: 10m 
  Work Description: akedin commented on a change in pull request #6382: 
[BEAM-4449] Finish splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#discussion_r218242179
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamAggregationRule.java
 ##
 @@ -54,29 +62,133 @@ public void onMatch(RelOptRuleCall call) {
 
   private static RelNode updateWindow(RelOptRuleCall call, Aggregate 
aggregate, Project project) {
 ImmutableBitSet groupByFields = aggregate.getGroupSet();
-List projectMapping = project.getProjects();
+ArrayList projects = new ArrayList(project.getProjects());
 
-Optional windowField = Optional.empty();
+WindowFn windowFn = null;
+int windowFieldIndex = -1;
 
 for (int groupFieldIndex : groupByFields.asList()) {
-  RexNode projNode = projectMapping.get(groupFieldIndex);
+  RexNode projNode = projects.get(groupFieldIndex);
   if (!(projNode instanceof RexCall)) {
 continue;
   }
 
-  windowField = AggregateWindowFactory.getWindowFieldAt((RexCall) 
projNode, groupFieldIndex);
+  RexCall rexCall = (RexCall) projNode;
+  WindowFn fn = createWindowFn(rexCall.getOperands(), rexCall.op.kind);
+  if (fn != null) {
 
 Review comment:
   nit: this is exactly what `Optional` is for


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145083)
Time Spent: 7h 20m  (was: 7h 10m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 7h 20m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4841) Test Auto JIRA Subtask 789

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou updated BEAM-4841:

Description: 
2018-07-25 20:34:03.782739

Please review and upgrade the org.slf4j:slf4j-jdk14 to the latest version 
1.8.0-beta2

cc:

  was:Test JIRA issue creation


> Test Auto JIRA Subtask 789
> --
>
> Key: BEAM-4841
> URL: https://issues.apache.org/jira/browse/BEAM-4841
> Project: Beam
>  Issue Type: Sub-task
>  Components: dependencies
>Reporter: Beam JIRA Bot
>Priority: Major
> Fix For: 2.8.0
>
>
> 2018-07-25 20:34:03.782739
> Please review and upgrade the org.slf4j:slf4j-jdk14 to the latest version 
> 1.8.0-beta2
> cc:



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5414) grpcio-tools 1.15.0 proto generation breaks compatibility with latest protobuf 3.6.1

[Charles Chen (JIRA)]

Charles Chen created BEAM-5414:
--

 Summary: grpcio-tools 1.15.0 proto generation breaks compatibility 
with latest protobuf 3.6.1
 Key: BEAM-5414
 URL: https://issues.apache.org/jira/browse/BEAM-5414
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Charles Chen


Currently, the latest grpcio-tools 1.15.0 proto generation breaks compatibility 
with the latest protobuf 3.6.1.  This means that any proto generation needs to 
happen with grpcio-tools 1.14.2 until this is fixed.

CC: [~angoenka]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4176) Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4176?focusedWorklogId=145079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145079
 ]

ASF GitHub Bot logged work on BEAM-4176:


Author: ASF GitHub Bot
Created on: 17/Sep/18 21:36
Start Date: 17/Sep/18 21:36
Worklog Time Spent: 10m 
  Work Description: ryan-williams commented on issue #6328: [BEAM-4176] 
enumerate primitive transforms in portable construction
URL: https://github.com/apache/beam/pull/6328#issuecomment-422180055
 
 
   I just ran VR tests again on this PR and am seeing the original failure 
count; I suspect I made a mistake running the tests, but will update here as I 
learn more.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145079)
Time Spent: 19h 20m  (was: 19h 10m)

> Java: Portable batch runner passes all ValidatesRunner tests that 
> non-portable runner passes
> 
>
> Key: BEAM-4176
> URL: https://issues.apache.org/jira/browse/BEAM-4176
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Ben Sidhom
>Priority: Major
> Attachments: Screen Shot 2018-08-14 at 4.18.31 PM.png, Screen Shot 
> 2018-09-03 at 11.07.38 AM.png
>
>  Time Spent: 19h 20m
>  Remaining Estimate: 0h
>
> We need this as a sanity check that runner execution is correct.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_PVR_Flink_Gradle #43

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5376) Row interface doesn't support nullability on all fields.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5376?focusedWorklogId=145076=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145076
 ]

ASF GitHub Bot logged work on BEAM-5376:


Author: ASF GitHub Bot
Created on: 17/Sep/18 21:17
Start Date: 17/Sep/18 21:17
Worklog Time Spent: 10m 
  Work Description: apilloud closed pull request #6383: [BEAM-5376] Support 
nullability on all Row types
URL: https://github.com/apache/beam/pull/6383
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
index 2c12a7f1a60..df74a95b52c 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
@@ -83,7 +83,7 @@
* Get a {@link TypeName#BYTE} value by field name, {@link 
IllegalStateException} is thrown if
* schema doesn't match.
*/
-  public byte getByte(String fieldName) {
+  public Byte getByte(String fieldName) {
 return getByte(getSchema().indexOf(fieldName));
   }
 
@@ -99,7 +99,7 @@ public byte getByte(String fieldName) {
* Get a {@link TypeName#INT16} value by field name, {@link 
IllegalStateException} is thrown if
* schema doesn't match.
*/
-  public short getInt16(String fieldName) {
+  public Short getInt16(String fieldName) {
 return getInt16(getSchema().indexOf(fieldName));
   }
 
@@ -107,7 +107,7 @@ public short getInt16(String fieldName) {
* Get a {@link TypeName#INT32} value by field name, {@link 
IllegalStateException} is thrown if
* schema doesn't match.
*/
-  public int getInt32(String fieldName) {
+  public Integer getInt32(String fieldName) {
 return getInt32(getSchema().indexOf(fieldName));
   }
 
@@ -115,7 +115,7 @@ public int getInt32(String fieldName) {
* Get a {@link TypeName#INT64} value by field name, {@link 
IllegalStateException} is thrown if
* schema doesn't match.
*/
-  public long getInt64(String fieldName) {
+  public Long getInt64(String fieldName) {
 return getInt64(getSchema().indexOf(fieldName));
   }
 
@@ -131,7 +131,7 @@ public BigDecimal getDecimal(String fieldName) {
* Get a {@link TypeName#FLOAT} value by field name, {@link 
IllegalStateException} is thrown if
* schema doesn't match.
*/
-  public float getFloat(String fieldName) {
+  public Float getFloat(String fieldName) {
 return getFloat(getSchema().indexOf(fieldName));
   }
 
@@ -139,7 +139,7 @@ public float getFloat(String fieldName) {
* Get a {@link TypeName#DOUBLE} value by field name, {@link 
IllegalStateException} is thrown if
* schema doesn't match.
*/
-  public double getDouble(String fieldName) {
+  public Double getDouble(String fieldName) {
 return getDouble(getSchema().indexOf(fieldName));
   }
 
@@ -155,7 +155,7 @@ public String getString(String fieldName) {
* Get a {@link TypeName#DATETIME} value by field name, {@link 
IllegalStateException} is thrown if
* schema doesn't match.
*/
-  public ReadableDateTime getDateTime(String fieldName) {
+  public @Nullable ReadableDateTime getDateTime(String fieldName) {
 return getDateTime(getSchema().indexOf(fieldName));
   }
 
@@ -163,7 +163,7 @@ public ReadableDateTime getDateTime(String fieldName) {
* Get a {@link TypeName#BOOLEAN} value by field name, {@link 
IllegalStateException} is thrown if
* schema doesn't match.
*/
-  public boolean getBoolean(String fieldName) {
+  public Boolean getBoolean(String fieldName) {
 return getBoolean(getSchema().indexOf(fieldName));
   }
 
@@ -258,9 +258,9 @@ public String getString(int idx) {
* Get a {@link TypeName#DATETIME} value by field index, {@link 
IllegalStateException} is thrown
* if schema doesn't match.
*/
-  public ReadableDateTime getDateTime(int idx) {
+  public @Nullable ReadableDateTime getDateTime(int idx) {
 ReadableInstant instant = getValue(idx);
-return new DateTime(instant).withZone(instant.getZone());
+return instant == null ? null : new 
DateTime(instant).withZone(instant.getZone());
   }
 
   /**
@@ -275,7 +275,7 @@ public BigDecimal getDecimal(int idx) {
* Get a {@link Boolean} value by field index, {@link ClassCastException} is 
thrown if schema
* doesn't match.
*/
-  public boolean getBoolean(int idx) {
+  public Boolean getBoolean(int idx) {
 return getValue(idx);
   }
 
diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/JavaBeanSchemaTest.java
index 9ef7f82e2ba..14ec6740cfd 

[beam] branch master updated (733b6b5 -> f386050)

[apilloud]

This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 733b6b5  Merge pull request #6403 from RobbeSneyders/master
 add 0c64b3a  [BEAM-5376] Support nullability on all Row types
 new f386050  Merge pull request #6383 from apilloud/null

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/beam/sdk/values/Row.java  | 22 
 .../beam/sdk/schemas/JavaBeanSchemaTest.java   | 18 +++
 .../beam/sdk/schemas/JavaFieldSchemaTest.java  | 18 +++
 .../java/org/apache/beam/sdk/values/RowTest.java   | 58 +-
 4 files changed, 74 insertions(+), 42 deletions(-)

[beam] 01/01: Merge pull request #6383 from apilloud/null

[apilloud]

This is an automated email from the ASF dual-hosted git repository.

apilloud pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f386050c919029af4a689863cdf4c9dcdacd8869
Merge: 733b6b5 0c64b3a
Author: Andrew Pilloud 
AuthorDate: Mon Sep 17 14:17:44 2018 -0700

Merge pull request #6383 from apilloud/null

[BEAM-5376] Support nullability on all Row types

 .../main/java/org/apache/beam/sdk/values/Row.java  | 22 
 .../beam/sdk/schemas/JavaBeanSchemaTest.java   | 18 +++
 .../beam/sdk/schemas/JavaFieldSchemaTest.java  | 18 +++
 .../java/org/apache/beam/sdk/values/RowTest.java   | 58 +-
 4 files changed, 74 insertions(+), 42 deletions(-)

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=145075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145075
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 17/Sep/18 21:15
Start Date: 17/Sep/18 21:15
Worklog Time Spent: 10m 
  Work Description: apilloud commented on a change in pull request #6382: 
[BEAM-4449] Finish splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#discussion_r218230516
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
 ##
 @@ -504,21 +501,6 @@ private static BeamSqlExpression 
getBeamSqlExpression(RexNode rexNode) {
   ret = new BeamSqlIsNotNullExpression(subExps.get(0));
   break;
 
-case "HOP":
-case "TUMBLE":
-case "SESSION":
-  ret = new BeamSqlWindowExpression(subExps, 
node.type.getSqlTypeName());
 
 Review comment:
   It actually doesn't disable them. The implementation of 
`BeamSqlWindowExpression` just passes through the timestamp: 
https://github.com/apache/beam/pull/6382/files#diff-ec3df29f3fc44a5ade2f7f8ef33920ecL51
   ```
   (ReadableInstant) operands.get(0).evaluate(inputRow, window, 
env).getValue());
   ```
   
This is now happening directly at the project layer: 
https://github.com/apache/beam/pull/6382/files#diff-b0b2c6af2f694e5f6d9a1c64cbc482a1R81
   ```
   projects.set(groupFieldIndex, rexCall.getOperands().get(0));
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145075)
Time Spent: 7h 10m  (was: 7h)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 7h 10m
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5288) Modify Environment to support non-dockerized SDK harness deployments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5288?focusedWorklogId=145073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145073
 ]

ASF GitHub Bot logged work on BEAM-5288:


Author: ASF GitHub Bot
Created on: 17/Sep/18 21:04
Start Date: 17/Sep/18 21:04
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6373: [BEAM-5288] Enhance 
Environment proto to support different types of environments
URL: https://github.com/apache/beam/pull/6373#issuecomment-422170933
 
 
   Ready for review and merge.
   I have kept the readdition of url in environment proto to make removal 
easier later.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145073)
Time Spent: 3h  (was: 2h 50m)

> Modify Environment to support non-dockerized SDK harness deployments 
> -
>
> Key: BEAM-5288
> URL: https://issues.apache.org/jira/browse/BEAM-5288
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Maximilian Michels
>Assignee: Ankur Goenka
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> As of mailing discussions and BEAM-5187, it has become clear that we need to 
> extend the Environment information. In addition to the Docker environment, 
> the extended environment holds deployment options for 1) a process-based 
> environment, 2) an externally managed environment.
> The proto definition, as of now, looks as follows:
> {noformat}
>  message Environment {
>// (Required) The URN of the payload
>string urn = 1;
>// (Optional) The data specifying any parameters to the URN. If
>// the URN does not require any arguments, this may be omitted.
>bytes payload = 2;
>  }
>  message StandardEnvironments {
>enum Environments {
>  DOCKER = 0 [(beam_urn) = "beam:env:docker:v1"];
>  PROCESS = 1 [(beam_urn) = "beam:env:process:v1"];
>  EXTERNAL = 2 [(beam_urn) = "beam:env:external:v1"];
>}
>  }
>  // The payload of a Docker image
>  message DockerPayload {
>string container_image = 1;  // implicitly linux_amd64.
>  }
>  message ProcessPayload {
>string os = 1;  // "linux", "darwin", ..
>string arch = 2;  // "amd64", ..
>string command = 3; // process to execute
>map env = 4; // environment variables
>  }
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5112) Investigate if Calcite can generate functions we need

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5112?focusedWorklogId=145070=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145070
 ]

ASF GitHub Bot logged work on BEAM-5112:


Author: ASF GitHub Bot
Created on: 17/Sep/18 21:00
Start Date: 17/Sep/18 21:00
Worklog Time Spent: 10m 
  Work Description: apilloud opened a new pull request #6417: [BEAM-5112] 
[WIP] Use Calcite codegen to implement BeamCalcRel
URL: https://github.com/apache/beam/pull/6417
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145070)
Time Spent: 10m
Remaining Estimate: 0h

> Investigate if Calcite can generate functions we need
> -
>
> Key: BEAM-5112
> URL: https://issues.apache.org/jira/browse/BEAM-5112
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian 

Build failed in Jenkins: beam_PostCommit_Python_Verify #6003

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Add already passing modules to python 3 test list

--
[...truncated 1.09 MB...]
test_deterministic_coder (apache_beam.coders.coders_test_common.CodersTest) ... 
ok
test_dill_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_fast_primitives_coder (apache_beam.coders.coders_test_common.CodersTest) 
... ok
test_float_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_global_window_coder (apache_beam.coders.coders_test_common.CodersTest) ... 
ok
test_interval_window_coder (apache_beam.coders.coders_test_common.CodersTest) 
... ok
test_iterable_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_iterable_coder_unknown_length 
(apache_beam.coders.coders_test_common.CodersTest) ... ok
test_length_prefix_coder (apache_beam.coders.coders_test_common.CodersTest) ... 
ok
test_nested_observables (apache_beam.coders.coders_test_common.CodersTest) ... 
ok
test_pickle_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_proto_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_singleton_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_timestamp_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_tuple_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_tuple_sequence_coder (apache_beam.coders.coders_test_common.CodersTest) 
... ok
test_utf8_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_varint_coder (apache_beam.coders.coders_test_common.CodersTest) ... ok
test_windowed_value_coder (apache_beam.coders.coders_test_common.CodersTest) 
... ok
test_windowedvalue_coder_paneinfo 
(apache_beam.coders.coders_test_common.CodersTest) ... ok
test_observable (apache_beam.coders.observable_test.ObservableMixinTest) ... 
:50:
 DeprecationWarning: Please use assertEqual instead.
  self.assertEquals(3, self.observed_count)
ok
test_base64_pickle_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_bytes_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_custom_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_deterministic_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_dill_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_fast_primitives_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_float_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_global_window_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_interval_window_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_iterable_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_iterable_coder_unknown_length 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_length_prefix_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_nested_observables 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_pickle_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_proto_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_singleton_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_timestamp_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_tuple_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_tuple_sequence_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_utf8_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_varint_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_windowed_value_coder 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_windowedvalue_coder_paneinfo 
(apache_beam.coders.slow_coders_test.transplant_class..C) ... ok
test_using_slow_impl (apache_beam.coders.slow_coders_test.SlowCoders) ... ok
test_standard_coders 
(apache_beam.coders.standard_coders_test.StandardCodersTest) ... 
:50:
 ResourceWarning: unclosed file <_io.TextIOWrapper 
name='
 mode='r' encoding='UTF-8'>
  for ix, spec in enumerate(yaml.load_all(open(test_yaml))):
ok
test_byte_counting (apache_beam.coders.stream_test.StreamTest) ... 
:131:
 DeprecationWarning: 

[jira] [Work logged] (BEAM-4449) Use Calc instead of Project and Filter separately

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4449?focusedWorklogId=145065=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145065
 ]

ASF GitHub Bot logged work on BEAM-4449:


Author: ASF GitHub Bot
Created on: 17/Sep/18 20:57
Start Date: 17/Sep/18 20:57
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on a change in pull request #6382: 
[BEAM-4449] Finish splitting Aggregate and Project
URL: https://github.com/apache/beam/pull/6382#discussion_r218219186
 
 

 ##
 File path: 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/interpreter/BeamSqlFnExecutor.java
 ##
 @@ -504,21 +501,6 @@ private static BeamSqlExpression 
getBeamSqlExpression(RexNode rexNode) {
   ret = new BeamSqlIsNotNullExpression(subExps.get(0));
   break;
 
-case "HOP":
-case "TUMBLE":
-case "SESSION":
-  ret = new BeamSqlWindowExpression(subExps, 
node.type.getSqlTypeName());
 
 Review comment:
   This change seems disabling grouped auxiliary functions (HOP/TUMBLE/SESSION 
START/END). Will code generation add them back?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145065)
Time Spent: 7h  (was: 6h 50m)

> Use Calc instead of Project and Filter separately
> -
>
> Key: BEAM-4449
> URL: https://issues.apache.org/jira/browse/BEAM-4449
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Calcite has a combined Calc operator that is amenable to more optimization, 
> and also means less code to manage as we adjust how the operators/expressions 
> are implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5105) Move load job poll to finishBundle() method to better parallelize execution

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5105?focusedWorklogId=145059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145059
 ]

ASF GitHub Bot logged work on BEAM-5105:


Author: ASF GitHub Bot
Created on: 17/Sep/18 20:37
Start Date: 17/Sep/18 20:37
Worklog Time Spent: 10m 
  Work Description: reuvenlax opened a new pull request #6416: [BEAM-5105] 
Better parallelize BigQuery load jobs
URL: https://github.com/apache/beam/pull/6416
 
 
   Instead of blocking in every processElement, start the load jobs and wait 
for all of them in finishBundle.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145059)
Time Spent: 10m
Remaining Estimate: 0h

> Move load job poll to finishBundle() method to better parallelize execution
> ---
>
> Key: BEAM-5105
> URL: https://issues.apache.org/jira/browse/BEAM-5105
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-gcp
>Reporter: Chamikara Jayalath
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> It appears that when we write to BigQuery using WriteTablesDoFn we start a 
> load job and wait for that job to finish.
> [https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L318]
>  
> In cases where we are trying to write a PCollection of tables (for example, 
> when user use dynamic destinations feature) this relies on dynamic work 
> rebalancing to parallellize execution of load jobs. If the runner does not 
> support dynamic work rebalancing or does not execute dynamic work rebalancing 
> from some reason this could have significant performance drawbacks. For 
> example, scheduling times for load jobs will add up.
>  
> A better approach might be to start load jobs at process() method but wait 
> for all load jobs to finish at finishBundle() method. This will parallelize 
> any overheads as well as job execution (assuming more than one job is 
> schedule by BQ.).
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5411) Separate BeamUnnest and BeamCalc

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5411?focusedWorklogId=145056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145056
 ]

ASF GitHub Bot logged work on BEAM-5411:


Author: ASF GitHub Bot
Created on: 17/Sep/18 20:30
Start Date: 17/Sep/18 20:30
Worklog Time Spent: 10m 
  Work Description: apilloud opened a new pull request #6415: [BEAM-5411] 
Simplify BeamUnnest
URL: https://github.com/apache/beam/pull/6415
 
 
   This simplifies BeamUnnest by moving most of the logic out into the 
conversion rule. This decouples the operation from our Calc implementation. 
This has the added benefit of failing on unsupported operations at planning 
time rather than execution time.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | --- | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145056)
Time Spent: 10m
Remaining Estimate: 0h

> Separate BeamUnnest and BeamCalc
> 
>
> Key: BEAM-5411
> URL: https://issues.apache.org/jira/browse/BEAM-5411
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew 

[jira] [Comment Edited] (BEAM-5334) [beam_PerformanceTests_Python] Incompatible packages in dependencies

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617989#comment-16617989
 ] 

Mark Liu edited comment on BEAM-5334 at 9/17/18 8:07 PM:
-

https://github.com/apache/beam/pull/6392 enables Python benchmark to setup and 
run test with Gradle, which should be helpful to fix the failure. 
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1776 is the 
corresponding changes on Perfkit. Once that's in, we should be able to change 
Jenkins benchmark job to use Gradle for test setup and execution.


was (Author: markflyhigh):
https://github.com/apache/beam/pull/6392 enables Python benchmark to setup and 
run test with Gradle, which should be helpful to fix the failure. 
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/pull/1776 is the 
corresponding changes on Perfkit. Once that's in, we should be able to change 
Jenkins benchmark job to used Gradle for test setup and execution.

> [beam_PerformanceTests_Python] Incompatible packages in dependencies
> 
>
> Key: BEAM-5334
> URL: https://issues.apache.org/jira/browse/BEAM-5334
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Mikhail Gryzykhin
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Failing job url: 
> https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console
> Relevant log:
> 05:18:36 Obtaining 
> file:///home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/src/sdks/python
> 05:18:37 Complete output from command python setup.py egg_info:
> 05:18:37 
> /home/jenkins/jenkins-slave/workspace/beam_PerformanceTests_Python/env/.beam_env/local/lib/python2.7/site-packages/setuptools/dist.py:398:
>  UserWarning: Normalizing '2.8.0.dev' to '2.8.0.dev0'
> 05:18:37 normalized_version,
> 05:18:37 running egg_info
> 05:18:37 Requirement already satisfied: future==0.16.0 in 
> /home/jenkins/.local/lib/python2.7/site-packages (0.16.0)
> 05:18:37 gapic-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-datastore-v1 0.90.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 proto-google-cloud-pubsub-v1 0.15.4 has requirement 
> oauth2client<4.0dev,>=2.0.0, but you'll have oauth2client 4.1.2 which is 
> incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement httplib2<0.10,>=0.9.1, but 
> you'll have httplib2 0.11.3 which is incompatible.
> 05:18:37 googledatastore 7.0.1 has requirement oauth2client<4.0.0,>=2.0.1, 
> but you'll have oauth2client 4.1.2 which is incompatible.
> 05:18:37 You are using pip version 10.0.1, however version 18.0 is available.
> 05:18:37 You should consider upgrading via the 'pip install --upgrade pip' 
> command.
> 05:18:37 error: [Errno 2] No such file or directory



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?focusedWorklogId=145047=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145047
 ]

ASF GitHub Bot logged work on BEAM-5351:


Author: ASF GitHub Bot
Created on: 17/Sep/18 20:05
Start Date: 17/Sep/18 20:05
Worklog Time Spent: 10m 
  Work Description: charlesccychen closed pull request #6412: Cherrypick 
b9fec03 [BEAM-5351] Fix missing pom.xml file in artifact jar.
URL: https://github.com/apache/beam/pull/6412
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy 
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 27d6661226a..d0d74092670 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -769,10 +769,11 @@ class BeamModulePlugin implements Plugin {
 
 // Create a task which emulates the maven-archiver plugin in 
generating a
 // pom.properties file.
+def pomPropertiesFile = 
"${project.buildDir}/publications/mavenJava/pom.properties"
 project.task('generatePomPropertiesFileForMavenJavaPublication') {
-  outputs.file 
"${project.buildDir}/publications/mavenJava/pom.properties"
+  outputs.file "${pomPropertiesFile}"
   doLast {
-new 
File("${project.buildDir}/publications/mavenJava/pom.properties").text =
+new File("${pomPropertiesFile}").text =
 """version=${project.version}
 groupId=${project.group}
 artifactId=${project.name}
@@ -783,10 +784,23 @@ artifactId=${project.name}
 // Have the shaded include both the generate pom.xml and its 
properties file
 // emulating the behavior of the maven-archiver plugin.
 project.shadowJar {
+  def pomFile = 
"${project.buildDir}/publications/mavenJava/pom-default.xml"
+
+  // Validate that the artifacts exist before copying them into the 
jar.
+  doFirst {
+if (!project.file("${pomFile}").exists()) {
+  throw new GradleException("Expected ${pomFile} to have been 
generated by the 'generatePomFileForMavenJavaPublication' task.")
+}
+if (!project.file("${pomPropertiesFile}").exists()) {
+  throw new GradleException("Expected ${pomPropertiesFile} to have 
been generated by the 'generatePomPropertiesFileForMavenJavaPublication' task.")
+}
+  }
+
   dependsOn 'generatePomFileForMavenJavaPublication'
-  into("META-INF/maven/${project.group}/${project.name}") { from 
"${project.buildDir}/publications/mavenJava/pom.xml" }
+  into("META-INF/maven/${project.group}/${project.name}") { from 
"${pomFile}" }
+
   dependsOn project.generatePomPropertiesFileForMavenJavaPublication
-  into("META-INF/maven/${project.group}/${project.name}") { from 
"${project.buildDir}/publications/mavenJava/pom.properties" }
+  into("META-INF/maven/${project.group}/${project.name}") { from 
"${pomPropertiesFile}" }
 }
 
 // Only build artifacts for archives if we are publishing


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145047)
Time Spent: 1h 20m  (was: 1h 10m)

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.8.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6412 from lukecwik/release270

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a commit to branch release-2.7.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit f29e74a4c3382ed072cbc6d5e5b236612ee3601d
Merge: c7f890b 81ab27d
Author: Charles Chen 
AuthorDate: Mon Sep 17 13:05:53 2018 -0700

Merge pull request #6412 from lukecwik/release270

Cherrypick b9fec03 [BEAM-5351] Fix missing pom.xml file in artifact jar.

 .../org/apache/beam/gradle/BeamModulePlugin.groovy | 22 ++
 1 file changed, 18 insertions(+), 4 deletions(-)

[jira] [Work logged] (BEAM-5351) 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5351?focusedWorklogId=145046=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145046
 ]

ASF GitHub Bot logged work on BEAM-5351:


Author: ASF GitHub Bot
Created on: 17/Sep/18 20:05
Start Date: 17/Sep/18 20:05
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #6412: Cherrypick 
b9fec03 [BEAM-5351] Fix missing pom.xml file in artifact jar.
URL: https://github.com/apache/beam/pull/6412#issuecomment-422151945
 
 
   Thanks, this LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 145046)
Time Spent: 1h 10m  (was: 1h)

> 2.7.0 RC1 jars missing META-INF/maven/groupId/artifactId/pom.xml
> 
>
> Key: BEAM-5351
> URL: https://issues.apache.org/jira/browse/BEAM-5351
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Affects Versions: 2.7.0
>Reporter: Luke Cwik
>Assignee: Charles Chen
>Priority: Major
> Fix For: 2.8.0
>
> Attachments: beam-examples-java-2.7.0.jar
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> It seems as though **generatePomFileForMavenJavaPublication** pom.xml was 
> renamed to pom-default.xml as output which broke the the shadowJar task 
> [here|https://github.com/apache/beam/blob/883f01536440bdf9b582d785b6e609a4a90cfdff/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L788].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.7.0 updated (c7f890b -> f29e74a)

[ccy]

This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch release-2.7.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c7f890b  [Gradle Release Plugin] - new version commit:  
'v2.7.0-SNAPSHOT'.
 add 81ab27d  [BEAM-5351] Fix missing pom.xml file in artifact jar.
 new f29e74a  Merge pull request #6412 from lukecwik/release270

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/gradle/BeamModulePlugin.groovy | 22 ++
 1 file changed, 18 insertions(+), 4 deletions(-)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1484

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_PVR_Flink_Gradle #42

[Apache Jenkins Server]

See 


Changes:

[robbe.sneyders] Add already passing modules to python 3 test list

--
[...truncated 6.32 MB...]
[grpc-default-executor-0] INFO sdk_worker.__init__ - Creating insecure control 
channel.
[grpc-default-executor-0] INFO sdk_worker.__init__ - Control channel 
established.
[grpc-default-executor-0] INFO sdk_worker.__init__ - Initializing SDKHarness 
with 12 workers.
[grpc-default-executor-0] INFO 
org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - 
Beam Fn Control client connected with id 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 1
[grpc-default-executor-0] INFO sdk_worker.run - Got work 3
[grpc-default-executor-0] INFO sdk_worker.run - Got work 6
[grpc-default-executor-0] INFO sdk_worker.run - Got work 2
[grpc-default-executor-1] INFO sdk_worker.run - Got work 5
[grpc-default-executor-1] INFO sdk_worker.run - Got work 4
[grpc-default-executor-1] INFO sdk_worker.run - Got work 7
[grpc-default-executor-1] INFO sdk_worker.create_state_handler - Creating 
channel for localhost:37969
[grpc-default-executor-0] INFO sdk_worker.run - Got work 8
[grpc-default-executor-0] INFO data_plane.create_data_channel - Creating 
channel for localhost:38081
[grpc-default-executor-0] INFO 
org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn Data client 
connected.
[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-0] INFO bundle_processor.process_bundle - start 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (5085f8d358e64049a75d1fc848106617) switched from 
RUNNING to FINISHED.
[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Freeing task resources for Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (5085f8d358e64049a75d1fc848106617).
[Source: Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1)] INFO org.apache.flink.runtime.taskmanager.Task - 
Ensuring all FileSystem streams are closed for task Source: Collection Source 
-> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (5085f8d358e64049a75d1fc848106617) [FINISHED]
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task Source: 
Collection Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem 5085f8d358e64049a75d1fc848106617.
[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[grpc-default-executor-1] INFO bundle_processor.process_bundle - finish 

[flink-akka.actor.default-dispatcher-3] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Collection 
Source -> 
19Create/Read/Impulse.None/jenkins-docker-apache.bintray.io/beam/python:latest:0
 -> ToKeyedWorkItem (1/1) (5085f8d358e64049a75d1fc848106617) switched from 
RUNNING to FINISHED.
[grpc-default-executor-1] INFO sdk_worker.run - Got work 9
[Source: Collection Source -> 

[jira] [Updated] (BEAM-5413) Add method for defining composite transforms as lambda expressions

[Jeff Klukas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Klukas updated BEAM-5413:
--
Description: 
Defining a composite transform today requires writing a full named subclass of 
PTransform (as [the programming guide 
documents|[https://beam.apache.org/documentation/programming-guide/#composite-transforms]]
 but there are cases where users may want to define a fairly trivial composite 
transform using a less verbose Java 8 lambda expression.

Consider an example where the user has defined MyDeserializationTransform that 
attempts to deserialize byte arrays into some object, returning a 
PCollectionTuple  with tags for successfully deserialized records (mainTag) and 
for errors (errorTag).

If we introduce a PTransform::compose method that takes in a 
SerializableFunction, the user can handle errors in a small lambda expression:

 
{code:java}
byteArrays
    .apply("attempt to deserialize messages", 
new MyDeserializationTransform())
    .apply("write deserialization errors",
    PTransform.compose((PCollectionTuple input) -> {
    input
  .get(errorTag)
  .apply(new MyErrorOutputTransform());
    return input.get(mainTag);
    })
    .apply("more processing on the deserialized messages", 
 new MyOtherTransform())
{code}
This style allows a more concise and fluent pipeline definition than is 
currently possible.

 

  was:
Defining a composite transform today requires writing a full named subclass of 
PTransform (as [the programming guide 
documents|[https://beam.apache.org/documentation/programming-guide/#composite-transforms]),]
 but there are cases where users may want to define a fairly trivial composite 
transform using a less verbose Java 8 lambda expression.

Consider an example where the user has defined MyDeserializationTransform that 
attempts to deserialize byte arrays into some object, returning a 
PCollectionTuple  with tags for successfully deserialized records (mainTag) and 
for errors (errorTag).

If we introduce a PTransform::compose method that takes in a 
SerializableFunction, the user can handle errors in a small lambda expression:

 
{code:java}
byteArrays
    .apply("attempt to deserialize messages", 
new MyDeserializationTransform())
    .apply("write deserialization errors",
    PTransform.compose((PCollectionTuple input) -> {
    input
  .get(errorTag)
  .apply(new MyErrorOutputTransform());
    return input.get(mainTag);
    })
    .apply("more processing on the deserialized messages", 
 new MyOtherTransform())
{code}
This style allows a more concise and fluent pipeline definition than is 
currently possible.

 


> Add method for defining composite transforms as lambda expressions
> --
>
> Key: BEAM-5413
> URL: https://issues.apache.org/jira/browse/BEAM-5413
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Jeff Klukas
>Assignee: Kenneth Knowles
>Priority: Minor
> Fix For: 2.8.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Defining a composite transform today requires writing a full named subclass 
> of PTransform (as [the programming guide 
> documents|[https://beam.apache.org/documentation/programming-guide/#composite-transforms]]
>  but there are cases where users may want to define a fairly trivial 
> composite transform using a less verbose Java 8 lambda expression.
> Consider an example where the user has defined MyDeserializationTransform 
> that attempts to deserialize byte arrays into some object, returning a 
> PCollectionTuple  with tags for successfully deserialized records (mainTag) 
> and for errors (errorTag).
> If we introduce a PTransform::compose method that takes in a 
> SerializableFunction, the user can handle errors in a small lambda expression:
>  
> {code:java}
> byteArrays
>     .apply("attempt to deserialize messages", 
> new MyDeserializationTransform())
>     .apply("write deserialization errors",
>     PTransform.compose((PCollectionTuple input) -> {
>     input
>   .get(errorTag)
>   .apply(new MyErrorOutputTransform());
>     return input.get(mainTag);
>     })
>     .apply("more processing on the deserialized messages", 
>  new MyOtherTransform())
> {code}
> This style allows a more concise and fluent pipeline definition than is 
> currently possible.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)