[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90694=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90694
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 13/Apr/18 04:51
Start Date: 13/Apr/18 04:51
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #4991: [BEAM-3983] [SQL] 
Tables interface supports BigQuery
URL: https://github.com/apache/beam/pull/4991#issuecomment-381023134
 
 
   The default action is changeable on a per user basis. The initial default is 
`Create a merge commit`, sounds like the documentation needs an update.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90694)
Time Spent: 4.5h  (was: 4h 20m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2990) support data type MAP

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90693
 ]

ASF GitHub Bot logged work on BEAM-2990:


Author: ASF GitHub Bot
Created on: 13/Apr/18 04:44
Start Date: 13/Apr/18 04:44
Worklog Time Spent: 10m 
  Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support 
MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#issuecomment-381021910
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90693)
Time Spent: 3h 40m  (was: 3.5h)

> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP   Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2990) support data type MAP

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2990?focusedWorklogId=90692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90692
 ]

ASF GitHub Bot logged work on BEAM-2990:


Author: ASF GitHub Bot
Created on: 13/Apr/18 04:42
Start Date: 13/Apr/18 04:42
Worklog Time Spent: 10m 
  Work Description: XuMingmin commented on issue #5079: [BEAM-2990] support 
MAP in SQL schema
URL: https://github.com/apache/beam/pull/5079#issuecomment-381021696
 
 
   any comments on the change? Would like to close this PR asap as my 
repository is broken after #4964 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90692)
Time Spent: 3.5h  (was: 3h 20m)

> support data type MAP
> -
>
> Key: BEAM-2990
> URL: https://issues.apache.org/jira/browse/BEAM-2990
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Xu Mingmin
>Assignee: Xu Mingmin
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> support Non-scalar types:
> MAP   Collection of keys mapped to values
> ARRAY Ordered, contiguous collection that may contain duplicates



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90665
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 13/Apr/18 01:30
Start Date: 13/Apr/18 01:30
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5024: 
[BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181264323
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -510,7 +551,48 @@ def match(self, patterns, limits=None):
 Raises:
   ``BeamIOError`` if any of the pattern match operations fail
 """
-raise NotImplementedError
+if limits is None:
+  limits = [None] * len(patterns)
+else:
+  err_msg = "Patterns and limits should be equal in length"
+  assert len(patterns) == len(limits), err_msg
+
+def _match(pattern, limit):
+  """Find all matching paths to the pattern provided."""
+  if pattern.endswith('/'):
+pattern += '*'
+  prefix_or_dir = re.match('^[^[*?]*', pattern).group(0)
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90665)
Time Spent: 2h  (was: 1h 50m)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90667
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 13/Apr/18 01:30
Start Date: 13/Apr/18 01:30
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5024: 
[BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181263543
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -498,9 +501,47 @@ def mkdirs(self, path):
 raise NotImplementedError
 
   @abc.abstractmethod
+  def has_dirs(self):
+"""Whether this FileSystem supports directories."""
+raise NotImplementedError
+
+  @abc.abstractmethod
+  def list(self, dir_or_prefix):
 
 Review comment:
   done


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90667)
Time Spent: 2h 20m  (was: 2h 10m)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90666
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 13/Apr/18 01:30
Start Date: 13/Apr/18 01:30
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5024: 
[BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181263847
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -498,9 +501,47 @@ def mkdirs(self, path):
 raise NotImplementedError
 
   @abc.abstractmethod
+  def has_dirs(self):
+"""Whether this FileSystem supports directories."""
+raise NotImplementedError
+
+  @abc.abstractmethod
+  def list(self, dir_or_prefix):
+"""List files in a location.
+
+Listing is non-recursive (for filesystems that support directories).
+
+Args:
+  dir_or_prefix: (string) A directory or location prefix (for filesystems
+that don't have directories).
+
+Returns:
+  Generator of ``FileMetadata`` objects.
+
+Raises:
+  ``BeamIOError`` if listing fails, but not if no files were found.
+"""
+raise NotImplementedError
+
+  @staticmethod
+  def _url_dirname(url_or_path):
+"""Like posixpath.dirname, but preserves scheme:// prefix.
+
+Args:
+  url_or_path: A string in the form of scheme://some/path OR /some/path.
+"""
+match = re.match(r'([a-z]+://)(.*)', url_or_path)
+if match is None:
+  return posixpath.dirname(url_or_path)
+url_prefix, path = match.groups()
+return url_prefix + posixpath.dirname(path)
+
   def match(self, patterns, limits=None):
 """Find all matching paths to the patterns provided.
 
+Pattern matching is done using fnmatch.fnmatch.
 
 Review comment:
   Clarified.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90666)
Time Spent: 2h 10m  (was: 2h)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90664
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 13/Apr/18 01:30
Start Date: 13/Apr/18 01:30
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5024: 
[BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181265403
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -579,6 +661,19 @@ def exists(self, path):
 raise NotImplementedError
 
   @abc.abstractmethod
+  def size(self, path):
 
 Review comment:
   _list() returns sizes, but it only works on prefixes (may return more than 
one result) or directories (fails on files).
   I added this method for the case where the pattern given doesn't end in `/` 
and has no globbing characters. In this case we return the size of the file or 
directory pointed to by the pattern (if it exists).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90664)
Time Spent: 2h  (was: 1h 50m)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90663
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 13/Apr/18 01:30
Start Date: 13/Apr/18 01:30
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5024: 
[BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181264935
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -510,7 +551,48 @@ def match(self, patterns, limits=None):
 Raises:
   ``BeamIOError`` if any of the pattern match operations fail
 """
-raise NotImplementedError
+if limits is None:
+  limits = [None] * len(patterns)
+else:
+  err_msg = "Patterns and limits should be equal in length"
+  assert len(patterns) == len(limits), err_msg
+
+def _match(pattern, limit):
+  """Find all matching paths to the pattern provided."""
+  if pattern.endswith('/'):
+pattern += '*'
+  prefix_or_dir = re.match('^[^[*?]*', pattern).group(0)
+
+  file_metadatas = []
+  if prefix_or_dir == pattern:
+# Short-circuit calling self.list() if there's no glob pattern to 
match.
+if self.exists(pattern):
+  file_metadatas = [FileMetadata(pattern, self.size(pattern))]
+  else:
+if self.has_dirs():
 
 Review comment:
   Yes. I explained it in the new comment a little above this line.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90663)
Time Spent: 1h 50m  (was: 1h 40m)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark_Gradle #82

[Apache Jenkins Server]

See 


Changes:

[tgroh] Remove AppliedPTransform methods in WatermarkManager

[tgroh] Add an ExecutableGraph interface

[tgroh] Genericise WatermarkManager and Dependencies

--
[...truncated 1.23 MB...]
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:79)
at 
org.apache.beam.runners.spark.translation.streaming.SparkRunnerStreamingContextFactory.call(SparkRunnerStreamingContextFactory.java:47)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:627)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$$anonfun$7.apply(JavaStreamingContext.scala:626)
at scala.Option.getOrElse(Option.scala:121)
at 
org.apache.spark.streaming.StreamingContext$.getOrCreate(StreamingContext.scala:828)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext$.getOrCreate(JavaStreamingContext.scala:626)
at 
org.apache.spark.streaming.api.java.JavaStreamingContext.getOrCreate(JavaStreamingContext.scala)
at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:169)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:123)
at 
org.apache.beam.runners.spark.TestSparkRunner.run(TestSparkRunner.java:83)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:346)
at org.apache.beam.sdk.testing.TestPipeline.run(TestPipeline.java:328)
at 
org.apache.beam.runners.spark.translation.streaming.CreateStreamTest.testFirstElementLate(CreateStreamTest.java:240)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at 
org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:317)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy3.processTestClass(Unknown Source)
at 

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #59

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-4063) Flink runner supports cluster-wide artifact deployments through the Distributed Cache

[Ben Sidhom (JIRA)]

Ben Sidhom created BEAM-4063:


 Summary: Flink runner supports cluster-wide artifact deployments 
through the Distributed Cache
 Key: BEAM-4063
 URL: https://issues.apache.org/jira/browse/BEAM-4063
 Project: Beam
  Issue Type: New Feature
  Components: runner-flink
Reporter: Ben Sidhom
Assignee: Aljoscha Krettek


As of now, Flink effectively has a dependency on an external storage system for 
artifact management. This is because the Flink Distributed Cache does not 
actually distribute and cache blobs itself, but rather expects that each node 
in a running cluster has access to a well-known artifact resource.

We should get this for free whenever 
[https://github.com/apache/flink/pull/5580] is merged (likely in 1.5). For now, 
we will have to defer to external storage systems like GCS or HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1583

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a ModelCoders class for Coder Utilities

[tgroh] Use ModelCoders in LengthPrefixUnknownCoders

[tgroh] Format UrnUtils

[XuMingmin] [BEAM-3983] [SQL] Tables interface supports BigQuery (#4991)

[kedin] [SQL] Infer Row type from input elements

--
[...truncated 98.15 KB...]
'apache-beam-testing:bqjob_r5369b60426b8b778_0162bc7ef824_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r5369b60426b8b778_0162bc7ef824_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r5369b60426b8b778_0162bc7ef824_1 ... (0s) Current status: DONE   
2018-04-13 00:54:22,222 f9036bd4 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-13 00:54:39,385 f9036bd4 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-13 00:54:41,607 f9036bd4 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r16a7583cb12c1244_0162bc7f437b_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r16a7583cb12c1244_0162bc7f437b_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r16a7583cb12c1244_0162bc7f437b_1 ... (0s) Current status: DONE   
2018-04-13 00:54:41,608 f9036bd4 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-13 00:55:01,051 f9036bd4 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-13 00:55:03,180 f9036bd4 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r93ec73a6c9f4946_0162bc7f9802_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r93ec73a6c9f4946_0162bc7f9802_1 ... (0s) 
Current status: RUNNING 
Waiting on 
bqjob_r93ec73a6c9f4946_0162bc7f9802_1 ... (0s) Current status: DONE   
2018-04-13 00:55:03,180 f9036bd4 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-13 00:55:22,426 f9036bd4 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-13 00:55:24,672 f9036bd4 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  

[jira] [Work logged] (BEAM-4019) Refactor HBaseIO splitting to produce ByteKeyRange objects

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4019?focusedWorklogId=90659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90659
 ]

ASF GitHub Bot logged work on BEAM-4019:


Author: ASF GitHub Bot
Created on: 13/Apr/18 00:58
Start Date: 13/Apr/18 00:58
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #5081: 
[BEAM-4019] Refactor HBaseIO splitting to produce ByteKeyRange objects
URL: https://github.com/apache/beam/pull/5081#discussion_r181261877
 
 

 ##
 File path: 
sdks/java/io/hbase/src/main/java/org/apache/beam/sdk/io/hbase/HBaseIO.java
 ##
 @@ -420,17 +313,38 @@ private long estimateSizeBytes() throws Exception {
   }
 
   try (Connection connection = 
ConnectionFactory.createConnection(read.getConfiguration())) {
-List regionLocations = getRegionLocations(connection);
-int realNumSplits = numSplits < regionLocations.size() ? 
regionLocations.size() : numSplits;
-LOG.debug("Suggested {} bundle(s) based on size", numSplits);
-LOG.debug("Suggested {} bundle(s) based on number of regions", 
regionLocations.size());
-final List sources = splitBasedOnRegions(regionLocations, 
realNumSplits);
-LOG.debug("Split into {} bundle(s)", sources.size());
-if (numSplits >= 1) {
+List regionLocations =
+HBaseUtils.getRegionLocations(connection, read.tableId, 
read.serializableScan.get());
+LOG.debug("Suggested {} source(s) based on size", numSplits);
+LOG.debug("Suggested {} source(s) based on number of regions", 
regionLocations.size());
+
+List ranges =
+HBaseUtils.getRanges(
+regionLocations, read.tableId, read.serializableScan.get());
+final int numSources = ranges.size();
+LOG.debug("Spliting into {} source(s)", numSources);
+if (numSources >= 1) {
 
 Review comment:
   `if numSources > 0` ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90659)
Time Spent: 1h  (was: 50m)

> Refactor HBaseIO splitting to produce ByteKeyRange objects
> --
>
> Key: BEAM-4019
> URL: https://issues.apache.org/jira/browse/BEAM-4019
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-hbase
>Reporter: Ismaël Mejía
>Assignee: Ismaël Mejía
>Priority: Minor
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> This allows to reuse the splitting logic for a future SDF-based 
> implementation by reusing it as part of the @SplitRestriction method.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #1141

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a ModelCoders class for Coder Utilities

[tgroh] Use ModelCoders in LengthPrefixUnknownCoders

[tgroh] Format UrnUtils

[XuMingmin] [BEAM-3983] [SQL] Tables interface supports BigQuery (#4991)

[kedin] [SQL] Infer Row type from input elements

--
[...truncated 1.73 KB...]
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2501330811125783405.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3397838017857592644.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7634402207774371508.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5609036226433070593.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3346274903289497878.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: urllib3<1.22,>=1.21.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.9.1->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #43

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a ModelCoders class for Coder Utilities

[tgroh] Use ModelCoders in LengthPrefixUnknownCoders

[tgroh] Format UrnUtils

[XuMingmin] [BEAM-3983] [SQL] Tables interface supports BigQuery (#4991)

[kedin] [SQL] Infer Row type from input elements

--
[...truncated 34.45 KB...]
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 

[beam] branch master updated (b696435 -> 20ccd78)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b696435  Merge pull request #5090: Add a ModelCoders class for Coder 
Utilities
 add d95d3f6  Remove AppliedPTransform methods in WatermarkManager
 add 7803753  Add an ExecutableGraph interface
 add ed313ae  Genericise WatermarkManager and Dependencies
 new 20ccd78  Merge pull request #5060: Reduce Usage of AppliedPTransform 
Methods in the DirectRunner

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/direct/CommittedResult.java   |  12 +-
 .../apache/beam/runners/direct/DirectGraph.java|  69 +
 .../beam/runners/direct/DirectGraphVisitor.java|   2 +-
 .../beam/runners/direct/EvaluationContext.java |  17 +-
 ...LifecycleManagers.java => ExecutableGraph.java} |  38 +++--
 .../beam/runners/direct/QuiescenceDriver.java  |   8 +-
 .../beam/runners/direct/WatermarkManager.java  | 171 ++---
 .../beam/runners/direct/CommittedResultTest.java   |  10 +-
 .../apache/beam/runners/direct/DirectGraphs.java   |  10 +-
 .../beam/runners/direct/EvaluationContextTest.java |   6 +-
 .../beam/runners/direct/WatermarkManagerTest.java  |  63 +---
 11 files changed, 204 insertions(+), 202 deletions(-)
 copy 
runners/direct-java/src/main/java/org/apache/beam/runners/direct/{DoFnLifecycleManagers.java
 => ExecutableGraph.java} (51%)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #5060: Reduce Usage of AppliedPTransform Methods in the DirectRunner

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 20ccd78500603531bc47d82bbedbdc2c97327276
Merge: b696435 ed313ae
Author: Thomas Groh 
AuthorDate: Thu Apr 12 17:41:23 2018 -0700

Merge pull request #5060: Reduce Usage of AppliedPTransform Methods in the 
DirectRunner

 .../beam/runners/direct/CommittedResult.java   |  12 +-
 .../apache/beam/runners/direct/DirectGraph.java|  69 +
 .../beam/runners/direct/DirectGraphVisitor.java|   2 +-
 .../beam/runners/direct/EvaluationContext.java |  17 +-
 .../beam/runners/direct/ExecutableGraph.java   |  43 ++
 .../beam/runners/direct/QuiescenceDriver.java  |   8 +-
 .../beam/runners/direct/WatermarkManager.java  | 171 ++---
 .../beam/runners/direct/CommittedResultTest.java   |  10 +-
 .../apache/beam/runners/direct/DirectGraphs.java   |  10 +-
 .../beam/runners/direct/EvaluationContextTest.java |   6 +-
 .../beam/runners/direct/WatermarkManagerTest.java  |  63 +---
 11 files changed, 229 insertions(+), 182 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-3036) Checkstyle error in IntelliJ

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3036?focusedWorklogId=90652=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90652
 ]

ASF GitHub Bot logged work on BEAM-3036:


Author: ASF GitHub Bot
Created on: 13/Apr/18 00:36
Start Date: 13/Apr/18 00:36
Worklog Time Spent: 10m 
  Work Description: youngoli opened a new pull request #420: [BEAM-3036] 
Adding Checkstyle version warning.
URL: https://github.com/apache/beam-site/pull/420
 
 
   Some people are experiencing Checkstyle errors due to differences in
   checkstyle.xml syntax between IntelliJ and the checkstyle version
   used for validating Beam. Since the errors cannot be fixed by editing
   checkstyle.xml, only by updating IntelliJ, I'm suggesting this warning.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90652)
Time Spent: 50m  (was: 40m)

> Checkstyle error in IntelliJ
> 
>
> Key: BEAM-3036
> URL: https://issues.apache.org/jira/browse/BEAM-3036
> Project: Beam
>  Issue Type: Task
>  Components: website
>Reporter: Kwang-in (Dennis) JUNG
>Assignee: Daniel Oliveira
>Priority: Trivial
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Hello.
> I'm following Beam contribution guide, and encountered at some bug when 
> setting checkstyle.
> 1.Install the “Checkstyle-IDEA” plugin from the IntelliJ plugin repository.
> 2.Configure the plugin by going to Settings -> Other Settings -> Checkstyle.
> 3.Set the “Scan Scope” to “Only Java sources (including tests)”.
> 4.In the “Configuration File” pane, add a new configuration using the plus 
> icon:
>   4-1.Set the “Description” to “Beam”.
>   4-2.Select “Use a local Checkstyle file”, and point it to 
> “sdks/java/build-tools/src/main/resources/beam/checkstyle.xml” within your 
> repository.
>   4-3.Check the box for “Store relative to project location”, and click 
> “Next”.
>   4-4.Configure the “checkstyle.suppressions.file” property value to 
> “suppressions.xml”, and click “Next”, then “Finish”.
> After I click Next in 4-4, it shows error below in dialog box.
> ```
> org.infernus.idea.checkstyle.exception.CheckStylePluginException: 
> The Checkstyle rules file could not be 
> parsed.SuppressionCommentFilter is not allowed as a child in 
> CheckerThe file has been blacklisted for 60s.
>   at 
> org.infernus.idea.checkstyle.checker.CheckerFactory.blacklistAndShowMessage(CheckerFactory.java:198)
>   at 
> org.infernus.idea.checkstyle.checker.CheckerFactory.blacklistAndShowMessage(CheckerFactory.java:214)
>   at 
> org.infernus.idea.checkstyle.checker.CheckerFactory.createChecker(CheckerFactory.java:146)
>   at 
> org.infernus.idea.checkstyle.checker.CheckerFactory.getOrCreateCachedChecker(CheckerFactory.java:74)
>   at 
> org.infernus.idea.checkstyle.checker.CheckerFactory.checker(CheckerFactory.java:56)
>   at 
> org.infernus.idea.checkstyle.checker.CheckerFactory.verify(CheckerFactory.java:43)
>   at 
> org.infernus.idea.checkstyle.ui.LocationDialogue.attemptLoadOfFile(LocationDialogue.java:205)
>   at 
> org.infernus.idea.checkstyle.ui.LocationDialogue.access$800(LocationDialogue.java:35)
>   at 
> org.infernus.idea.checkstyle.ui.LocationDialogue$NextAction.actionPerformed(LocationDialogue.java:256)
>   at 
> javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
>   at 
> javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
>   at 
> javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
>   at 
> javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
>   at 
> javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
>   at 
> java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
>   at java.awt.Component.processMouseEvent(Component.java:6541)
>   at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
>   at java.awt.Component.processEvent(Component.java:6306)
>   at java.awt.Container.processEvent(Container.java:2237)
>   at java.awt.Component.dispatchEventImpl(Component.java:4897)
>   at java.awt.Container.dispatchEventImpl(Container.java:2295)
>   at java.awt.Component.dispatchEvent(Component.java:4719)
>   at 
> java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4889)
>   at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4526)
>   at 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #134

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a ModelCoders class for Coder Utilities

[tgroh] Use ModelCoders in LengthPrefixUnknownCoders

[tgroh] Format UrnUtils

[XuMingmin] [BEAM-3983] [SQL] Tables interface supports BigQuery (#4991)

[kedin] [SQL] Infer Row type from input elements

--
[...truncated 141.78 KB...]
[INFO] Excluding com.google.cloud.bigdataoss:gcsio:jar:1.4.5 from the shaded 
jar.
[INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from 

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90649=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90649
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 13/Apr/18 00:21
Start Date: 13/Apr/18 00:21
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5043: [BEAM-4028] 
Adding NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#issuecomment-380983169
 
 
   Thanks Pablo!  This LGTM.
   
   R: @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90649)
Time Spent: 2h 50m  (was: 2h 40m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90644
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 13/Apr/18 00:17
Start Date: 13/Apr/18 00:17
Worklog Time Spent: 10m 
  Work Description: pabloem commented on a change in pull request #5043: 
[BEAM-4028] Adding NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#discussion_r181257174
 
 

 ##
 File path: sdks/python/apache_beam/runners/common.py
 ##
 @@ -39,6 +39,75 @@
 from apache_beam.utils.windowed_value import WindowedValue
 
 
+class NameContext(object):
+  """Holds the name information for a step."""
+
+  def __init__(self, step_name):
+"""Creates a new step NameContext.
+
+Args:
+  step_name: The name of the step.
+"""
+self.step_name = step_name
+
+  def __eq__(self, other):
+return self.step_name == other.step_name
+
+  def __ne__(self, other):
+return not self == other
+
+  def __repr__(self):
+return 'NameContext(%s)' % self.__dict__()
+
+  def __hash__(self):
+return hash(self.step_name)
+
+  def metrics_name(self):
+"""Returns the step name used for metrics reporting."""
+return self.step_name
+
+  def logging_name(self):
+"""Returns the step name used for logging."""
+return self.step_name
+
+
+class DataflowNameContext(NameContext):
 
 Review comment:
   For now, yes. This is because the Dataflow-specific setup is done in 
`create_operation`. This will be followed by 1) internal change, and then 2)PR 
removing it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90644)
Time Spent: 2.5h  (was: 2h 20m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90645=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90645
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 13/Apr/18 00:17
Start Date: 13/Apr/18 00:17
Worklog Time Spent: 10m 
  Work Description: pabloem commented on a change in pull request #5043: 
[BEAM-4028] Adding NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#discussion_r181257183
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/operations.py
 ##
 @@ -104,34 +104,45 @@ class Operation(object):
   one or more receiver operations that will take that as input.
   """
 
-  def __init__(self, operation_name, spec, counter_factory, state_sampler):
+  def __init__(self, name_context, spec, counter_factory, state_sampler):
 """Initializes a worker operation instance.
 
 Args:
-  operation_name: The system name assigned by the runner for this
-operation.
+  name_context: A NameContext instance or string(deprecated), with the
+name information for this operation.
   spec: A operation_specs.Worker* instance.
   counter_factory: The CounterFactory to use for our counters.
   state_sampler: The StateSampler for the current operation.
 """
-self.operation_name = operation_name
+if isinstance(name_context, common.NameContext):
+  #TODO(pabloem) - Clean this up once it's completely migrated.
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90645)
Time Spent: 2h 40m  (was: 2.5h)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4028) Step / Operation naming should rely on a NameContext class

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4028?focusedWorklogId=90643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90643
 ]

ASF GitHub Bot logged work on BEAM-4028:


Author: ASF GitHub Bot
Created on: 13/Apr/18 00:17
Start Date: 13/Apr/18 00:17
Worklog Time Spent: 10m 
  Work Description: pabloem commented on a change in pull request #5043: 
[BEAM-4028] Adding NameContext to Python SDK.
URL: https://github.com/apache/beam/pull/5043#discussion_r181257252
 
 

 ##
 File path: sdks/python/apache_beam/runners/worker/operations.py
 ##
 @@ -682,14 +701,14 @@ def execute(self):
 # The order of the elements is important because the inputs use
 # list indexes as references.
 
-step_names = (
-self._map_task.step_names or [None] * len(self._map_task.operations))
 for ix, spec in enumerate(self._map_task.operations):
   # This is used for logging and assigning names to counters.
-  operation_name = self._map_task.system_names[ix]
-  step_name = step_names[ix]
+  name_context = common.DataflowNameContext(
 
 Review comment:
   No. The maptask will have the name contexts after internal change + PR, and 
this won't be necessary any longer.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90643)
Time Spent: 2.5h  (was: 2h 20m)

> Step / Operation naming should rely on a NameContext class
> --
>
> Key: BEAM-4028
> URL: https://issues.apache.org/jira/browse/BEAM-4028
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Steps can have different names depending on the runner (stage, step, user, 
> system name...). 
> Depending on the needs of different components (operations, logging, metrics, 
> statesampling) these step names are passed around without a specific order.
> Instead, SDK should rely on `NameContext` objects that carry all the naming 
> information for a single step.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #97

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=90639=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90639
 ]

ASF GitHub Bot logged work on BEAM-4059:


Author: ASF GitHub Bot
Created on: 12/Apr/18 23:58
Start Date: 12/Apr/18 23:58
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #5116: [BEAM-4059] 
Increasing timeout for Dataflow ValidatesRunner.
URL: https://github.com/apache/beam/pull/5116
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
index 07f3b6ca6c3..adb52114bc6 100644
--- 
a/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
+++ 
b/.test-infra/jenkins/job_beam_PostCommit_Java_ValidatesRunner_Dataflow.groovy
@@ -25,8 +25,10 @@ job('beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle') {
   previousNames('beam_PostCommit_Java_ValidatesRunner_Dataflow')
   previousNames('beam_PostCommit_Java_RunnableOnService_Dataflow')
 
-  // Set common parameters. Sets a long (3 hour) timeout due to timeouts in 
[BEAM-3775].
-  common_job_properties.setTopLevelMainJobProperties(delegate, 'master', 180)
+  // Set common parameters. 
+  // Sets a long (4 hour) timeout due to timeouts in BEAM-4059.
+  // TODO(BEAM-4059) - Speed up this test.
+  common_job_properties.setTopLevelMainJobProperties(delegate, 'master', 240)
 
   // Publish all test results to Jenkins
   publishers {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90639)
Time Spent: 0.5h  (was: 20m)

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3444) Fix flake8 detected errors E999 (AST compile error)

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3444?focusedWorklogId=90638=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90638
 ]

ASF GitHub Bot logged work on BEAM-3444:


Author: ASF GitHub Bot
Created on: 12/Apr/18 23:47
Start Date: 12/Apr/18 23:47
Worklog Time Spent: 10m 
  Work Description: shoyer commented on a change in pull request #4376: 
[BEAM-3444] Fix python3 flake8 errors e999
URL: https://github.com/apache/beam/pull/4376#discussion_r181253388
 
 

 ##
 File path: sdks/python/apache_beam/runners/common.py
 ##
 @@ -512,7 +514,7 @@ def _reraise_augmented(self, exn):
   traceback.format_exception_only(type(exn), exn)[-1].strip()
   + step_annotation)
   new_exn._tagged_with_step = True
-raise new_exn, None, original_traceback
+six.raise_from(new_exn, original_traceback)
 
 Review comment:
   This use of `raise_from` instead of `reraise` led to a bug with dropped 
stacktraces: https://issues.apache.org/jira/projects/BEAM/issues/BEAM-3956
   
   Are the other uses of `raise_from` instead of `reraise` in this PR 
appropriate?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90638)
Time Spent: 6h 10m  (was: 6h)

> Fix flake8 detected errors E999 (AST compile error)
> ---
>
> Key: BEAM-3444
> URL: https://issues.apache.org/jira/browse/BEAM-3444
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: holdenk
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Fix flake8 detected errors E999 (AST compile error) so that we can run flake8 
> to catch potential python3 breaking issues.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #58

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Infer Row type from input elements

--
[...truncated 18.65 MB...]
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Extract keys 
as step s15
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/GroupByKey as step s16
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Sample 
keys/Combine.GroupedValues as step s17
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/GBKaSVForData/ParDo(GroupByKeyHashAndSortByKeyAndWindow) as step s18
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/GBKaSVForData/BatchViewOverrides.GroupByKeyAndSortValuesOnly as step s19
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/ParMultiDo(ToIsmRecordForMapLike) as step s20
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/GBKaSVForSize as step s21
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/ParDo(ToIsmMetadataRecordForSize) as step s22
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/GBKaSVForKeys as step s23
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/ParDo(ToIsmMetadataRecordForKey) as step s24
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/Flatten.PCollections as step s25
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Keys sample 
as view/CreateDataflowView as step s26
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Partition 
input as step s27
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Group by 
partition as step s28
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Batch 
mutations together as step s29
Apr 12, 2018 11:14:41 PM 
org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep
INFO: Adding SpannerIO.Write/Write mutations to Cloud Spanner/Write 
mutations to Spanner as step s30
Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: Staging pipeline description to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testwrite-jenkins-0412231426-3b70b920/output/results/staging/
Apr 12, 2018 11:14:41 PM org.apache.beam.runners.dataflow.util.PackageUtil 
tryStagePackage
INFO: Uploading <80355 bytes, hash maM2DFrrTwC0SWeilGLwUw> to 
gs://temp-storage-for-end-to-end-tests/spannerwriteit0testwrite-jenkins-0412231426-3b70b920/output/results/staging/pipeline-maM2DFrrTwC0SWeilGLwUw.pb

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_OUT
Dataflow SDK version: 2.5.0-SNAPSHOT

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_ERROR
Apr 12, 2018 11:14:42 PM org.apache.beam.runners.dataflow.DataflowRunner run
INFO: To access the Dataflow monitoring console, please navigate to 
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-04-12_16_14_41-11638957145981735383?project=apache-beam-testing

org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT > testWrite STANDARD_OUT

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=90632=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90632
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 12/Apr/18 23:01
Start Date: 12/Apr/18 23:01
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-380970018
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90632)
Time Spent: 90h 10m  (was: 90h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 90h 10m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3776) StateMerging.mergeWatermarks sets a late watermark hold for late merging windows that depend only on the window

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3776?focusedWorklogId=90629=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90629
 ]

ASF GitHub Bot logged work on BEAM-3776:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:59
Start Date: 12/Apr/18 22:59
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #4793: [BEAM-3776] Fix issue 
with merging late windows where a watermark hold could be added behind the 
input watermark.
URL: https://github.com/apache/beam/pull/4793#issuecomment-380969648
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90629)
Time Spent: 5h  (was: 4h 50m)

> StateMerging.mergeWatermarks sets a late watermark hold for late merging 
> windows that depend only on the window
> ---
>
> Key: BEAM-3776
> URL: https://issues.apache.org/jira/browse/BEAM-3776
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Affects Versions: 2.1.0, 2.2.0, 2.3.0
>Reporter: Sam Whittle
>Assignee: Sam Whittle
>Priority: Critical
>  Time Spent: 5h
>  Remaining Estimate: 0h
>
> WatermarkHold.addElementHold and WatermarkHold.addGarbageCollectionHold take 
> to not add holds that would be before the input watermark.
> However WatermarkHold.onMerge calls StateMerging.mergeWatermarks which if the 
> window depends only on window, sets a hold for the end of the window 
> regardless of the input watermark.
> Thus if you have a WindowingStrategy such as:
> WindowingStrategy.of(Sessions.withGapDuration(gapDuration))
>  .withMode(AccumulationMode.DISCARDING_FIRED_PANES)
>  .withTrigger(
>  Repeatedly.forever(
>  AfterWatermark.pastEndOfWindow()
>  .withLateFirings(AfterPane.elementCountAtLeast(10
>  .withAllowedLateness(allowedLateness))
> and you merge windows that are late, you might end up holding the watermark 
> until the allowedLateness has passed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4056) Identify Side Inputs by PTransform ID and local name

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4056?focusedWorklogId=90626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90626
 ]

ASF GitHub Bot logged work on BEAM-4056:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:58
Start Date: 12/Apr/18 22:58
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5118: 
[BEAM-4056] Identify side inputs by transform id and local name
URL: https://github.com/apache/beam/pull/5118#discussion_r181237746
 
 

 ##
 File path: model/pipeline/src/main/proto/beam_runner_api.proto
 ##
 @@ -217,12 +217,12 @@ message ExecutableStagePayload {
   // PTransform the ExecutableStagePayload is the payload of.
   string input = 2;
 
-  // Side Input PCollection ids. Each must be present as a value in the inputs 
of
-  // any PTransform the ExecutableStagePayload is the payload of.
-  repeated string side_inputs = 3;
+  // The side inputs required for this executable stage. Each must be prsent 
as a side input of
 
 Review comment:
   spelling


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90626)
Time Spent: 0.5h  (was: 20m)

> Identify Side Inputs by PTransform ID and local name
> 
>
> Key: BEAM-4056
> URL: https://issues.apache.org/jira/browse/BEAM-4056
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> This is necessary in order to correctly identify side inputs during all 
> phases of portable pipeline execution (fusion, translation, and SDK 
> execution).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (670c75e -> b696435)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 670c75e  Merge pull request #4649: [SQL] Infer row types
 add 5022762  Add a ModelCoders class for Coder Utilities
 add 767f8c9  Use ModelCoders in LengthPrefixUnknownCoders
 add 8c6592a  Format UrnUtils
 new b696435  Merge pull request #5090: Add a ModelCoders class for Coder 
Utilities

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../core/construction/ModelCoderRegistrar.java | 35 +++-
 .../runners/core/construction/ModelCoders.java | 65 ++
 .../beam/runners/core/construction/UrnUtils.java   |  8 +--
 .../graph/LengthPrefixUnknownCoders.java   | 18 ++
 4 files changed, 94 insertions(+), 32 deletions(-)
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ModelCoders.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[jira] [Work logged] (BEAM-4056) Identify Side Inputs by PTransform ID and local name

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4056?focusedWorklogId=90627=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90627
 ]

ASF GitHub Bot logged work on BEAM-4056:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:58
Start Date: 12/Apr/18 22:58
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5118: 
[BEAM-4056] Identify side inputs by transform id and local name
URL: https://github.com/apache/beam/pull/5118#discussion_r181238037
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SideInputReference.java
 ##
 @@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.core.construction.graph;
+
+import com.google.auto.value.AutoValue;
+import org.apache.beam.model.pipeline.v1.RunnerApi;
+import 
org.apache.beam.model.pipeline.v1.RunnerApi.ExecutableStagePayload.SideInputId;
+import 
org.apache.beam.runners.core.construction.graph.PipelineNode.PCollectionNode;
+
+/**
+ * A reference to a side input. This includes the PTransform that references 
the side input as well
+ * as the PCollection referenced. Both are necessary in order to fully resolve 
a view.
+ */
+@AutoValue
+public abstract class SideInputReference {
+
+  /** Create a side input reference. */
+  public static SideInputReference of(String transformId, String localName,
 
 Review comment:
   Maybe a PTransformNode? Would that be available everywhere we're 
constructing this?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90627)
Time Spent: 40m  (was: 0.5h)

> Identify Side Inputs by PTransform ID and local name
> 
>
> Key: BEAM-4056
> URL: https://issues.apache.org/jira/browse/BEAM-4056
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This is necessary in order to correctly identify side inputs during all 
> phases of portable pipeline execution (fusion, translation, and SDK 
> execution).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3952) GreedyStageFuserTest broken

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3952?focusedWorklogId=90624=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90624
 ]

ASF GitHub Bot logged work on BEAM-3952:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:58
Start Date: 12/Apr/18 22:58
Worklog Time Spent: 10m 
  Work Description: tgroh commented on issue #4995: [BEAM-3952][BEAM-3988] 
Fix GreedyPipelineFuser test
URL: https://github.com/apache/beam/pull/4995#issuecomment-380969320
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90624)
Time Spent: 0.5h  (was: 20m)

> GreedyStageFuserTest broken
> ---
>
> Key: BEAM-3952
> URL: https://issues.apache.org/jira/browse/BEAM-3952
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Thomas Groh
>Priority: Minor
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The materializesWithDifferentEnvConsumer test is currently failing due to a 
> bad assertion. The fused subgraph contains the parDo.out PCollection but the 
> test expects an empty output.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4056) Identify Side Inputs by PTransform ID and local name

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4056?focusedWorklogId=90628=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90628
 ]

ASF GitHub Bot logged work on BEAM-4056:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:58
Start Date: 12/Apr/18 22:58
Worklog Time Spent: 10m 
  Work Description: tgroh commented on a change in pull request #5118: 
[BEAM-4056] Identify side inputs by transform id and local name
URL: https://github.com/apache/beam/pull/5118#discussion_r181237867
 
 

 ##
 File path: 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
 ##
 @@ -122,11 +123,16 @@ default PTransform toPTransform() {
 pt.putInputs("input", getInputPCollection().getId());
 payload.setInput(input.getId());
 
-int sideInputIndex = 0;
-for (PCollectionNode sideInputNode : getSideInputPCollections()) {
-  pt.putInputs(String.format("side_input_%s", sideInputIndex), 
sideInputNode.getId());
-  payload.addSideInputs(sideInputNode.getId());
-  sideInputIndex++;
+for (SideInputReference sideInput : getSideInputs()) {
+  // Side inputs of the ExecutableStage itself can be uniquely identified 
by inner PTransform
+  // name and local name.
+  String outerLocalName = String.format("%s:%s",
+  sideInput.transformId(), sideInput.localName());
+  pt.putInputs(outerLocalName, sideInput.getCollection().getId());
+  payload.addSideInputs(SideInputId.newBuilder()
 
 Review comment:
   Your formatting looks funky here


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90628)
Time Spent: 50m  (was: 40m)

> Identify Side Inputs by PTransform ID and local name
> 
>
> Key: BEAM-4056
> URL: https://issues.apache.org/jira/browse/BEAM-4056
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> This is necessary in order to correctly identify side inputs during all 
> phases of portable pipeline execution (fusion, translation, and SDK 
> execution).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #5090: Add a ModelCoders class for Coder Utilities

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b6964352c2235dcc624b2ed9e92321a6f42d01c0
Merge: 670c75e 8c6592a
Author: Thomas Groh 
AuthorDate: Thu Apr 12 15:59:01 2018 -0700

Merge pull request #5090: Add a ModelCoders class for Coder Utilities

 .../core/construction/ModelCoderRegistrar.java | 35 +++-
 .../runners/core/construction/ModelCoders.java | 65 ++
 .../beam/runners/core/construction/UrnUtils.java   |  8 +--
 .../graph/LengthPrefixUnknownCoders.java   | 18 ++
 4 files changed, 94 insertions(+), 32 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam-site] 02/04: Typo fix

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 2b44c8c104ab7c13bdb6c8e7b04a552dcf2f8d08
Author: Melissa Pashniak 
AuthorDate: Thu Apr 12 13:59:32 2018 -0700

Typo fix
---
 src/contribute/eclipse.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/contribute/eclipse.md b/src/contribute/eclipse.md
index 592851e..2cf0484 100644
--- a/src/contribute/eclipse.md
+++ b/src/contribute/eclipse.md
@@ -48,11 +48,11 @@ Package.
 
 Verify that your workspace is correctly configured by invoking
 'beam-runners-direct-java:build' from the "Gradle Tasks" pane. The build should
-succeed with now errors.
+succeed with no errors.
 
 ## Building
 
-Once your Eclipse workspace is properly setup, you will have a "Gradle
+After your Eclipse workspace is properly setup, you will have a "Gradle
 Tasks" window with a set of operations. If you don't see the pane, open it
 from "Window" > "Show View" > Other.. > "Gradle" > "Gradle Tasks".
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (6cfd3ba -> c2e16d4)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 6cfd3ba  Prepare repository for deployment.
 new e12f932  Update Eclipse documentation for Gradle build.
 new 2b44c8c  Typo fix
 new 07dd232  Remove reference to now-deleted documentation
 new c2e16d4  This closes #419

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/eclipse/index.html | 75 +--
 src/contribute/eclipse.md | 66 +-
 2 files changed, 84 insertions(+), 57 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 04/04: This closes #419

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c2e16d4b2996450173f31f8c8fd678a001a1ddca
Merge: 6cfd3ba 07dd232
Author: Mergebot 
AuthorDate: Thu Apr 12 15:53:03 2018 -0700

This closes #419

 content/contribute/eclipse/index.html | 75 +--
 src/contribute/eclipse.md | 66 +-
 2 files changed, 84 insertions(+), 57 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/04: Update Eclipse documentation for Gradle build.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit e12f93209d6055f6a63d3675217a8c8db1d59712
Author: Scott Wegner 
AuthorDate: Wed Apr 11 16:23:36 2018 -0700

Update Eclipse documentation for Gradle build.
---
 src/contribute/eclipse.md | 57 ++-
 1 file changed, 31 insertions(+), 26 deletions(-)

diff --git a/src/contribute/eclipse.md b/src/contribute/eclipse.md
index e0cf477..592851e 100644
--- a/src/contribute/eclipse.md
+++ b/src/contribute/eclipse.md
@@ -17,42 +17,47 @@ section_menu: section-menu/contribute.html
 
 ## Eclipse version
 
-Use a recent Eclipse version that includes m2e. Currently we recommend Eclipse
-Neon. Start Eclipse with a fresh workspace in a separate directory from your
+Use a recent Eclipse version that includes
+[Buildship](https://projects.eclipse.org/projects/tools.buildship) for Gradle 
integration.
+Currently we recommend Eclipse Oxygen. Start Eclipse with a fresh workspace in 
a separate directory from your
 checkout.
 
 ## Initial setup
 
-1. Install m2e-apt: Beam uses apt annotation processing to provide auto
-   generated code. One example is the usage of [Google
-   AutoValue](https://github.com/google/auto/tree/master/value). By default m2e
-   does not support this and you will see compile errors.
+1. Before setting up Eclipse integration, verify that you can successfully 
build
+   from the commandline using instructions from the [Contribution guide]({{ 
site.baseurl }}/contribute/contribution-guide#building).
 
-   Help
-   -> Eclipse Marketplace
-   -> Search for "m2 apt"
-   -> Install m2e-apt 1.2 or higher
+1. If running on Mac or Linux, launch Eclipse from a terminal. This is 
necessary
+   to ensure that Eclipse has proper environment setup from user profile
+   scripts, i.e. .bashrc.
 
-2. Activate the apt processing
+1. Install [Buildship Gradle
+   
Integration](https://marketplace.eclipse.org/content/buildship-gradle-integration).
+   This will allow importing and interacting with the Gradle build.
 
-   Window
-   -> Preferences
-   -> Maven
-   -> Annotation processing
-   -> Switch to Experimental: Delegate annotation processing ...
-   -> Ok
+1. Open the project import wizard in Eclipse via "File" > "Import".
 
-3. Import the beam projects
+1. From the "Import" screen, select "Gradle" > "Existing Gradle Project", and 
click
+   Next.
 
-   File
-   -> Import...
-   -> Existing Maven Projects
-   -> Browse to the directory you cloned into and select "beam"
-   -> make sure all beam projects are selected
-   -> Finalize
+1. From the "Import Gradle Project" screen, fill in the Project root directory
+   with your local git path, and click Finish.
 
-You now should have all the beam projects imported into Eclipse and should see
-no compile errors.
+Eclipse will scan the project tree and import each as a separate
+Package.
+
+Verify that your workspace is correctly configured by invoking
+'beam-runners-direct-java:build' from the "Gradle Tasks" pane. The build should
+succeed with now errors.
+
+## Building
+
+Once your Eclipse workspace is properly setup, you will have a "Gradle
+Tasks" window with a set of operations. If you don't see the pane, open it
+from "Window" > "Show View" > Other.. > "Gradle" > "Gradle Tasks".
+
+From the "Gradle Tasks" window, you can build any task registered with Gradle.
+For example, if you are working on Kinesis IO, select 
'beam-sdks-java-io-kinesis:build'.
 
 ## Checkstyle
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 03/04: Remove reference to now-deleted documentation

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 07dd232fa9a1ecedcff32448fd90aed215fdb40f
Author: Scott Wegner 
AuthorDate: Thu Apr 12 14:24:27 2018 -0700

Remove reference to now-deleted documentation
---
 content/contribute/eclipse/index.html | 75 +--
 src/contribute/eclipse.md | 17 +---
 2 files changed, 57 insertions(+), 35 deletions(-)

diff --git a/content/contribute/eclipse/index.html 
b/content/contribute/eclipse/index.html
index 089d5b7..b75aaa9 100644
--- a/content/contribute/eclipse/index.html
+++ b/content/contribute/eclipse/index.html
@@ -147,6 +147,7 @@
 
   Eclipse version
   Initial setup
+  Building
   Checkstyle
   Code Style
 
@@ -171,48 +172,62 @@
 
 Eclipse version
 
-Use a recent Eclipse version that includes m2e. Currently we recommend 
Eclipse
-Neon. Start Eclipse with a fresh workspace in a separate directory from your
-checkout.
+Use a recent Eclipse version that includes
+https://projects.eclipse.org/projects/tools.buildship;>Buildship 
for Gradle
+integration. Currently we recommend Eclipse Oxygen. Start Eclipse with a fresh
+workspace in a separate directory from your checkout.
 
 Initial setup
 
+Before setting up Eclipse integration, verify that you can successfully 
build
+from the commandline by building a sample package:
+
+./gradlew 
:beam-examples-java:build
+
+
+
+If you receive any errors, first verify your environment setup based on the
+Contribution 
guide.
+
 
   
-Install m2e-apt: Beam uses apt annotation processing to provide auto
-generated code. One example is the usage of https://github.com/google/auto/tree/master/value;>Google
-AutoValue. By default m2e
-does not support this and you will see compile errors.
-
-Help
- - Eclipse Marketplace
- - Search for “m2 apt”
- - Install m2e-apt 1.2 or higher
+If running on Mac or Linux, launch Eclipse from a terminal. This is 
necessary
+to ensure that Eclipse has proper environment setup from user profile
+scripts, i.e. .bashrc.
   
   
-Activate the apt processing
-
-Window
- - Preferences
- - Maven
- - Annotation processing
- - Switch to Experimental: Delegate annotation processing …
- - Ok
+Install https://marketplace.eclipse.org/content/buildship-gradle-integration;>Buildship
 Gradle
+Integration.
+This will allow importing and interacting with the Gradle build.
   
   
-Import the beam projects
-
-File
- - Import…
- - Existing Maven Projects
- - Browse to the directory you cloned into and select “beam”
- - make sure all beam projects are selected
- - Finalize
+Open the project import wizard in Eclipse via “File”  “Import”.
+  
+  
+From the “Import” screen, select “Gradle”  “Existing Gradle 
Project”, and click
+Next.
+  
+  
+From the “Import Gradle Project” screen, fill in the Project root 
directory
+with your local git path, and click Finish.
   
 
 
-You now should have all the beam projects imported into Eclipse and should 
see
-no compile errors.
+Eclipse will scan the project tree and import each as a separate
+Package.
+
+Verify that your workspace is correctly configured by invoking
+‘beam-runners-direct-java:build’ from the “Gradle Tasks” pane. The build should
+succeed with no errors.
+
+Building
+
+After your Eclipse workspace is properly setup, you will have a “Gradle
+Tasks” window with a set of operations. If you don’t see the pane, open it
+from “Window”  “Show View”  Other..  “Gradle”  “Gradle 
Tasks”.
+
+From the “Gradle Tasks” window, you can build any task registered with 
Gradle.
+For example, if you are working on Kinesis IO, select 
‘beam-sdks-java-io-kinesis:build’.
 
 Checkstyle
 
diff --git a/src/contribute/eclipse.md b/src/contribute/eclipse.md
index 2cf0484..346fbaa 100644
--- a/src/contribute/eclipse.md
+++ b/src/contribute/eclipse.md
@@ -18,14 +18,21 @@ section_menu: section-menu/contribute.html
 ## Eclipse version
 
 Use a recent Eclipse version that includes
-[Buildship](https://projects.eclipse.org/projects/tools.buildship) for Gradle 
integration.
-Currently we recommend Eclipse Oxygen. Start Eclipse with a fresh workspace in 
a separate directory from your
-checkout.
+[Buildship](https://projects.eclipse.org/projects/tools.buildship) for Gradle
+integration. Currently we recommend Eclipse Oxygen. Start Eclipse with a fresh
+workspace in a separate directory from your checkout.
 
 ## Initial setup
 
-1. Before setting up Eclipse integration, verify that you can successfully 
build
-   from the commandline using instructions from the [Contribution guide]({{ 
site.baseurl }}/contribute/contribution-guide#building).
+Before setting up Eclipse integration, verify that you can successfully build
+from the commandline by building a sample package:
+
+```
+./gradlew :beam-examples-java:build
+```
+
+If you receive any errors, first 

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90619
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:46
Start Date: 12/Apr/18 22:46
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #4947: [BEAM-3983] [SQL] 
Add utils for converting to BigQuery types
URL: https://github.com/apache/beam/pull/4947#issuecomment-380967263
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90619)
Time Spent: 4h 20m  (was: 4h 10m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2898) Flink supports chaining/fusion of single-SDK stages

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2898?focusedWorklogId=90617=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90617
 ]

ASF GitHub Bot logged work on BEAM-2898:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:42
Start Date: 12/Apr/18 22:42
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #4783: [BEAM-2898] Support 
Impulse transforms in Flink batch runner
URL: https://github.com/apache/beam/pull/4783#issuecomment-380966483
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90617)
Time Spent: 4.5h  (was: 4h 20m)

> Flink supports chaining/fusion of single-SDK stages
> ---
>
> Key: BEAM-2898
> URL: https://issues.apache.org/jira/browse/BEAM-2898
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-flink
>Reporter: Henning Rohde
>Priority: Major
>  Labels: portability
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> The Fn API supports fused stages, which avoids unnecessarily round-tripping 
> the data over the Fn API between stages. The Flink runner should use that 
> capability for better performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_ValidatesRunner_Dataflow #1339

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3249) Use Gradle to build/release project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3249?focusedWorklogId=90612=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90612
 ]

ASF GitHub Bot logged work on BEAM-3249:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:21
Start Date: 12/Apr/18 22:21
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5117: [BEAM-3249] Clean-up 
and use shaded test jars, removing evaluationDependsOn
URL: https://github.com/apache/beam/pull/5117#issuecomment-380962180
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90612)
Time Spent: 13h  (was: 12h 50m)

> Use Gradle to build/release project
> ---
>
> Key: BEAM-3249
> URL: https://issues.apache.org/jira/browse/BEAM-3249
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 13h
>  Remaining Estimate: 0h
>
> I have collected data by running several builds against master using Gradle 
> and Maven without using Gradle's support for incremental builds.
> Gradle (mins)
> min: 25.04
> max: 160.14
> median: 45.78
> average: 52.19
> stdev: 30.80
> Maven (mins)
> min: 56.86
> max: 216.55
> median: 87.93
> average: 109.10
> stdev: 48.01
> I excluded a few timeouts (240 mins) that happened during the Maven build 
> from its numbers but we can see conclusively that Gradle is about twice as 
> fast for the build when compared to Maven when run using Jenkins.
> Original dev@ thread: 
> https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E
> The data is available here 
> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #96

[Apache Jenkins Server]

See 


Changes:

[kedin] [SQL] Infer Row type from input elements

--
[...truncated 84.41 MB...]
Apr 12, 2018 10:19:06 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (33a16686b930106ffaa2972dfe87566f) switched from RUNNING 
to FINISHED.
Apr 12, 2018 10:19:06 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (33a16686b930106ffaa2972dfe87566f).
Apr 12, 2018 10:19:06 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (1/1) (33a16686b930106ffaa2972dfe87566f) [FINISHED]
Apr 12, 2018 10:19:06 PM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (33a16686b930106ffaa2972dfe87566f)
Apr 12, 2018 10:19:06 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: PAssert$164/GroupGlobally/GroupDummyAndContents -> 
PAssert$164/GroupGlobally/Values/Values/Map/ParMultiDo(Anonymous) -> 
PAssert$164/GroupGlobally/ParDo(Concat)/ParMultiDo(Concat) -> 
PAssert$164/GetPane/Map/ParMultiDo(Anonymous) -> 
PAssert$164/RunChecks/ParMultiDo(GroupedValuesChecker) -> 
PAssert$164/VerifyAssertions/ParDo(DefaultConclude)/ParMultiDo(DefaultConclude) 
(1/1) (ccd7ae4fb7fdc1268d49b007514f07a6) switched from RUNNING to FINISHED.
Apr 12, 2018 10:19:06 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
PAssert$164/GroupGlobally/GroupDummyAndContents -> 
PAssert$164/GroupGlobally/Values/Values/Map/ParMultiDo(Anonymous) -> 
PAssert$164/GroupGlobally/ParDo(Concat)/ParMultiDo(Concat) -> 
PAssert$164/GetPane/Map/ParMultiDo(Anonymous) -> 
PAssert$164/RunChecks/ParMultiDo(GroupedValuesChecker) -> 
PAssert$164/VerifyAssertions/ParDo(DefaultConclude)/ParMultiDo(DefaultConclude) 
(1/1) (ccd7ae4fb7fdc1268d49b007514f07a6).
Apr 12, 2018 10:19:06 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
PAssert$164/GroupGlobally/GroupDummyAndContents -> 
PAssert$164/GroupGlobally/Values/Values/Map/ParMultiDo(Anonymous) -> 
PAssert$164/GroupGlobally/ParDo(Concat)/ParMultiDo(Concat) -> 
PAssert$164/GetPane/Map/ParMultiDo(Anonymous) -> 
PAssert$164/RunChecks/ParMultiDo(GroupedValuesChecker) -> 
PAssert$164/VerifyAssertions/ParDo(DefaultConclude)/ParMultiDo(DefaultConclude) 
(1/1) (ccd7ae4fb7fdc1268d49b007514f07a6) [FINISHED]
Apr 12, 2018 10:19:06 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
Combine.perKey(TestCombineFnWithContext)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> PAssert$165/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$165/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$165/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$165/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (cef2acfb71442388a9f8aacd4fbfd1e4) switched from 
RUNNING to FINISHED.
Apr 12, 2018 10:19:06 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
Combine.perKey(TestCombineFnWithContext)/Combine.GroupedValues/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> PAssert$165/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$165/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$165/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$165/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> 

[jira] [Work logged] (BEAM-3513) Use portable CombinePayload in Java DataflowRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3513?focusedWorklogId=90608=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90608
 ]

ASF GitHub Bot logged work on BEAM-3513:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:17
Start Date: 12/Apr/18 22:17
Worklog Time Spent: 10m 
  Work Description: youngoli opened a new pull request #5119: [BEAM-3513] 
Removing PrimitiveCombineGroupedValues override w/ FnAPI.
URL: https://github.com/apache/beam/pull/5119
 
 
   When using the FnAPI, make it so that Combine.GroupedValues are not
   replaced by the Dataflow specific PTransforms. The reason for this is
   to allow the Combines to be translated to a Dataflow pipeline as the
   constituent parts of the composite transform (in other words, send
   Dataflow a GBK and a ParDo, instead of a CombineGroupedValues black
   box).
   
   This does not fully accomplish the final goal of portable combines since
   this method doesn't do anything to help Combiner lifting, but it works
   for the case of the basic Combine that has been implemented. It
   successfully treats the Combine as a GBK and ParDo when not lifted.
   
   Additional changes will need to be made to detect when we want to lift
   Combines and in those situations handle that properly, which may mean
   putting the Combines in the black box.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [x] Write a pull request description that is detailed enough to 
understand:
  - [x] What the pull request does
  - [x] Why it does it
  - [x] How it does it
  - [x] Why this approach
- [x] Each commit in the pull request should have a meaningful subject line 
and body.
- [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90608)
Time Spent: 10m
Remaining Estimate: 0h

> Use portable CombinePayload in Java DataflowRunner
> --
>
> Key: BEAM-3513
> URL: https://issues.apache.org/jira/browse/BEAM-3513
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-dataflow
>Reporter: Kenneth Knowles
>Assignee: Daniel Oliveira
>Priority: Major
>  Labels: portability
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The Java-specific blobs transmitted to Dataflow need more context, in the 
> form of portability framework protos.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4056) Identify Side Inputs by PTransform ID and local name

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4056?focusedWorklogId=90607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90607
 ]

ASF GitHub Bot logged work on BEAM-4056:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:12
Start Date: 12/Apr/18 22:12
Worklog Time Spent: 10m 
  Work Description: bsidhom commented on issue #5118: [BEAM-4056] Identify 
side inputs by transform id and local name
URL: https://github.com/apache/beam/pull/5118#issuecomment-380960266
 
 
   R: @tgroh 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90607)
Time Spent: 20m  (was: 10m)

> Identify Side Inputs by PTransform ID and local name
> 
>
> Key: BEAM-4056
> URL: https://issues.apache.org/jira/browse/BEAM-4056
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> This is necessary in order to correctly identify side inputs during all 
> phases of portable pipeline execution (fusion, translation, and SDK 
> execution).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4056) Identify Side Inputs by PTransform ID and local name

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4056?focusedWorklogId=90606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90606
 ]

ASF GitHub Bot logged work on BEAM-4056:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:12
Start Date: 12/Apr/18 22:12
Worklog Time Spent: 10m 
  Work Description: bsidhom opened a new pull request #5118: [BEAM-4056] 
Identify side inputs by transform id and local name
URL: https://github.com/apache/beam/pull/5118
 
 
   This is necessary to identify side inputs during portable pipeline 
translation and execution.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
- [ ] Write a pull request description that is detailed enough to 
understand:
  - [ ] What the pull request does
  - [ ] Why it does it
  - [ ] How it does it
  - [ ] Why this approach
- [ ] Each commit in the pull request should have a meaningful subject line 
and body.
- [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90606)
Time Spent: 10m
Remaining Estimate: 0h

> Identify Side Inputs by PTransform ID and local name
> 
>
> Key: BEAM-4056
> URL: https://issues.apache.org/jira/browse/BEAM-4056
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core
>Reporter: Ben Sidhom
>Assignee: Ben Sidhom
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This is necessary in order to correctly identify side inputs during all 
> phases of portable pipeline execution (fusion, translation, and SDK 
> execution).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4062) Performance regression in FileBasedSink

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-4062:

Fix Version/s: 2.5.0

> Performance regression in FileBasedSink
> ---
>
> Key: BEAM-4062
> URL: https://issues.apache.org/jira/browse/BEAM-4062
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Blocker
> Fix For: 2.5.0
>
>
> [https://github.com/apache/beam/pull/4648] has added:
>  * 3 or more stat() calls per output file (in pre_finalize and 
> finalize_writes)
>  * serial unbatched delete()s (in pre_finalize)
> Solution will be to list files in a batch operation (match()), and to 
> delete() in batch mode, or use multiple threads if that's not possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4062) Performance regression in FileBasedSink

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-4062:

Priority: Blocker  (was: Major)

> Performance regression in FileBasedSink
> ---
>
> Key: BEAM-4062
> URL: https://issues.apache.org/jira/browse/BEAM-4062
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Blocker
> Fix For: 2.5.0
>
>
> [https://github.com/apache/beam/pull/4648] has added:
>  * 3 or more stat() calls per output file (in pre_finalize and 
> finalize_writes)
>  * serial unbatched delete()s (in pre_finalize)
> Solution will be to list files in a batch operation (match()), and to 
> delete() in batch mode, or use multiple threads if that's not possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4062) Performance regression in FileBasedSink

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-4062:

Affects Version/s: (was: 2.5.0)

> Performance regression in FileBasedSink
> ---
>
> Key: BEAM-4062
> URL: https://issues.apache.org/jira/browse/BEAM-4062
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Blocker
> Fix For: 2.5.0
>
>
> [https://github.com/apache/beam/pull/4648] has added:
>  * 3 or more stat() calls per output file (in pre_finalize and 
> finalize_writes)
>  * serial unbatched delete()s (in pre_finalize)
> Solution will be to list files in a batch operation (match()), and to 
> delete() in batch mode, or use multiple threads if that's not possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4062) Performance regression in FileBasedSink

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri updated BEAM-4062:

Affects Version/s: 2.5.0

> Performance regression in FileBasedSink
> ---
>
> Key: BEAM-4062
> URL: https://issues.apache.org/jira/browse/BEAM-4062
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Blocker
> Fix For: 2.5.0
>
>
> [https://github.com/apache/beam/pull/4648] has added:
>  * 3 or more stat() calls per output file (in pre_finalize and 
> finalize_writes)
>  * serial unbatched delete()s (in pre_finalize)
> Solution will be to list files in a batch operation (match()), and to 
> delete() in batch mode, or use multiple threads if that's not possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-4062) Performance regression in FileBasedSink

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri reassigned BEAM-4062:
---

Assignee: Udi Meiri  (was: Ahmet Altay)

> Performance regression in FileBasedSink
> ---
>
> Key: BEAM-4062
> URL: https://issues.apache.org/jira/browse/BEAM-4062
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>
> [https://github.com/apache/beam/pull/4648] has added:
>  * 3 or more stat() calls per output file (in pre_finalize and 
> finalize_writes)
>  * serial unbatched delete()s (in pre_finalize)
> Solution will be to list files in a batch operation (match()), and to 
> delete() in batch mode, or use multiple threads if that's not possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4062) Performance regression in FileBasedSink

[Udi Meiri (JIRA)]

Udi Meiri created BEAM-4062:
---

 Summary: Performance regression in FileBasedSink
 Key: BEAM-4062
 URL: https://issues.apache.org/jira/browse/BEAM-4062
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Udi Meiri
Assignee: Ahmet Altay


[https://github.com/apache/beam/pull/4648] has added:
 * 3 or more stat() calls per output file (in pre_finalize and finalize_writes)
 * serial unbatched delete()s (in pre_finalize)

Solution will be to list files in a batch operation (match()), and to delete() 
in batch mode, or use multiple threads if that's not possible.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90603
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 12/Apr/18 22:08
Start Date: 12/Apr/18 22:08
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #4947: [BEAM-3983] [SQL] 
Add utils for converting to BigQuery types
URL: https://github.com/apache/beam/pull/4947#issuecomment-380959185
 
 
   I've updated this to use Beam Schemas and moved the common stuff out of SQL.
   
   @reuvenlax You should make sure I'm not doing anything crazy in 
4cfa16b050c80d73199ff790d0327d854a2af608


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90603)
Time Spent: 4h 10m  (was: 4h)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (e981b43 -> 670c75e)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e981b43  [BEAM-3983] [SQL] Tables interface supports BigQuery (#4991)
 add 875da55  [SQL] Infer Row type from input elements
 new 670c75e  Merge pull request #4649: [SQL] Infer row types

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/values/reflect/GetterFactory.java |   3 +-
 .../beam/sdk/values/reflect/InferredRowCoder.java  | 117 ++
 .../values/reflect/ReflectionGetterFactory.java|   4 +-
 .../apache/beam/sdk/values/reflect/RowFactory.java |  35 ++-
 .../beam/sdk/values/reflect/RowTypeFactory.java|   3 +-
 .../sdk/values/reflect/InferredRowCoderTest.java   |  98 
 .../beam/sdk/values/reflect/RowFactoryTest.java|   7 +-
 .../beam/sdk/extensions/sql/QueryTransform.java|   5 +-
 .../beam/sdk/extensions/sql/SchemaHelper.java  |  69 ++
 .../extensions/sql/InferredRowCoderSqlTest.java| 252 +
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  18 +-
 11 files changed, 590 insertions(+), 21 deletions(-)
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/values/reflect/InferredRowCoder.java
 create mode 100644 
sdks/java/core/src/test/java/org/apache/beam/sdk/values/reflect/InferredRowCoderTest.java
 create mode 100644 
sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/SchemaHelper.java
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/InferredRowCoderSqlTest.java

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[beam] 01/01: Merge pull request #4649: [SQL] Infer row types

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 670c75e94795ad9da6a0690647e996dc97b60718
Merge: e981b43 875da55
Author: Kenn Knowles 
AuthorDate: Thu Apr 12 14:59:17 2018 -0700

Merge pull request #4649: [SQL] Infer row types

 .../beam/sdk/values/reflect/GetterFactory.java |   3 +-
 .../beam/sdk/values/reflect/InferredRowCoder.java  | 117 ++
 .../values/reflect/ReflectionGetterFactory.java|   4 +-
 .../apache/beam/sdk/values/reflect/RowFactory.java |  35 ++-
 .../beam/sdk/values/reflect/RowTypeFactory.java|   3 +-
 .../sdk/values/reflect/InferredRowCoderTest.java   |  98 
 .../beam/sdk/values/reflect/RowFactoryTest.java|   7 +-
 .../beam/sdk/extensions/sql/QueryTransform.java|   5 +-
 .../beam/sdk/extensions/sql/SchemaHelper.java  |  69 ++
 .../extensions/sql/InferredRowCoderSqlTest.java| 252 +
 .../apache/beam/sdk/extensions/sql/TestUtils.java  |  18 +-
 11 files changed, 590 insertions(+), 21 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

[jira] [Work logged] (BEAM-4061) Chaining SpannerIO#write() transforms

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4061?focusedWorklogId=90600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90600
 ]

ASF GitHub Bot logged work on BEAM-4061:


Author: ASF GitHub Bot
Created on: 12/Apr/18 21:56
Start Date: 12/Apr/18 21:56
Worklog Time Spent: 10m 
  Work Description: mairbek commented on issue #4264: [BEAM-4061] 
Introduced SpannerWriteResult
URL: https://github.com/apache/beam/pull/4264#issuecomment-380956438
 
 
   Done, @iemejia please take look when you have a chance


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90600)
Time Spent: 10m
Remaining Estimate: 0h

> Chaining SpannerIO#write() transforms
> -
>
> Key: BEAM-4061
> URL: https://issues.apache.org/jira/browse/BEAM-4061
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Mairbek Khadikov
>Assignee: Mairbek Khadikov
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> It should be possible to chain several Cloud Spanner writes. In practice, we 
> can leverage Wait.on transform by returning a result object from 
> SpannerIO#write.
> One particular example, when this feature is useful is full database import. 
> When data in parent tables should be injected before interleaved tables. See 
> more about table hierarchies in Spanner here 
> https://cloud.google.com/spanner/docs/schema-and-data-model#creating_a_hierarchy_of_interleaved_tables



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4061) Chaining SpannerIO#write() transforms

[Mairbek Khadikov (JIRA)]

Mairbek Khadikov created BEAM-4061:
--

 Summary: Chaining SpannerIO#write() transforms
 Key: BEAM-4061
 URL: https://issues.apache.org/jira/browse/BEAM-4061
 Project: Beam
  Issue Type: Bug
  Components: io-java-gcp
Reporter: Mairbek Khadikov
Assignee: Mairbek Khadikov
 Fix For: 2.5.0


It should be possible to chain several Cloud Spanner writes. In practice, we 
can leverage Wait.on transform by returning a result object from 
SpannerIO#write.

One particular example, when this feature is useful is full database import. 
When data in parent tables should be injected before interleaved tables. See 
more about table hierarchies in Spanner here 
https://cloud.google.com/spanner/docs/schema-and-data-model#creating_a_hierarchy_of_interleaved_tables



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1338

[Apache Jenkins Server]

See 


--
[...truncated 131.49 KB...]
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.0.2.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_list_twice (apache_beam.transforms.sideinputs_test.SideInputsTest) ... 
ok
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
test_as_list_and_as_dict_side_inputs 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-39.0.1.zip
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.0.2.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-39.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.0.2.tar.gz
test_as_singleton_with_different_defaults 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded 
/tmp/dataflow-requirements-cache/setuptools-39.0.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
  File was already downloaded /tmp/dataflow-requirements-cache/six-1.11.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded 
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
  File was already downloaded /tmp/dataflow-requirements-cache/pbr-4.0.2.tar.gz
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr
test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
:339:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
:137:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You 

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90590=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90590
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 12/Apr/18 21:17
Start Date: 12/Apr/18 21:17
Worklog Time Spent: 10m 
  Work Description: XuMingmin commented on issue #4991: [BEAM-3983] [SQL] 
Tables interface supports BigQuery
URL: https://github.com/apache/beam/pull/4991#issuecomment-380947345
 
 
   What is `“Merge pull request” (the default)` in 
https://beam.apache.org/contribute/committer-guide/#merging-it? I only see 
`Squash and merge`(default value I see), `Create a commit merge` and `Rebase 
and merge`. Is it `Create a commit merge` ?
   
   Regarding to squash I would follow the guideline, ask contributors to 
pre-squash it and avoid fixup commits.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90590)
Time Spent: 4h  (was: 3h 50m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=90589=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90589
 ]

ASF GitHub Bot logged work on BEAM-4059:


Author: ASF GitHub Bot
Created on: 12/Apr/18 21:05
Start Date: 12/Apr/18 21:05
Worklog Time Spent: 10m 
  Work Description: pabloem commented on issue #5116: [BEAM-4059] 
Increasing timeout for Dataflow ValidatesRunner.
URL: https://github.com/apache/beam/pull/5116#issuecomment-380944043
 
 
   Run Dataflow ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90589)
Time Spent: 20m  (was: 10m)

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=90577=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90577
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 12/Apr/18 21:00
Start Date: 12/Apr/18 21:00
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5040: [BEAM-4044] [SQL] 
Refresh DDL from 1.16
URL: https://github.com/apache/beam/pull/5040#issuecomment-380942823
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90577)
Time Spent: 3.5h  (was: 3h 20m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3249) Use Gradle to build/release project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3249?focusedWorklogId=90575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90575
 ]

ASF GitHub Bot logged work on BEAM-3249:


Author: ASF GitHub Bot
Created on: 12/Apr/18 20:54
Start Date: 12/Apr/18 20:54
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5117: [BEAM-3249] Clean-up 
and use shaded test jars, removing evaluationDependsOn
URL: https://github.com/apache/beam/pull/5117#issuecomment-380941379
 
 
   R: @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90575)
Time Spent: 12h 50m  (was: 12h 40m)

> Use Gradle to build/release project
> ---
>
> Key: BEAM-3249
> URL: https://issues.apache.org/jira/browse/BEAM-3249
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 12h 50m
>  Remaining Estimate: 0h
>
> I have collected data by running several builds against master using Gradle 
> and Maven without using Gradle's support for incremental builds.
> Gradle (mins)
> min: 25.04
> max: 160.14
> median: 45.78
> average: 52.19
> stdev: 30.80
> Maven (mins)
> min: 56.86
> max: 216.55
> median: 87.93
> average: 109.10
> stdev: 48.01
> I excluded a few timeouts (240 mins) that happened during the Maven build 
> from its numbers but we can see conclusively that Gradle is about twice as 
> fast for the build when compared to Maven when run using Jenkins.
> Original dev@ thread: 
> https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E
> The data is available here 
> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3249) Use Gradle to build/release project

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3249?focusedWorklogId=90574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90574
 ]

ASF GitHub Bot logged work on BEAM-3249:


Author: ASF GitHub Bot
Created on: 12/Apr/18 20:54
Start Date: 12/Apr/18 20:54
Worklog Time Spent: 10m 
  Work Description: swegner commented on issue #5117: [BEAM-3249] Clean-up 
and use shaded test jars, removing evaluationDependsOn
URL: https://github.com/apache/beam/pull/5117#issuecomment-380941309
 
 
   FYI, this is the previous commit from #5107 which seems to have somehow 
gotten excluded after a force push.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90574)
Time Spent: 12h 40m  (was: 12.5h)

> Use Gradle to build/release project
> ---
>
> Key: BEAM-3249
> URL: https://issues.apache.org/jira/browse/BEAM-3249
> Project: Beam
>  Issue Type: Improvement
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Luke Cwik
>Priority: Major
>  Time Spent: 12h 40m
>  Remaining Estimate: 0h
>
> I have collected data by running several builds against master using Gradle 
> and Maven without using Gradle's support for incremental builds.
> Gradle (mins)
> min: 25.04
> max: 160.14
> median: 45.78
> average: 52.19
> stdev: 30.80
> Maven (mins)
> min: 56.86
> max: 216.55
> median: 87.93
> average: 109.10
> stdev: 48.01
> I excluded a few timeouts (240 mins) that happened during the Maven build 
> from its numbers but we can see conclusively that Gradle is about twice as 
> fast for the build when compared to Maven when run using Jenkins.
> Original dev@ thread: 
> https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E
> The data is available here 
> https://docs.google.com/spreadsheets/d/1MHVjF-xoI49_NJqEQakUgnNIQ7Qbjzu8Y1q_h3dbF1M/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=90572=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90572
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 12/Apr/18 20:43
Start Date: 12/Apr/18 20:43
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #412: [BEAM-3985] Document 
Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#issuecomment-380938273
 
 
   good question -- it looks like the changes are up on the website too, so I 
am unsure what that error means. @jasonkuster or @alanmyrvold should we just 
ignore this error?
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90572)
Time Spent: 3h 50m  (was: 3h 40m)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4059?focusedWorklogId=90570=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90570
 ]

ASF GitHub Bot logged work on BEAM-4059:


Author: ASF GitHub Bot
Created on: 12/Apr/18 20:38
Start Date: 12/Apr/18 20:38
Worklog Time Spent: 10m 
  Work Description: pabloem opened a new pull request #5116: [BEAM-4059] 
Increasing timeout for Dataflow ValidatesRunner.
URL: https://github.com/apache/beam/pull/5116
 
 
   This increases the timeout to 4 hours, while we can figure out why it's 
running so slowly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90570)
Time Spent: 10m
Remaining Estimate: 0h

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90568=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90568
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 12/Apr/18 20:37
Start Date: 12/Apr/18 20:37
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request #418: 
[BEAM-3995] Update Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418#discussion_r181214223
 
 

 ##
 File path: src/documentation/sdks/nexmark.md
 ##
 @@ -112,17 +122,19 @@ Performance:
 
 ## Benchmark launch configuration
 
-We can specify the Beam runner to use with maven profiles, available profiles 
are:
-
-direct-runner
-spark-runner
-flink-runner
-apex-runner
+The Nexmark launcher accepts the `--runner` argument as usual for programs that
+use Beam PipelineOptions to manage their command line arguments. In addition
+to this, the necessary dependencies must be configured.
 
-The runner must also be specified like in any other Beam pipeline using:
+When running via Gradle, the following two parameters control the execution:
 
---runner
+-P nexmark.args
+The command line to pass to the Nexmark main program.
 
+-P nexmark.runner
 
 Review comment:
   The alternative which I mean is that we could just make it like before where 
you choose a profile and `-P nexmark.runner` is a choice from an enum, not a 
magic Gradle subproject name.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90568)
Time Spent: 8.5h  (was: 8h 20m)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90565
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 12/Apr/18 20:36
Start Date: 12/Apr/18 20:36
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request #418: 
[BEAM-3995] Update Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418#discussion_r181213684
 
 

 ##
 File path: src/documentation/sdks/nexmark.md
 ##
 @@ -417,124 +429,192 @@ Yet to come
 
 ### Running SMOKE suite on the DirectRunner (local)
 
+The DirectRunner is default, so it is not required to pass `-Pnexmark.runner`.
+Here we do it for maximum clarity.
+
+The direct runner does not have separate batch and streaming modes, but the
+Nexmark launch does.
+
+These parameters leave on many of the DirectRunner's extra safety checks so the
+SMOKE suite can make sure there is nothing broken in the Nexmark suite.
+
 Batch Mode:
 
-mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main 
-Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE 
--streaming=false --manageResources=false --monitorJobs=true 
--enforceEncodability=true --enforceImmutability=true"
+./gradlew :beam-sdks-java-nexmark:run \
+-Pnexmark.runner=":beam-runners-direct-java" \
+-Pnexmark.args="
+--runner=DirectRunner
+--streaming=false
+--suite=SMOKE
+--manageResources=false
+--monitorJobs=true
+--enforceEncodability=true
+--enforceImmutability=true"
 
 Streaming Mode:
 
-mvn exec:java -Dexec.mainClass=org.apache.beam.sdk.nexmark.Main 
-Pdirect-runner -Dexec.args="--runner=DirectRunner --suite=SMOKE 
--streaming=true --manageResources=false --monitorJobs=true 
--enforceEncodability=true --enforceImmutability=true"
-
+./gradlew :beam-sdks-java-nexmark:run \
+-Pnexmark.runner=":beam-runners-direct-java" \
+-Pnexmark.args="
+--runner=DirectRunner
+--streaming=true
+--suite=SMOKE
+--manageResources=false
+--monitorJobs=true
+--enforceEncodability=true
+--enforceImmutability=true"
 
 ### Running SMOKE suite on the SparkRunner (local)
 
+The SparkRunner is special-cased in the Nexmark gradle launch. The task will
+provide the version of Spark that the SparkRunner is built against, and
+configure logging.
 
 Review comment:
   This is a comment about the hardcoding above. The nexmark build tweaks the 
dependencies based on special needs of the SparkRunner instead of being just a 
generic suite of pipelines. I will delete the comment here since it "just 
works" for users copying and pasting from here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90565)
Time Spent: 8h 10m  (was: 8h)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3995) Launch Nexmark suites from gradle and update web page docs

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3995?focusedWorklogId=90566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90566
 ]

ASF GitHub Bot logged work on BEAM-3995:


Author: ASF GitHub Bot
Created on: 12/Apr/18 20:36
Start Date: 12/Apr/18 20:36
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on a change in pull request #418: 
[BEAM-3995] Update Nexmark launch instructions for Gradle
URL: https://github.com/apache/beam-site/pull/418#discussion_r181212848
 
 

 ##
 File path: src/documentation/sdks/nexmark.md
 ##
 @@ -112,17 +122,19 @@ Performance:
 
 ## Benchmark launch configuration
 
-We can specify the Beam runner to use with maven profiles, available profiles 
are:
-
-direct-runner
-spark-runner
-flink-runner
-apex-runner
+The Nexmark launcher accepts the `--runner` argument as usual for programs that
+use Beam PipelineOptions to manage their command line arguments. In addition
+to this, the necessary dependencies must be configured.
 
-The runner must also be specified like in any other Beam pipeline using:
+When running via Gradle, the following two parameters control the execution:
 
---runner
+-P nexmark.args
+The command line to pass to the Nexmark main program.
 
+-P nexmark.runner
 
 Review comment:
   Yes, Gradle puts them on the classpath. The one exception is the Spark 
runner. I used to have a `-P nexmark.provideSpark` parameter to keep it a 
little decoupled, but since it is pretty much still "magic" knowledge, I 
followed @lukecwik's suggestion and just hardcoded knowledge of the dependency, 
so I did that 
[here](https://github.com/apache/beam/blob/master/sdks/java/nexmark/build.gradle#L35).
   
   I think when I add the fat jar capability it may require a separate fat jar 
for each known runner config. At that point, pretending that the runner 
dependency can be anything (versus a selection of a hardcoded profile) may not 
be useful.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90566)
Time Spent: 8h 20m  (was: 8h 10m)

> Launch Nexmark suites from gradle and update web page docs
> --
>
> Key: BEAM-3995
> URL: https://issues.apache.org/jira/browse/BEAM-3995
> Project: Beam
>  Issue Type: Sub-task
>  Components: examples-nexmark, website
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> Currently our instructions for running Nexmark benchmarks on various runners 
> is pretty tightly tied to Maven. We need a good story for running them with 
> gradle (or just building an executable with gradle and running that 
> standalone).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436260#comment-16436260
 ] 

Pablo Estrada commented on BEAM-4059:
-

I think for now the best thing is to increase the timeout, to get the tests to 
pass, and later work on improving the speed of the test. With that in mind, I 
will increase it to 4 hours, and add a todo pointing to this.

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90552
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 12/Apr/18 20:04
Start Date: 12/Apr/18 20:04
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4991: [BEAM-3983] [SQL] 
Tables interface supports BigQuery
URL: https://github.com/apache/beam/pull/4991#issuecomment-380927988
 
 
   Yea, it is subtle but doing the most natural "git" thing works best in the 
most situations. We have now documented this as the best practice in our 
committer guide: https://beam.apache.org/contribute/committer-guide/#merging-it


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90552)
Time Spent: 3h 50m  (was: 3h 40m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=90550=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90550
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 12/Apr/18 19:56
Start Date: 12/Apr/18 19:56
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #5040: [BEAM-4044] [SQL] 
Refresh DDL from 1.16
URL: https://github.com/apache/beam/pull/5040#issuecomment-380925898
 
 
   This is rebased and I changed the copyright message.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90550)
Time Spent: 3h 20m  (was: 3h 10m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90549
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 12/Apr/18 19:55
Start Date: 12/Apr/18 19:55
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #4991: [BEAM-3983] [SQL] 
Tables interface supports BigQuery
URL: https://github.com/apache/beam/pull/4991#issuecomment-380925504
 
 
   There are a few ways to get commits into master. This was merged by a rebase 
and squash. Github calls this "squash and merge", but there is no git merge 
performed. This operation changed the history of the commits. As a result, 
github was not able to automatically determine that the two changes shared a 
common subset. If it was merged with a merge commit, the commit chain would 
have been incorporated into master without change and the operation would not 
have broken #5040.
   
   The github documentation describes the difference here: 
https://help.github.com/articles/about-pull-request-merges/


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90549)
Time Spent: 3h 40m  (was: 3.5h)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4057) Ensure generated pom don't break consumers

[Romain Manni-Bucau (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436178#comment-16436178
 ] 

Romain Manni-Bucau commented on BEAM-4057:
--

[~kenn] all tools allows native browsing (check idea) and it is common to 
browse a project like that to grab dep cause like 60 times faster for some 
cases so it is done by users you like it or not (note i didnt say i like it ;)).

> Ensure generated pom don't break consumers
> --
>
> Key: BEAM-4057
> URL: https://issues.apache.org/jira/browse/BEAM-4057
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Romain Manni-Bucau
>Assignee: Luke Cwik
>Priority: Major
>
> Out of my head here are the requirements:
> 1. dependencies are all here (all scopes and well scoped: this means that 
> provided or test dependencies are not in compile scope for instance)
> 2. META-INF should contain the pom.xml and pom.properties as maven generates 
> them (it is consumes by tools and libraries to grab the dependencies or scan 
> some classpath/lib folder)
> 3. ensure the compiler plugin at least is defined with the java 
> version+compiler flags (a usage is to check if -parameters is activated for 
> instance)
> 4. (nice to have) dont put all the boilerplate in all poms (license, etc) but 
> keep it in the parent pom as it was
> 5. (if possible) respect the hierarchy (parents) - this is used sometimes as 
> a shortcut for dependencies analyzis cause it is faster than analyzing the 
> dependencies, probably not the best practise ever but it is efficient in 
> general
> 6. ensure meta used by mainstream tools like mvnrepository are here 
> (description etc, should be a passthrough from gradle)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #81

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436163#comment-16436163
 ] 

Pablo Estrada commented on BEAM-4059:
-

Issue tracking same problem, back when Maven timeout was increased: 
https://issues.apache.org/jira/browse/BEAM-3775

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3775) Failure in beam_PostCommit_Java_ValidatesRunner_Dataflow/

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436162#comment-16436162
 ] 

Pablo Estrada commented on BEAM-3775:
-

I've created a subtask for the gradle migration that covers this in 
https://issues.apache.org/jira/browse/BEAM-4059.

> Failure in beam_PostCommit_Java_ValidatesRunner_Dataflow/
> -
>
> Key: BEAM-3775
> URL: https://issues.apache.org/jira/browse/BEAM-3775
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: María GH
>Assignee: Thomas Groh
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Build timed out (after 120 minutes). Marking the build as aborted.
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/5070/
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436158#comment-16436158
 ] 

Pablo Estrada commented on BEAM-4059:
-

The previous Maven Build ran 283 dataflow jobs, and ran in about 2 hours: 
[https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/5290/consoleFull]

It doesn't seem like Jenkins is trying to build many more than that number of 
jobs, but still failed. Perhaps Maven has more executors?

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90538
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:57
Start Date: 12/Apr/18 18:57
Worklog Time Spent: 10m 
  Work Description: XuMingmin commented on issue #4991: [BEAM-3983] [SQL] 
Tables interface supports BigQuery
URL: https://github.com/apache/beam/pull/4991#issuecomment-380909893
 
 
   it should be merged to master already, I can see your change in latest code. 
Could you take another look?
   
   And yes, it does create a conflict with #5040 as the two both change some 
files and you need to rebase #5040 before it's able to merge.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90538)
Time Spent: 3.5h  (was: 3h 20m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4060) meta-info/poms missing in snapshots

[Rafal Wojdyla (JIRA)]

Rafal Wojdyla created BEAM-4060:
---

 Summary: meta-info/poms missing in snapshots
 Key: BEAM-4060
 URL: https://issues.apache.org/jira/browse/BEAM-4060
 Project: Beam
  Issue Type: Bug
  Components: build-system
Affects Versions: 2.5.0
Reporter: Rafal Wojdyla
Assignee: Luke Cwik


Current snapshots are missing bunch of meta-info files, including pom.xml and 
pom.properties:

2.4.0-SNAPSHOT example:
{noformat}
jar -tf 
~/.ivy2/cache/org.apache.beam/beam-runners-direct-java/jars/beam-runners-direct-java-2.4.0-SNAPSHOT.jar
 | grep META-INFMETA-INF/
META-INF/DEPENDENCIES
META-INF/LICENSE
META-INF/MANIFEST.MF
META-INF/NOTICE
META-INF/maven/
META-INF/maven/com.google.code.findbugs/
META-INF/maven/com.google.code.findbugs/jsr305/
META-INF/maven/com.google.code.findbugs/jsr305/pom.properties
META-INF/maven/com.google.code.findbugs/jsr305/pom.xml
META-INF/maven/com.google.guava/
META-INF/maven/com.google.guava/guava/
META-INF/maven/com.google.guava/guava/pom.properties
META-INF/maven/com.google.guava/guava/pom.xml
META-INF/maven/com.google.protobuf/
META-INF/maven/com.google.protobuf/protobuf-java-util/
META-INF/maven/com.google.protobuf/protobuf-java-util/pom.properties
META-INF/maven/com.google.protobuf/protobuf-java-util/pom.xml
META-INF/maven/com.google.protobuf/protobuf-java/
META-INF/maven/com.google.protobuf/protobuf-java/pom.properties
META-INF/maven/com.google.protobuf/protobuf-java/pom.xml
META-INF/maven/org.apache.beam/
META-INF/maven/org.apache.beam/beam-model-pipeline/
META-INF/maven/org.apache.beam/beam-model-pipeline/pom.properties
META-INF/maven/org.apache.beam/beam-model-pipeline/pom.xml
META-INF/maven/org.apache.beam/beam-runners-core-construction-java/
META-INF/maven/org.apache.beam/beam-runners-core-construction-java/pom.properties
META-INF/maven/org.apache.beam/beam-runners-core-construction-java/pom.xml
META-INF/maven/org.apache.beam/beam-runners-core-java/
META-INF/maven/org.apache.beam/beam-runners-core-java/pom.properties
META-INF/maven/org.apache.beam/beam-runners-core-java/pom.xml
META-INF/maven/org.apache.beam/beam-runners-direct-java/
META-INF/maven/org.apache.beam/beam-runners-direct-java/pom.properties
META-INF/maven/org.apache.beam/beam-runners-direct-java/pom.xml
META-INF/services/
META-INF/services/org.apache.beam.runners.direct.repackaged.runners.core.construction.CoderTranslatorRegistrar
META-INF/services/org.apache.beam.runners.direct.repackaged.runners.core.construction.TransformPayloadTranslatorRegistrar
META-INF/services/org.apache.beam.sdk.options.PipelineOptionsRegistrar
META-INF/services/org.apache.beam.sdk.runners.PipelineRunnerRegistrar
{noformat}

2.5.0-SNAPSHOT:

{noformat}
circleci@5abc19b95c60:~/scio$ jar -tf 
~/.ivy2/cache/org.apache.beam/beam-runners-direct-java/jars/beam-runners-direct-java-2.5.0-SNAPSHOT.jar
  | grep META-INF
META-INF/
META-INF/MANIFEST.MF
META-INF/services/
META-INF/maven/
META-INF/maven/com.google.guava/
META-INF/maven/com.google.guava/guava/
META-INF/maven/com.google.guava/guava/pom.properties
META-INF/maven/com.google.guava/guava/pom.xml
META-INF/maven/com.google.protobuf/
META-INF/maven/com.google.protobuf/protobuf-java-util/
META-INF/maven/com.google.protobuf/protobuf-java-util/pom.properties
META-INF/maven/com.google.protobuf/protobuf-java-util/pom.xml
META-INF/maven/com.google.protobuf/protobuf-java/
META-INF/maven/com.google.protobuf/protobuf-java/pom.properties
META-INF/maven/com.google.protobuf/protobuf-java/pom.xml
META-INF/services/org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.TransformPayloadTranslatorRegistrar
META-INF/services/org.apache.beam.sdk.runners.PipelineRunnerRegistrar
META-INF/services/org.apache.beam.sdk.options.PipelineOptionsRegistrar
META-INF/services/org.apache.beam.repackaged.beam_runners_direct_java.runners.core.construction.CoderTranslatorRegistrar
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90535=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90535
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:37
Start Date: 12/Apr/18 18:37
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #4991: [BEAM-3983] [SQL] 
Tables interface supports BigQuery
URL: https://github.com/apache/beam/pull/4991#issuecomment-380904102
 
 
   You did not merge this, you rebased it and squashed. That created a merge 
conflict with #5040 and #4991. Would you please merge in the future?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90535)
Time Spent: 3h 20m  (was: 3h 10m)

> BigQuery writes from pure SQL
> -
>
> Key: BEAM-3983
> URL: https://issues.apache.org/jira/browse/BEAM-3983
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> It would be nice if you could write to BigQuery in SQL without writing any 
> java code. For example:
> {code:java}
> INSERT INTO bigquery SELECT * FROM PCOLLECTION{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Apex_Gradle #80

[Apache Jenkins Server]

See 


--
[...truncated 14.65 MB...]
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-2
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-2 msg: [container-2] Entering heartbeat loop..
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-3
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-3 msg: [container-3] Entering heartbeat loop..
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-4
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-4 msg: [container-4] Entering heartbeat loop..
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-5
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-5 msg: [container-5] Entering heartbeat loop..
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-6
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-6 msg: [container-6] Entering heartbeat loop..
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-7
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-7 msg: [container-7] Entering heartbeat loop..
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-8
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-8 msg: [container-8] Entering heartbeat loop..
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-9
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-9 msg: [container-9] Entering heartbeat loop..
Apr 12, 2018 6:37:19 PM com.datatorrent.stram.Journal write
WARNING: Journal output stream is null. Skipping write to the WAL.
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$LocalStreamingContainerLauncher run
INFO: Started container container-10
Apr 12, 2018 6:37:19 PM 
com.datatorrent.stram.StramLocalCluster$UmbilicalProtocolLocalImpl log
INFO: container-10 msg: [container-10] Entering heartbeat loop..
Apr 12, 2018 6:37:20 PM com.datatorrent.stram.engine.StreamingContainer 
heartbeatLoop
INFO: Waiting for pending request.
Apr 12, 2018 6:37:20 PM com.datatorrent.stram.engine.StreamingContainer 
heartbeatLoop
INFO: Waiting for pending request.
Apr 12, 2018 6:37:20 PM com.datatorrent.stram.engine.StreamingContainer 
heartbeatLoop
INFO: Waiting for pending request.
Apr 12, 2018 6:37:20 PM com.datatorrent.stram.engine.StreamingContainer 
heartbeatLoop
INFO: Waiting for pending request.
Apr 12, 2018 6:37:20 PM com.datatorrent.stram.engine.StreamingContainer 
heartbeatLoop
INFO: Waiting for pending request.
Apr 12, 2018 6:37:20 PM com.datatorrent.stram.engine.StreamingContainer 
heartbeatLoop
INFO: Waiting for pending request.
Apr 12, 

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=90534=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90534
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:34
Start Date: 12/Apr/18 18:34
Worklog Time Spent: 10m 
  Work Description: XuMingmin commented on issue #5040: [BEAM-4044] [SQL] 
Refresh DDL from 1.16
URL: https://github.com/apache/beam/pull/5040#issuecomment-380903084
 
 
   can you rebase the PR as some files are also changed by #4991 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90534)
Time Spent: 3h 10m  (was: 3h)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3983) BigQuery writes from pure SQL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3983?focusedWorklogId=90531=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90531
 ]

ASF GitHub Bot logged work on BEAM-3983:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:33
Start Date: 12/Apr/18 18:33
Worklog Time Spent: 10m 
  Work Description: XuMingmin closed pull request #4991: [BEAM-3983] [SQL] 
Tables interface supports BigQuery
URL: https://github.com/apache/beam/pull/4991
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
index 5d484c90011..6598ee3e892 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
@@ -23,7 +23,7 @@
 import org.apache.beam.sdk.schemas.Schema;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.POutput;
 import org.apache.beam.sdk.values.Row;
 
 /**
@@ -46,7 +46,7 @@
* create a {@code IO.write()} instance to write to target.
*
*/
-   PTransform, PDone> buildIOWriter();
+   PTransform, POutput> buildIOWriter();
 
   /**
* Get the schema info of the table.
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
index 794e3e69966..15e8b960658 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
@@ -20,7 +20,6 @@
 import com.alibaba.fastjson.JSON;
 import com.alibaba.fastjson.JSONObject;
 import com.google.common.base.Strings;
-import java.net.URI;
 import java.util.List;
 import org.apache.calcite.sql.SqlCall;
 import org.apache.calcite.sql.SqlIdentifier;
@@ -109,8 +108,8 @@ public String tableName() {
 return tblName.toString();
   }
 
-  public URI location() {
-return location == null ? null : URI.create(getString(location));
+  public String location() {
+return location == null ? null : getString(location);
   }
 
   public String type() {
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
index 5a24f473366..0b10b4c6b19 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
@@ -22,7 +22,7 @@
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollection.IsBounded;
-import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.POutput;
 import org.apache.beam.sdk.values.Row;
 
 /**
@@ -56,7 +56,7 @@ public BeamIOType getSourceType() {
   }
 
   @Override
-  public PTransform, PDone> buildIOWriter() {
+  public PTransform, POutput> buildIOWriter() {
 throw new IllegalArgumentException("cannot use [BeamPCollectionTable] as 
target");
   }
 
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/Table.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/Table.java
index 4af82a0d6bd..a3339afc37d 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/Table.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/Table.java
@@ -21,7 +21,6 @@
 import com.alibaba.fastjson.JSONObject;
 import com.google.auto.value.AutoValue;
 import java.io.Serializable;
-import java.net.URI;
 import java.util.List;
 import javax.annotation.Nullable;
 
@@ -37,7 +36,7 @@
   @Nullable
   public abstract String getComment();
   @Nullable
-  public abstract URI getLocation();
+  public abstract String getLocation();
   @Nullable
   public abstract JSONObject getProperties();
 
@@ -45,14 +44,6 @@ public static Builder builder() {
 return new 
org.apache.beam.sdk.extensions.sql.meta.AutoValue_Table.Builder();
   }
 
-  public String getLocationAsString() 

[beam] branch master updated: [BEAM-3983] [SQL] Tables interface supports BigQuery (#4991)

[mingmxu]

This is an automated email from the ASF dual-hosted git repository.

mingmxu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new e981b43  [BEAM-3983] [SQL] Tables interface supports BigQuery (#4991)
e981b43 is described below

commit e981b439b6924dba20e7285091d343ae4d41765a
Author: Andrew Pilloud 
AuthorDate: Thu Apr 12 11:33:00 2018 -0700

[BEAM-3983] [SQL] Tables interface supports BigQuery (#4991)

* [SQL] POutput interface instead of PDone type

* [SQL] Location isn't necessarily a URI
---
 .../apache/beam/sdk/extensions/sql/BeamSqlTable.java|  4 ++--
 .../sdk/extensions/sql/impl/parser/SqlCreateTable.java  |  5 ++---
 .../sql/impl/schema/BeamPCollectionTable.java   |  4 ++--
 .../org/apache/beam/sdk/extensions/sql/meta/Table.java  | 13 ++---
 .../sql/meta/provider/kafka/BeamKafkaTable.java |  8 
 .../sql/meta/provider/text/BeamTextCSVTable.java|  4 ++--
 .../meta/provider/text/BeamTextCSVTableIOWriter.java|  6 +++---
 .../sql/meta/provider/text/BeamTextTable.java   |  3 +--
 .../sql/meta/provider/text/TextTableProvider.java   |  4 ++--
 .../apache/beam/sdk/extensions/sql/BeamSqlCliTest.java  |  8 
 .../extensions/sql/impl/parser/BeamSqlParserTest.java   | 17 ++---
 .../sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java |  4 ++--
 .../sql/meta/provider/kafka/KafkaTableProviderTest.java |  3 +--
 .../sql/meta/provider/text/TextTableProviderTest.java   |  3 +--
 .../sql/meta/store/InMemoryMetaStoreTest.java   |  3 +--
 .../sdk/extensions/sql/mock/MockedBoundedTable.java |  5 +++--
 .../beam/sdk/extensions/sql/mock/MockedTable.java   |  4 ++--
 17 files changed, 40 insertions(+), 58 deletions(-)

diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
index 5d484c9..6598ee3 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
@@ -23,7 +23,7 @@ import 
org.apache.beam.sdk.extensions.sql.impl.schema.BeamIOType;
 import org.apache.beam.sdk.schemas.Schema;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.values.PCollection;
-import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.POutput;
 import org.apache.beam.sdk.values.Row;
 
 /**
@@ -46,7 +46,7 @@ public interface BeamSqlTable {
* create a {@code IO.write()} instance to write to target.
*
*/
-   PTransform, PDone> buildIOWriter();
+   PTransform, POutput> buildIOWriter();
 
   /**
* Get the schema info of the table.
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
index 794e3e6..15e8b96 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/parser/SqlCreateTable.java
@@ -20,7 +20,6 @@ package org.apache.beam.sdk.extensions.sql.impl.parser;
 import com.alibaba.fastjson.JSON;
 import com.alibaba.fastjson.JSONObject;
 import com.google.common.base.Strings;
-import java.net.URI;
 import java.util.List;
 import org.apache.calcite.sql.SqlCall;
 import org.apache.calcite.sql.SqlIdentifier;
@@ -109,8 +108,8 @@ public class SqlCreateTable extends SqlCall {
 return tblName.toString();
   }
 
-  public URI location() {
-return location == null ? null : URI.create(getString(location));
+  public String location() {
+return location == null ? null : getString(location);
   }
 
   public String type() {
diff --git 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
index 5a24f47..0b10b4c 100644
--- 
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
+++ 
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BeamPCollectionTable.java
@@ -22,7 +22,7 @@ import org.apache.beam.sdk.schemas.Schema;
 import org.apache.beam.sdk.transforms.PTransform;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollection.IsBounded;
-import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.POutput;
 import org.apache.beam.sdk.values.Row;
 
 /**
@@ -56,7 +56,7 @@ public class BeamPCollectionTable 

Build failed in Jenkins: beam_PerformanceTests_Spark #1582

[Apache Jenkins Server]

See 


Changes:

[shoyer] [BEAM-3956] Preserve stacktraces for Python exceptions

[XuMingmin] KafkaIO : fix a typo in method name. (#5106)

--
[...truncated 89.92 KB...]
'apache-beam-testing:bqjob_r44a211605ba50a70_0162bb1c112d_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r44a211605ba50a70_0162bb1c112d_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r44a211605ba50a70_0162bb1c112d_1 ... (0s) Current status: DONE   
2018-04-12 18:26:43,647 c004ee49 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-12 18:27:11,681 c004ee49 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-12 18:27:13,935 c004ee49 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r55730b9a544aa5f_0162bb1c8856_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r55730b9a544aa5f_0162bb1c8856_1 ... (0s) 
Current status: RUNNING 
Waiting on 
bqjob_r55730b9a544aa5f_0162bb1c8856_1 ... (0s) Current status: DONE   
2018-04-12 18:27:13,936 c004ee49 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-12 18:27:38,280 c004ee49 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-12 18:27:41,192 c004ee49 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4e2052996ecf0424_0162bb1cf001_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)
Upload complete.Waiting on bqjob_r4e2052996ecf0424_0162bb1cf001_1 ... (0s) 
Current status: RUNNING 
 Waiting on 
bqjob_r4e2052996ecf0424_0162bb1cf001_1 ... (0s) Current status: DONE   
2018-04-12 18:27:41,192 c004ee49 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-12 18:28:05,179 c004ee49 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-12 18:28:07,912 c004ee49 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: 

BigQuery error in load operation: Error processing job

[jira] [Work logged] (BEAM-4044) Take advantage of Calcite DDL

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4044?focusedWorklogId=90530=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90530
 ]

ASF GitHub Bot logged work on BEAM-4044:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:30
Start Date: 12/Apr/18 18:30
Worklog Time Spent: 10m 
  Work Description: XuMingmin commented on a change in pull request #5040: 
[BEAM-4044] [SQL] Refresh DDL from 1.16
URL: https://github.com/apache/beam/pull/5040#discussion_r181179993
 
 

 ##
 File path: sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl
 ##
 @@ -1,106 +1,162 @@
-<#-- Licensed to the Apache Software Foundation (ASF) under one or more 
contributor
-  license agreements. See the NOTICE file distributed with this work for 
additional
-  information regarding copyright ownership. The ASF licenses this file to
-  You under the Apache License, Version 2.0 (the "License"); you may not use
-  this file except in compliance with the License. You may obtain a copy of
-  the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
-  by applicable law or agreed to in writing, software distributed under the
-  License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
-  OF ANY KIND, either express or implied. See the License for the specific
-  language governing permissions and limitations under the License. -->
+<#--
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to you under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
 
 Review comment:
   I remember there was a checkstyle error before, and James exclude the file 
and comment it. @apilloud could you revert these lines?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90530)
Time Spent: 3h  (was: 2h 50m)

> Take advantage of Calcite DDL
> -
>
> Key: BEAM-4044
> URL: https://issues.apache.org/jira/browse/BEAM-4044
> Project: Beam
>  Issue Type: New Feature
>  Components: dsl-sql
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> In Calcite 1.15 support for abstract DDL moved into calcite core. We should 
> take advantage of that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4059) Make sure Dataflow ValidatesRunner tests pass in Gradle

[Pablo Estrada (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436101#comment-16436101
 ] 

Pablo Estrada commented on BEAM-4059:
-

The test used to take 2 hours while running in Maven. I'm looking into why it 
has started taking over 3 hours - and thus timing out - in Gradle.

> Make sure Dataflow ValidatesRunner tests pass in Gradle
> ---
>
> Key: BEAM-4059
> URL: https://issues.apache.org/jira/browse/BEAM-4059
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>Priority: Major
>
> See 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT #136

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_XmlIOIT_HDFS #40

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT_HDFS #47

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90519=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90519
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:15
Start Date: 12/Apr/18 18:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5024: [BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181169858
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -498,9 +501,47 @@ def mkdirs(self, path):
 raise NotImplementedError
 
   @abc.abstractmethod
+  def has_dirs(self):
+"""Whether this FileSystem supports directories."""
+raise NotImplementedError
+
+  @abc.abstractmethod
+  def list(self, dir_or_prefix):
+"""List files in a location.
+
+Listing is non-recursive (for filesystems that support directories).
+
+Args:
+  dir_or_prefix: (string) A directory or location prefix (for filesystems
+that don't have directories).
+
+Returns:
+  Generator of ``FileMetadata`` objects.
+
+Raises:
+  ``BeamIOError`` if listing fails, but not if no files were found.
+"""
+raise NotImplementedError
+
+  @staticmethod
+  def _url_dirname(url_or_path):
+"""Like posixpath.dirname, but preserves scheme:// prefix.
+
+Args:
+  url_or_path: A string in the form of scheme://some/path OR /some/path.
+"""
+match = re.match(r'([a-z]+://)(.*)', url_or_path)
+if match is None:
+  return posixpath.dirname(url_or_path)
+url_prefix, path = match.groups()
+return url_prefix + posixpath.dirname(path)
+
   def match(self, patterns, limits=None):
 """Find all matching paths to the patterns provided.
 
+Pattern matching is done using fnmatch.fnmatch.
 
 Review comment:
   I think we should clarify if match for \/* is recursive or not. I 
think most users will use a pattern in the form \/\* which to 
avoid matching all sub-directories anyways.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90519)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90516
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:15
Start Date: 12/Apr/18 18:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5024: [BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181165982
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -498,9 +501,47 @@ def mkdirs(self, path):
 raise NotImplementedError
 
   @abc.abstractmethod
+  def has_dirs(self):
+"""Whether this FileSystem supports directories."""
+raise NotImplementedError
+
+  @abc.abstractmethod
+  def list(self, dir_or_prefix):
 
 Review comment:
   I think having both list() and match() as public can be confusing to users. 
Let's keep list() as private (and move it to filesystem implementations)  if 
there's no compelling use-case to keep it in the interface.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90516)
Time Spent: 1h 20m  (was: 1h 10m)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90517=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90517
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:15
Start Date: 12/Apr/18 18:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5024: [BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181161482
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -510,7 +551,48 @@ def match(self, patterns, limits=None):
 Raises:
   ``BeamIOError`` if any of the pattern match operations fail
 """
-raise NotImplementedError
+if limits is None:
+  limits = [None] * len(patterns)
+else:
+  err_msg = "Patterns and limits should be equal in length"
+  assert len(patterns) == len(limits), err_msg
+
+def _match(pattern, limit):
+  """Find all matching paths to the pattern provided."""
+  if pattern.endswith('/'):
+pattern += '*'
+  prefix_or_dir = re.match('^[^[*?]*', pattern).group(0)
 
 Review comment:
   Please add a comment explaining this regex.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90517)
Time Spent: 1.5h  (was: 1h 20m)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90515=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90515
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:15
Start Date: 12/Apr/18 18:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5024: [BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181162315
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -510,7 +551,48 @@ def match(self, patterns, limits=None):
 Raises:
   ``BeamIOError`` if any of the pattern match operations fail
 """
-raise NotImplementedError
+if limits is None:
+  limits = [None] * len(patterns)
+else:
+  err_msg = "Patterns and limits should be equal in length"
+  assert len(patterns) == len(limits), err_msg
+
+def _match(pattern, limit):
+  """Find all matching paths to the pattern provided."""
+  if pattern.endswith('/'):
+pattern += '*'
+  prefix_or_dir = re.match('^[^[*?]*', pattern).group(0)
+
+  file_metadatas = []
+  if prefix_or_dir == pattern:
+# Short-circuit calling self.list() if there's no glob pattern to 
match.
+if self.exists(pattern):
+  file_metadatas = [FileMetadata(pattern, self.size(pattern))]
+  else:
+if self.has_dirs():
 
 Review comment:
   Wasn't sure from code, but do we try to list "\/\*" for 
"\/\*" ?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90515)
Time Spent: 1h 10m  (was: 1h)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=90518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-90518
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 12/Apr/18 18:15
Start Date: 12/Apr/18 18:15
Worklog Time Spent: 10m 
  Work Description: chamikaramj commented on a change in pull request 
#5024: [BEAM-4011] Unify Python IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#discussion_r181163732
 
 

 ##
 File path: sdks/python/apache_beam/io/filesystem.py
 ##
 @@ -579,6 +661,19 @@ def exists(self, path):
 raise NotImplementedError
 
   @abc.abstractmethod
+  def size(self, path):
 
 Review comment:
   Why do we need a separate method for size() ? I think we can already stat 
files using the match() method (it returns FileMetada objects).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 90518)
Time Spent: 1h 40m  (was: 1.5h)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #1140

[Apache Jenkins Server]

See 


Changes:

[shoyer] [BEAM-3956] Preserve stacktraces for Python exceptions

[XuMingmin] KafkaIO : fix a typo in method name. (#5106)

--
[...truncated 3.98 KB...]
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2156203211161947545.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:339:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
:137:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
  Using cached 
https://files.pythonhosted.org/packages/eb/be/737f3df5806192ac4096e549e48c8c76cfaa2fb880a1c62a7bb085adaa9b/numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: idna<2.6,>=2.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.9.1->pywinrm->-r 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #42

[Apache Jenkins Server]

See 


Changes:

[shoyer] [BEAM-3956] Preserve stacktraces for Python exceptions

[XuMingmin] KafkaIO : fix a typo in method name. (#5106)

--
[...truncated 40.30 KB...]
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding 

Jenkins build is back to normal : beam_PerformanceTests_AvroIOIT_HDFS #41

[Apache Jenkins Server]

See