[jira] [Created] (BEAM-5520) Flink runner per operator SDK harness option

[Thomas Weise (JIRA)]

Thomas Weise created BEAM-5520:
--

 Summary: Flink runner per operator SDK harness option
 Key: BEAM-5520
 URL: https://issues.apache.org/jira/browse/BEAM-5520
 Project: Beam
  Issue Type: Improvement
  Components: runner-flink
Reporter: Thomas Weise
Assignee: Thomas Weise


For streaming pipelines, the runner currently uses a single SDK harness process 
for all subtasks of a job that get deployed on the same task manager. In common 
deployments with 16 or more tasks slots, many executable stage operators all 
use the same SDK harness process. To scale, we need an option to run separate 
harness processes per subtask.

   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1568

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1567

[Apache Jenkins Server]

See 


Changes:

[altay] update dataflow container name

--
[...truncated 39.38 MB...]
at 
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
... 3 more

Sep 28, 2018 1:22:22 AM 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn processElement
WARNING: Failed to submit the mutation group
com.google.cloud.spanner.SpannerException: FAILED_PRECONDITION: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:456)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.commit(GrpcSpannerRpc.java:404)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:797)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:794)
at 
com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:227)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl.writeAtLeastOnce(SpannerImpl.java:793)
at 
com.google.cloud.spanner.SessionPool$PooledSession.writeAtLeastOnce(SessionPool.java:319)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:60)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn.processElement(SpannerIO.java:1108)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:240)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:78)
at 
org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:207)
at 
org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:55)
at 
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
at 
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:450)
... 21 more
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must 
not be NULL in table users.
at io.grpc.Status.asRuntimeException(Status.java:526)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:468)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190)
at 

[jira] [Work logged] (BEAM-5443) Simplify Python pipeline options for portable runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5443?focusedWorklogId=148985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148985
 ]

ASF GitHub Bot logged work on BEAM-5443:


Author: ASF GitHub Bot
Created on: 28/Sep/18 00:30
Start Date: 28/Sep/18 00:30
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6512: 
[BEAM-5443] Pipeline option defaults for portable runner.
URL: https://github.com/apache/beam/pull/6512#discussion_r221114840
 
 

 ##
 File path: sdks/python/apache_beam/pipeline.py
 ##
 @@ -152,6 +153,12 @@ def __init__(self, runner=None, options=None, argv=None):
   raise ValueError(
   'Pipeline has validations errors: \n' + '\n'.join(errors))
 
+# set default experiments for portable runner
+# (needs to occur prior to pipeline construction)
+if self._options.view_as(StandardOptions).runner == 'PortableRunner':
+  self._options.view_as(DebugOptions).experiments = (
+self._options.view_as(DebugOptions).experiments or ['beam_fn_api'])
 
 Review comment:
   done!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148985)
Time Spent: 1h 10m  (was: 1h)

> Simplify Python pipeline options for portable runner
> 
>
> Key: BEAM-5443
> URL: https://issues.apache.org/jira/browse/BEAM-5443
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Affects Versions: 2.7.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently, the user needs to specify several extra pipeline options to run a 
> Python pipeline with the portable runner. It would be nice to remove the need 
> to explicitly provide these options when they can be inferred / have defaults.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Website_Publish #24

[Apache Jenkins Server]

See 


--
[...truncated 8.18 KB...]
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 1.49 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.025 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for 
':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for 
':buildSrc',5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.0 secs.
:buildSrc:test (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.002 secs.
:buildSrc:check (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:buildSrc:check (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:build (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:build
Skipping task 

Build failed in Jenkins: beam_PostCommit_Website_Publish #23

[Apache Jenkins Server]

See 


Changes:

[daniel.o.programmer] [BEAM-5304] Adding ReferenceRunner Job Server Gradle 
subproject.

--
[...truncated 8.32 KB...]
:buildSrc:assemble (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
completed. Took 0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
completed. Took 1.407 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) completed. Took 0.023 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
completed. Took 0.001 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) completed. Took 0.001 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
8,5,main]) completed. Took 0.001 secs.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 8,5,main]) 
completed. Took 0.0 secs.
:buildSrc:test (Thread[Task worker for ':buildSrc' Thread 8,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Task worker for ':buildSrc' Thread 8,5,main]) completed. 
Took 0.002 secs.
:buildSrc:check (Thread[Task worker for ':buildSrc' Thread 8,5,main]) started.

> Task 

[jira] [Work logged] (BEAM-5304) Support running user pipelines with the Universal Local Runner in Java.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5304?focusedWorklogId=148950=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148950
 ]

ASF GitHub Bot logged work on BEAM-5304:


Author: ASF GitHub Bot
Created on: 27/Sep/18 23:36
Start Date: 27/Sep/18 23:36
Worklog Time Spent: 10m 
  Work Description: pabloem closed pull request #6498: [BEAM-5304] Adding 
ReferenceRunner Job Server Gradle subproject.
URL: https://github.com/apache/beam/pull/6498
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/reference/java/build.gradle 
b/runners/reference/java/build.gradle
index 7bbaed09126..932a788b25f 100644
--- a/runners/reference/java/build.gradle
+++ b/runners/reference/java/build.gradle
@@ -29,6 +29,7 @@ configurations {
 }
 
 dependencies {
+  compile library.java.guava
   compile library.java.hamcrest_library
   shadow project(path: ":beam-model-pipeline", configuration: "shadow")
   shadow project(path: ":beam-runners-core-construction-java", configuration: 
"shadow")
diff --git a/runners/reference/job-server/build.gradle 
b/runners/reference/job-server/build.gradle
new file mode 100644
index 000..1c81fba2dd5
--- /dev/null
+++ b/runners/reference/job-server/build.gradle
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import groovy.json.JsonOutput
+
+apply plugin: org.apache.beam.gradle.BeamModulePlugin
+applyJavaNature(
+  validateShadowJar: false,
+  shadowClosure: {
+  }
+)
+
+description = "Apache Beam :: Runners :: Reference :: Job Server"
+
+apply plugin: "application"
+
+mainClassName = 
"org.apache.beam.runners.direct.portable.job.ReferenceRunnerJobServer"
+
+dependencies {
+  compile project(path: ":beam-runners-direct-java", configuration: "shadow")
+  compile project(path: ":beam-runners-java-fn-execution", configuration: 
"shadow")
+  compile library.java.slf4j_simple
+}
+
+run {
+  args = []
+  if (project.hasProperty('port'))
+args += ["--port=${project.property('port')}"]
+
+  // Enable remote debugging.
+  jvmArgs = ["-Xdebug", 
"-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005"]
+  if (project.hasProperty("logLevel"))
+jvmArgs += 
["-Dorg.slf4j.simpleLogger.defaultLogLevel=${project.property('logLevel')}"]
+}
diff --git a/settings.gradle b/settings.gradle
index 2c806ed0a47..3600e41e862 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -58,6 +58,8 @@ include "beam-runners-local-java-core"
 project(":beam-runners-local-java-core").dir = file("runners/local-java")
 include "beam-runners-reference-java"
 project(":beam-runners-reference-java").dir = file("runners/reference/java")
+include "beam-runners-reference-job-server"
+project(":beam-runners-reference-job-server").dir = 
file("runners/reference/job-server")
 include "beam-runners-spark"
 project(":beam-runners-spark").dir = file("runners/spark")
 include "beam-runners-samza"


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148950)
Time Spent: 1h 10m  (was: 1h)

> Support running user pipelines with the Universal Local Runner in Java.
> ---
>
> Key: BEAM-5304
> URL: https://issues.apache.org/jira/browse/BEAM-5304
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> In order to aid testing, devs should be able to write pipelines and 

[beam] branch master updated (e0b8c9f -> 5d298db)

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e0b8c9f  Merge pull request #6513 from aaltay/contc
 add 472feab  [BEAM-5304] Adding ReferenceRunner Job Server Gradle 
subproject.
 new 5d298db  Merge pull request #6498 from youngoli/beam5304

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/reference/java/build.gradle|  1 +
 .../reference/job-server}/build.gradle | 32 +-
 settings.gradle|  2 ++
 3 files changed, 28 insertions(+), 7 deletions(-)
 copy {model/fn-execution => runners/reference/job-server}/build.gradle (51%)

[beam] 01/01: Merge pull request #6498 from youngoli/beam5304

[pabloem]

This is an automated email from the ASF dual-hosted git repository.

pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5d298db4c20bbb8876a5b75142341332c1e3fb8d
Merge: e0b8c9f 472feab
Author: Pablo 
AuthorDate: Thu Sep 27 16:36:11 2018 -0700

Merge pull request #6498 from youngoli/beam5304

[BEAM-5304] Adding ReferenceRunner Job Server Gradle subproject.

 runners/reference/java/build.gradle   |  1 +
 runners/reference/job-server/build.gradle | 48 +++
 settings.gradle   |  2 ++
 3 files changed, 51 insertions(+)

[jira] [Work logged] (BEAM-5304) Support running user pipelines with the Universal Local Runner in Java.

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5304?focusedWorklogId=148947=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148947
 ]

ASF GitHub Bot logged work on BEAM-5304:


Author: ASF GitHub Bot
Created on: 27/Sep/18 23:31
Start Date: 27/Sep/18 23:31
Worklog Time Spent: 10m 
  Work Description: youngoli commented on issue #6498: [BEAM-5304] Adding 
ReferenceRunner Job Server Gradle subproject.
URL: https://github.com/apache/beam/pull/6498#issuecomment-425274484
 
 
   @pabloem For committer approval


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148947)
Time Spent: 1h  (was: 50m)

> Support running user pipelines with the Universal Local Runner in Java.
> ---
>
> Key: BEAM-5304
> URL: https://issues.apache.org/jira/browse/BEAM-5304
> Project: Beam
>  Issue Type: Sub-task
>  Components: runner-direct
>Reporter: Daniel Oliveira
>Assignee: Daniel Oliveira
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> In order to aid testing, devs should be able to write pipelines and then 
> easily run them with the ULR. The Flink runner seems to have this 
> functionality, so the implementation could be based on that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5509) Python pipeline_options doesn't handle int type

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5509?focusedWorklogId=148939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148939
 ]

ASF GitHub Bot logged work on BEAM-5509:


Author: ASF GitHub Bot
Created on: 27/Sep/18 23:14
Start Date: 27/Sep/18 23:14
Worklog Time Spent: 10m 
  Work Description: tweise opened a new pull request #6514: [BEAM-5509] Fix 
int/long pipeline option serialization in portable_runner
URL: https://github.com/apache/beam/pull/6514
 
 
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148939)
Time Spent: 10m
Remaining Estimate: 0h

> Python pipeline_options doesn't handle int type
> ---
>
> Key: BEAM-5509
> URL: https://issues.apache.org/jira/browse/BEAM-5509
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise

[jira] [Work logged] (BEAM-5509) Python pipeline_options doesn't handle int type

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5509?focusedWorklogId=148940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148940
 ]

ASF GitHub Bot logged work on BEAM-5509:


Author: ASF GitHub Bot
Created on: 27/Sep/18 23:14
Start Date: 27/Sep/18 23:14
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6514: [BEAM-5509] Fix 
int/long pipeline option serialization in portable_runner
URL: https://github.com/apache/beam/pull/6514#issuecomment-425271502
 
 
   R: @angoenka 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148940)
Time Spent: 20m  (was: 10m)

> Python pipeline_options doesn't handle int type
> ---
>
> Key: BEAM-5509
> URL: https://issues.apache.org/jira/browse/BEAM-5509
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> The int option supplied at the command line is turned into a decimal during 
> serialization and then the parser in SDK harness fails to restore it as int.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Website_Publish #22

[Apache Jenkins Server]

See 


Changes:

[altay] update dataflow container name

--
[...truncated 8.04 KB...]
> Task :buildSrc:assemble
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessGroovy (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Daemon worker,5,main]) completed. Took 2.64 
secs.
:buildSrc:spotlessGroovyCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Daemon worker,5,main]) completed. Took 
0.0 secs.
:buildSrc:spotlessGroovyGradle (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Daemon worker,5,main]) completed. Took 
0.029 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Daemon worker,5,main]) completed. 
Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 0.001 
secs.
:buildSrc:compileTestGroovy (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Daemon worker,5,main]) completed. Took 
0.001 secs.
:buildSrc:processTestResources (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Daemon worker,5,main]) completed. Took 
0.0 secs.
:buildSrc:testClasses (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:test (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.005 secs.
:buildSrc:check (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:buildSrc:check (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:build (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:build
Skipping task ':buildSrc:build' as it has no actions.
:buildSrc:build (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
Settings evaluated using settings file 

[beam] 01/01: Merge pull request #6513 from aaltay/contc

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit e0b8c9f636459e89d05cd7eeaba70d71fc4c6211
Merge: 97bf456 20f79b0
Author: Ahmet Altay 
AuthorDate: Thu Sep 27 16:12:32 2018 -0700

Merge pull request #6513 from aaltay/contc

update dataflow container name

 sdks/python/apache_beam/runners/dataflow/internal/names.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[beam] branch master updated (97bf456 -> e0b8c9f)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 97bf456  Merge pull request #6503 from kevinsi4508/update_bigtable_io2
 add 20f79b0  update dataflow container name
 new e0b8c9f  Merge pull request #6513 from aaltay/contc

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/dataflow/internal/names.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

[jira] [Work logged] (BEAM-5443) Simplify Python pipeline options for portable runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5443?focusedWorklogId=148936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148936
 ]

ASF GitHub Bot logged work on BEAM-5443:


Author: ASF GitHub Bot
Created on: 27/Sep/18 23:03
Start Date: 27/Sep/18 23:03
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6512: 
[BEAM-5443] Pipeline option defaults for portable runner.
URL: https://github.com/apache/beam/pull/6512#discussion_r221102810
 
 

 ##
 File path: sdks/python/apache_beam/pipeline.py
 ##
 @@ -152,6 +153,12 @@ def __init__(self, runner=None, options=None, argv=None):
   raise ValueError(
   'Pipeline has validations errors: \n' + '\n'.join(errors))
 
+# set default experiments for portable runner
+# (needs to occur prior to pipeline construction)
+if self._options.view_as(StandardOptions).runner == 'PortableRunner':
+  self._options.view_as(DebugOptions).experiments = (
+self._options.view_as(DebugOptions).experiments or ['beam_fn_api'])
 
 Review comment:
   I think it should be ok to add beam_fn_api always as without it the pipeline 
will fail. 
   And the behavior will be inconsistent where when no experiment is provided 
then the pipeline works but when when experiments is provided without explicit 
beam_fn_api, the pipeline fails.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148936)
Time Spent: 1h  (was: 50m)

> Simplify Python pipeline options for portable runner
> 
>
> Key: BEAM-5443
> URL: https://issues.apache.org/jira/browse/BEAM-5443
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Affects Versions: 2.7.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently, the user needs to specify several extra pipeline options to run a 
> Python pipeline with the portable runner. It would be nice to remove the need 
> to explicitly provide these options when they can be inferred / have defaults.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5509) Python pipeline_options doesn't handle int type

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise reassigned BEAM-5509:
--

Assignee: Thomas Weise  (was: Robert Bradshaw)

> Python pipeline_options doesn't handle int type
> ---
>
> Key: BEAM-5509
> URL: https://issues.apache.org/jira/browse/BEAM-5509
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability-flink
>
> The int option supplied at the command line is turned into a decimal during 
> serialization and then the parser in SDK harness fails to restore it as int.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5509) Python pipeline_options doesn't handle int type

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631152#comment-16631152
 ] 

Thomas Weise commented on BEAM-5509:


The unwanted conversion to floating point occurs in 
[https://github.com/apache/beam/blob/6f10dd36b7f01758675e244f6da86f27bcbcea6a/sdks/python/apache_beam/runners/job/utils.py#L30]
{code:java}
json_format.Parse(json.dumps(dict_obj), struct_pb2.Struct()){code}
Specifically, when the json (which does not contain a floating point literal is 
turned back to the struct.

Converting all int and long values to string prior to calling this utility 
avoids the issue.

> Python pipeline_options doesn't handle int type
> ---
>
> Key: BEAM-5509
> URL: https://issues.apache.org/jira/browse/BEAM-5509
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-harness
>Reporter: Thomas Weise
>Assignee: Robert Bradshaw
>Priority: Major
>  Labels: portability-flink
>
> The int option supplied at the command line is turned into a decimal during 
> serialization and then the parser in SDK harness fails to restore it as int.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5496) MqttIO fails to deserialize checkpoint

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-5496:
---

Assignee: Island Chen

> MqttIO fails to deserialize checkpoint
> --
>
> Key: BEAM-5496
> URL: https://issues.apache.org/jira/browse/BEAM-5496
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-mqtt
>Reporter: Luke Cwik
>Assignee: Island Chen
>Priority: Major
>
> Source of bug report: 
> [https://lists.apache.org/thread.html/3de5a946bcb539dea9f18a31f712d6af5b66f9fbb6b01eed452c5afb@%3Cdev.beam.apache.org%3E]
>  
> There is a bug of the built-in MqttIO, please check the 
> ,
>  this readObject() method forget to invoke the "stream.defaultReadObject()" 
> method.
>  
> {code:java}
> // set an empty list to messages when deserialize
> private void readObject(java.io.ObjectInputStream stream)
> throws IOException, ClassNotFoundException {
>   messages = new ArrayList<>();
> }{code}
>  
> So there is an exception while the runner tried to deserialize the checkpoint 
> object.
> {code:java}
> java.lang.RuntimeException: org.apache.beam.sdk.coders.CoderException: 95 
> unexpected extra bytes after decoding 
> org.apache.beam.sdk.io.mqtt.MqttIO$MqttCheckpointMark@6764e219
>  at 
> org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:340)
> ...{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5378) Ensure all Go SDK examples run successfully

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5378?focusedWorklogId=148928=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148928
 ]

ASF GitHub Bot logged work on BEAM-5378:


Author: ASF GitHub Bot
Created on: 27/Sep/18 22:09
Start Date: 27/Sep/18 22:09
Worklog Time Spent: 10m 
  Work Description: melap commented on a change in pull request #6386: 
[BEAM-5378] - Update minimal_wordcount.go to reflect documentation
URL: https://github.com/apache/beam/pull/6386#discussion_r221092847
 
 

 ##
 File path: website/src/get-started/wordcount-example.md
 ##
 @@ -90,7 +90,7 @@ To view the full code in Python, see
 
 {:.language-go}
 To view the full code in Go, see
-**[wordcount_minimal.go](https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go).**
+**[minimal_wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go).**
 
 Review comment:
   Correct, changes to website markdown still need to be done in beam-site 
until the migration is complete


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148928)
Time Spent: 3h 40m  (was: 3.5h)

> Ensure all Go SDK examples run successfully
> ---
>
> Key: BEAM-5378
> URL: https://issues.apache.org/jira/browse/BEAM-5378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Tomas Roos
>Priority: Major
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> I've been spending a day or so running through the example available for the 
> Go SDK in order to see what works and on what runner (direct, dataflow), and 
> what doesn't and here's the results.
> All available examples for the go sdk. For me as a new developer on apache 
> beam and dataflow it would be a tremendous value to have all examples running 
> because many of them have legitimate use-cases behind them. 
> {code:java}
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> ├── contains
> │   └── contains.go
> ├── cookbook
> │   ├── combine
> │   │   └── combine.go
> │   ├── filter
> │   │   └── filter.go
> │   ├── join
> │   │   └── join.go
> │   ├── max
> │   │   └── max.go
> │   └── tornadoes
> │   └── tornadoes.go
> ├── debugging_wordcount
> │   └── debugging_wordcount.go
> ├── forest
> │   └── forest.go
> ├── grades
> │   └── grades.go
> ├── minimal_wordcount
> │   └── minimal_wordcount.go
> ├── multiout
> │   └── multiout.go
> ├── pingpong
> │   └── pingpong.go
> ├── streaming_wordcap
> │   └── wordcap.go
> ├── windowed_wordcount
> │   └── windowed_wordcount.go
> ├── wordcap
> │   └── wordcap.go
> ├── wordcount
> │   └── wordcount.go
> └── yatzy
> └── yatzy.go
> {code}
> All examples that are supposed to be runnable by the direct driver (not 
> depending on gcp platform services) are runnable.
> On the otherhand these are the tests that needs to be updated because its not 
> runnable on the dataflow platform for various reasons.
> I tried to figure them out and all I can do is to pin point at least where it 
> fails since my knowledge so far in the beam / dataflow internals is limited.
> .
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> Runs successfully if swapping the input to one of the shakespear data files 
> from gs://
> But when running this it yields a error from the top.Largest func (discussed 
> in another issue that top.Largest needs to have a serializeable combinator / 
> accumulator)
> ➜  autocomplete git:(master) ✗ ./autocomplete --project fair-app-213019 
> --runner dataflow --staging_location=gs://fair-app-213019/staging-test2 
> --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
>  
> 2018/09/11 15:35:26 Running autocomplete
> Unable to encode combiner for lifting: failed to encode custom coder: bad 
> underlying type: bad field type: bad element: unencodable type: interface 
> {}2018/09/11 15:35:26 Using running binary as worker binary: './autocomplete'
> 2018/09/11 15:35:26 Staging worker binary: ./autocomplete
> ├── contains
> │   └── contains.go
> Fails when running debug.Head for some mysterious reason, might have to do 
> with the param passing into the x,y iterator. Frankly I dont know and could 
> not figure.
> But removing the debug.Head call everything works as expected and succeeds.
> ├── cookbook
> │   ├── combine
> │   │   └── combine.go
> 

[jira] [Work logged] (BEAM-5378) Ensure all Go SDK examples run successfully

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5378?focusedWorklogId=148926=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148926
 ]

ASF GitHub Bot logged work on BEAM-5378:


Author: ASF GitHub Bot
Created on: 27/Sep/18 22:07
Start Date: 27/Sep/18 22:07
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6386: 
[BEAM-5378] - Update minimal_wordcount.go to reflect documentation
URL: https://github.com/apache/beam/pull/6386#discussion_r221092281
 
 

 ##
 File path: website/src/get-started/wordcount-example.md
 ##
 @@ -90,7 +90,7 @@ To view the full code in Python, see
 
 {:.language-go}
 To view the full code in Go, see
-**[wordcount_minimal.go](https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go).**
+**[minimal_wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go).**
 
 Review comment:
   FYI, I don't think the site uses this copy in beam just yet.
   
   @melap Is that still true? Does this change need to go into the old site 
repo still?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148926)
Time Spent: 3h 20m  (was: 3h 10m)

> Ensure all Go SDK examples run successfully
> ---
>
> Key: BEAM-5378
> URL: https://issues.apache.org/jira/browse/BEAM-5378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Tomas Roos
>Priority: Major
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> I've been spending a day or so running through the example available for the 
> Go SDK in order to see what works and on what runner (direct, dataflow), and 
> what doesn't and here's the results.
> All available examples for the go sdk. For me as a new developer on apache 
> beam and dataflow it would be a tremendous value to have all examples running 
> because many of them have legitimate use-cases behind them. 
> {code:java}
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> ├── contains
> │   └── contains.go
> ├── cookbook
> │   ├── combine
> │   │   └── combine.go
> │   ├── filter
> │   │   └── filter.go
> │   ├── join
> │   │   └── join.go
> │   ├── max
> │   │   └── max.go
> │   └── tornadoes
> │   └── tornadoes.go
> ├── debugging_wordcount
> │   └── debugging_wordcount.go
> ├── forest
> │   └── forest.go
> ├── grades
> │   └── grades.go
> ├── minimal_wordcount
> │   └── minimal_wordcount.go
> ├── multiout
> │   └── multiout.go
> ├── pingpong
> │   └── pingpong.go
> ├── streaming_wordcap
> │   └── wordcap.go
> ├── windowed_wordcount
> │   └── windowed_wordcount.go
> ├── wordcap
> │   └── wordcap.go
> ├── wordcount
> │   └── wordcount.go
> └── yatzy
> └── yatzy.go
> {code}
> All examples that are supposed to be runnable by the direct driver (not 
> depending on gcp platform services) are runnable.
> On the otherhand these are the tests that needs to be updated because its not 
> runnable on the dataflow platform for various reasons.
> I tried to figure them out and all I can do is to pin point at least where it 
> fails since my knowledge so far in the beam / dataflow internals is limited.
> .
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> Runs successfully if swapping the input to one of the shakespear data files 
> from gs://
> But when running this it yields a error from the top.Largest func (discussed 
> in another issue that top.Largest needs to have a serializeable combinator / 
> accumulator)
> ➜  autocomplete git:(master) ✗ ./autocomplete --project fair-app-213019 
> --runner dataflow --staging_location=gs://fair-app-213019/staging-test2 
> --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
>  
> 2018/09/11 15:35:26 Running autocomplete
> Unable to encode combiner for lifting: failed to encode custom coder: bad 
> underlying type: bad field type: bad element: unencodable type: interface 
> {}2018/09/11 15:35:26 Using running binary as worker binary: './autocomplete'
> 2018/09/11 15:35:26 Staging worker binary: ./autocomplete
> ├── contains
> │   └── contains.go
> Fails when running debug.Head for some mysterious reason, might have to do 
> with the param passing into the x,y iterator. Frankly I dont know and could 
> not figure.
> But removing the debug.Head call everything works as expected and succeeds.
> ├── cookbook
> 

[jira] [Work logged] (BEAM-5378) Ensure all Go SDK examples run successfully

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5378?focusedWorklogId=148927=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148927
 ]

ASF GitHub Bot logged work on BEAM-5378:


Author: ASF GitHub Bot
Created on: 27/Sep/18 22:07
Start Date: 27/Sep/18 22:07
Worklog Time Spent: 10m 
  Work Description: lostluck commented on a change in pull request #6386: 
[BEAM-5378] - Update minimal_wordcount.go to reflect documentation
URL: https://github.com/apache/beam/pull/6386#discussion_r221092281
 
 

 ##
 File path: website/src/get-started/wordcount-example.md
 ##
 @@ -90,7 +90,7 @@ To view the full code in Python, see
 
 {:.language-go}
 To view the full code in Go, see
-**[wordcount_minimal.go](https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go).**
+**[minimal_wordcount.go](https://github.com/apache/beam/blob/master/sdks/go/examples/minimal_wordcount/minimal_wordcount.go).**
 
 Review comment:
   FYI, I don't think the site uses this copy in master just yet.
   
   @melap Is that still true? Does this change need to go into the old site 
repo still?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148927)
Time Spent: 3.5h  (was: 3h 20m)

> Ensure all Go SDK examples run successfully
> ---
>
> Key: BEAM-5378
> URL: https://issues.apache.org/jira/browse/BEAM-5378
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Affects Versions: Not applicable
>Reporter: Tomas Roos
>Priority: Major
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> I've been spending a day or so running through the example available for the 
> Go SDK in order to see what works and on what runner (direct, dataflow), and 
> what doesn't and here's the results.
> All available examples for the go sdk. For me as a new developer on apache 
> beam and dataflow it would be a tremendous value to have all examples running 
> because many of them have legitimate use-cases behind them. 
> {code:java}
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> ├── contains
> │   └── contains.go
> ├── cookbook
> │   ├── combine
> │   │   └── combine.go
> │   ├── filter
> │   │   └── filter.go
> │   ├── join
> │   │   └── join.go
> │   ├── max
> │   │   └── max.go
> │   └── tornadoes
> │   └── tornadoes.go
> ├── debugging_wordcount
> │   └── debugging_wordcount.go
> ├── forest
> │   └── forest.go
> ├── grades
> │   └── grades.go
> ├── minimal_wordcount
> │   └── minimal_wordcount.go
> ├── multiout
> │   └── multiout.go
> ├── pingpong
> │   └── pingpong.go
> ├── streaming_wordcap
> │   └── wordcap.go
> ├── windowed_wordcount
> │   └── windowed_wordcount.go
> ├── wordcap
> │   └── wordcap.go
> ├── wordcount
> │   └── wordcount.go
> └── yatzy
> └── yatzy.go
> {code}
> All examples that are supposed to be runnable by the direct driver (not 
> depending on gcp platform services) are runnable.
> On the otherhand these are the tests that needs to be updated because its not 
> runnable on the dataflow platform for various reasons.
> I tried to figure them out and all I can do is to pin point at least where it 
> fails since my knowledge so far in the beam / dataflow internals is limited.
> .
> ├── complete
> │   └── autocomplete
> │   └── autocomplete.go
> Runs successfully if swapping the input to one of the shakespear data files 
> from gs://
> But when running this it yields a error from the top.Largest func (discussed 
> in another issue that top.Largest needs to have a serializeable combinator / 
> accumulator)
> ➜  autocomplete git:(master) ✗ ./autocomplete --project fair-app-213019 
> --runner dataflow --staging_location=gs://fair-app-213019/staging-test2 
> --worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
>  
> 2018/09/11 15:35:26 Running autocomplete
> Unable to encode combiner for lifting: failed to encode custom coder: bad 
> underlying type: bad field type: bad element: unencodable type: interface 
> {}2018/09/11 15:35:26 Using running binary as worker binary: './autocomplete'
> 2018/09/11 15:35:26 Staging worker binary: ./autocomplete
> ├── contains
> │   └── contains.go
> Fails when running debug.Head for some mysterious reason, might have to do 
> with the param passing into the x,y iterator. Frankly I dont know and could 
> not figure.
> But removing the debug.Head call everything works as expected and succeeds.
> ├── cookbook
> │ 

[jira] [Work logged] (BEAM-5443) Simplify Python pipeline options for portable runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5443?focusedWorklogId=148925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148925
 ]

ASF GitHub Bot logged work on BEAM-5443:


Author: ASF GitHub Bot
Created on: 27/Sep/18 22:03
Start Date: 27/Sep/18 22:03
Worklog Time Spent: 10m 
  Work Description: tweise commented on a change in pull request #6512: 
[BEAM-5443] Pipeline option defaults for portable runner.
URL: https://github.com/apache/beam/pull/6512#discussion_r221091810
 
 

 ##
 File path: sdks/python/apache_beam/pipeline.py
 ##
 @@ -152,6 +153,12 @@ def __init__(self, runner=None, options=None, argv=None):
   raise ValueError(
   'Pipeline has validations errors: \n' + '\n'.join(errors))
 
+# set default experiments for portable runner
+# (needs to occur prior to pipeline construction)
+if self._options.view_as(StandardOptions).runner == 'PortableRunner':
+  self._options.view_as(DebugOptions).experiments = (
+self._options.view_as(DebugOptions).experiments or ['beam_fn_api'])
 
 Review comment:
   The idea was to only touch experiments when the user did not set it. Do you 
think we should just always add `beam_fn_api` instead?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148925)
Time Spent: 50m  (was: 40m)

> Simplify Python pipeline options for portable runner
> 
>
> Key: BEAM-5443
> URL: https://issues.apache.org/jira/browse/BEAM-5443
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Affects Versions: 2.7.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently, the user needs to specify several extra pipeline options to run a 
> Python pipeline with the portable runner. It would be nice to remove the need 
> to explicitly provide these options when they can be inferred / have defaults.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5443) Simplify Python pipeline options for portable runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5443?focusedWorklogId=148923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148923
 ]

ASF GitHub Bot logged work on BEAM-5443:


Author: ASF GitHub Bot
Created on: 27/Sep/18 21:58
Start Date: 27/Sep/18 21:58
Worklog Time Spent: 10m 
  Work Description: angoenka commented on a change in pull request #6512: 
[BEAM-5443] Pipeline option defaults for portable runner.
URL: https://github.com/apache/beam/pull/6512#discussion_r221090444
 
 

 ##
 File path: sdks/python/apache_beam/pipeline.py
 ##
 @@ -152,6 +153,12 @@ def __init__(self, runner=None, options=None, argv=None):
   raise ValueError(
   'Pipeline has validations errors: \n' + '\n'.join(errors))
 
+# set default experiments for portable runner
+# (needs to occur prior to pipeline construction)
+if self._options.view_as(StandardOptions).runner == 'PortableRunner':
+  self._options.view_as(DebugOptions).experiments = (
+self._options.view_as(DebugOptions).experiments or ['beam_fn_api'])
 
 Review comment:
   We should check and append`beam_fn_api` to experiments.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148923)
Time Spent: 40m  (was: 0.5h)

> Simplify Python pipeline options for portable runner
> 
>
> Key: BEAM-5443
> URL: https://issues.apache.org/jira/browse/BEAM-5443
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Affects Versions: 2.7.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently, the user needs to specify several extra pipeline options to run a 
> Python pipeline with the portable runner. It would be nice to remove the need 
> to explicitly provide these options when they can be inferred / have defaults.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5443) Simplify Python pipeline options for portable runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5443?focusedWorklogId=148920=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148920
 ]

ASF GitHub Bot logged work on BEAM-5443:


Author: ASF GitHub Bot
Created on: 27/Sep/18 21:54
Start Date: 27/Sep/18 21:54
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6512: [BEAM-5443] Pipeline 
option defaults for portable runner.
URL: https://github.com/apache/beam/pull/6512#issuecomment-425255509
 
 
   The command line with these changes:
   ```
   python -m apache_beam.examples.wordcount   --input=/etc/profile   
--output=/tmp/py-wordcount-direct   --runner=PortableRunner  
--job_endpoint=localhost:8099  --streaming
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148920)
Time Spent: 0.5h  (was: 20m)

> Simplify Python pipeline options for portable runner
> 
>
> Key: BEAM-5443
> URL: https://issues.apache.org/jira/browse/BEAM-5443
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Affects Versions: 2.7.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Currently, the user needs to specify several extra pipeline options to run a 
> Python pipeline with the portable runner. It would be nice to remove the need 
> to explicitly provide these options when they can be inferred / have defaults.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch asf-site updated: Publishing website 2018/09/27 21:52:18 at commit 901588e

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new b71c78d  Publishing website 2018/09/27 21:52:18 at commit 901588e
b71c78d is described below

commit b71c78d61b95c179dd9532de78c7d557b6befc11
Author: jenkins 
AuthorDate: Thu Sep 27 21:52:19 2018 +

Publishing website 2018/09/27 21:52:18 at commit 901588e

[jira] [Work logged] (BEAM-5443) Simplify Python pipeline options for portable runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5443?focusedWorklogId=148917=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148917
 ]

ASF GitHub Bot logged work on BEAM-5443:


Author: ASF GitHub Bot
Created on: 27/Sep/18 21:48
Start Date: 27/Sep/18 21:48
Worklog Time Spent: 10m 
  Work Description: tweise opened a new pull request #6512: [BEAM-5443] 
Pipeline option defaults for portable runner.
URL: https://github.com/apache/beam/pull/6512
 
 
   Attempt to set portable runner specific defaults for experiments and 
sdk_location options.
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148917)
Time Spent: 10m
Remaining Estimate: 0h

> Simplify Python pipeline options for portable runner
> 
>
> Key: BEAM-5443
> URL: https://issues.apache.org/jira/browse/BEAM-5443
> Project: Beam
>  Issue Type: Improvement
>  

[jira] [Work logged] (BEAM-5443) Simplify Python pipeline options for portable runner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5443?focusedWorklogId=148918=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148918
 ]

ASF GitHub Bot logged work on BEAM-5443:


Author: ASF GitHub Bot
Created on: 27/Sep/18 21:48
Start Date: 27/Sep/18 21:48
Worklog Time Spent: 10m 
  Work Description: tweise commented on issue #6512: [BEAM-5443] Pipeline 
option defaults for portable runner.
URL: https://github.com/apache/beam/pull/6512#issuecomment-425254052
 
 
   R: @angoenka 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148918)
Time Spent: 20m  (was: 10m)

> Simplify Python pipeline options for portable runner
> 
>
> Key: BEAM-5443
> URL: https://issues.apache.org/jira/browse/BEAM-5443
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Affects Versions: 2.7.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently, the user needs to specify several extra pipeline options to run a 
> Python pipeline with the portable runner. It would be nice to remove the need 
> to explicitly provide these options when they can be inferred / have defaults.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5443) Simplify Python pipeline options for portable runner

[Thomas Weise (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Weise reassigned BEAM-5443:
--

Assignee: Thomas Weise

> Simplify Python pipeline options for portable runner
> 
>
> Key: BEAM-5443
> URL: https://issues.apache.org/jira/browse/BEAM-5443
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-harness
>Affects Versions: 2.7.0
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Major
>  Labels: portability
>
> Currently, the user needs to specify several extra pipeline options to run a 
> Python pipeline with the portable runner. It would be nice to remove the need 
> to explicitly provide these options when they can be inferred / have defaults.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch asf-site updated: Publishing website 2018/09/27 21:33:04 at commit 24de10e

[git-site-role]

This is an automated email from the ASF dual-hosted git repository.

git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/asf-site by this push:
 new dccd0bd  Publishing website 2018/09/27 21:33:04 at commit 24de10e
dccd0bd is described below

commit dccd0bde793942357a940e9821167678653c4fa2
Author: jenkins 
AuthorDate: Thu Sep 27 21:33:05 2018 +

Publishing website 2018/09/27 21:33:04 at commit 24de10e
---
 website/generated-content/contribute/index.html  | 2 +-
 .../generated-content/documentation/programming-guide/index.html | 2 +-
 website/generated-content/get-started/quickstart-go/index.html   | 3 +++
 .../generated-content/get-started/wordcount-example/index.html   | 9 +
 4 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/website/generated-content/contribute/index.html 
b/website/generated-content/contribute/index.html
index f4a3af3..099c100 100644
--- a/website/generated-content/contribute/index.html
+++ b/website/generated-content/contribute/index.html
@@ -618,7 +618,7 @@ When submitting a new PR, please tag https://github.com/robbesneyders;>
 Work to support the next LTS release of Java is in progress. For more 
details about the scope and info on the various tasks please see the JIRA 
ticket.
 
 
-  JIRA: https://issues.apache.org/jira/issues/BEAM-2530;>BEAM-2530
+  JIRA: https://issues.apache.org/jira/browse/BEAM-2530;>BEAM-2530
   Contact: mailto:ieme...@gmail.com;>Ismaël Mejía
 
 
diff --git 
a/website/generated-content/documentation/programming-guide/index.html 
b/website/generated-content/documentation/programming-guide/index.html
index 2f19f9c..4836731 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -2902,7 +2902,7 @@ least 10 minutes:
 
 from apache_beam import window
 session_windowed_items = (
-items | 'window'  beam.WindowInto(window.Sessions(10)))
+items | 'window'  beam.WindowInto(window.Sessions(10 
* 60)))
 
 
 
diff --git a/website/generated-content/get-started/quickstart-go/index.html 
b/website/generated-content/get-started/quickstart-go/index.html
index 8a2435a..dbfae68 100644
--- a/website/generated-content/get-started/quickstart-go/index.html
+++ b/website/generated-content/get-started/quickstart-go/index.html
@@ -245,11 +245,14 @@ $ wordcount --input PATH_TO_INPUT_FILE --output 
counts
 
 
 $ 
go install github.com/apache/beam/sdks/go/examples/wordcount
+# As part of the initial setup, for non linux users - install package unix 
before run
+$ go get -u golang.org/x/sys/unix
 $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
 --output gs://your-gcs-bucket/counts \
 --runner dataflow \
 --project your-gcp-project \
 --temp_location gs://your-gcs-bucket/tmp/ \
+--staging_location gs://your-gcs-bucket/binaries/ \
 
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
 
 
diff --git a/website/generated-content/get-started/wordcount-example/index.html 
b/website/generated-content/get-started/wordcount-example/index.html
index 0b8d502..7878541 100644
--- a/website/generated-content/get-started/wordcount-example/index.html
+++ b/website/generated-content/get-started/wordcount-example/index.html
@@ -689,11 +689,14 @@ $ wordcount --input PATH_TO_INPUT_FILE --output 
counts
 
 
 $ 
go install github.com/apache/beam/sdks/go/examples/wordcount
+# As part of the initial setup, for non linux users - install package unix 
before run
+$ go get -u golang.org/x/sys/unix
 $ wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
 --output gs://your-gcs-bucket/counts \
 --runner dataflow \
 --project your-gcp-project \
 --temp_location gs://your-gcs-bucket/tmp/ \
+--staging_location gs://your-gcs-bucket/binaries/ \
 
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
 
 
@@ -1009,11 +1012,14 @@ $ debugging_wordcount --input 
PATH_TO_INPUT_FILE --output counts
 
 
 $ 
go install github.com/apache/beam/sdks/go/examples/debugging_wordcount
+# As part of the initial setup, for non linux users - install package unix 
before run
+$ go get -u golang.org/x/sys/unix
 $ debugging_wordcount --input gs://dataflow-samples/shakespeare/kinglear.txt \
   --output gs://your-gcs-bucket/counts \
   --runner dataflow \
   --project your-gcp-project \
   --temp_location gs://your-gcs-bucket/tmp/ \
+  --staging_location 
gs://your-gcs-bucket/binaries/ \
   
--worker_harness_container_image=apache-docker-beam-snapshots-docker.bintray.io/beam/go:20180515
 
 
@@ -1343,11 

[jira] [Work logged] (BEAM-3912) Add batching support for HadoopOutputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3912?focusedWorklogId=148911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148911
 ]

ASF GitHub Bot logged work on BEAM-3912:


Author: ASF GitHub Bot
Created on: 27/Sep/18 21:25
Start Date: 27/Sep/18 21:25
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on a change in pull request 
#6306: [BEAM-3912] Add HadoopOutputFormatIO support
URL: https://github.com/apache/beam/pull/6306#discussion_r221082211
 
 

 ##
 File path: 
sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
 ##
 @@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for additional 
information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 
KIND, either express
+ * or implied. See the License for the specific language governing permissions 
and limitations under
+ * the License.
+ */
+package org.apache.beam.sdk.io.hadoop.format;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.util.concurrent.ExecutionException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.io.hadoop.SerializableConfiguration;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapreduce.JobID;
+import org.apache.hadoop.mapreduce.OutputCommitter;
+import org.apache.hadoop.mapreduce.OutputFormat;
+import org.apache.hadoop.mapreduce.RecordWriter;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.JobContextImpl;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TODO: For the purpose of unification of InputFormat and OutputFormat into 
one common
+ * HadoopFormatIO the code of old HadoopInputFormat should be moved to here 
and HadoopInputFormatIO
+ * becomes deprecated in "hadoop-input-format" module.
+ *
+ * A {@link HadoopFormatIO.Write} is a Transform for writing data to any 
sink which implements
+ * Hadoop {@link OutputFormat}. For example - Cassandra, Elasticsearch, HBase, 
Redis, Postgres etc.
 
 Review comment:
   Nit:
   Suggest removing those that require sorting (e.g. HBase) which this won't 
support - perhaps OrcOutputFormat, CarbonTableOutputFormat, DBOutputFormat are 
better examples?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148911)
Time Spent: 6h 40m  (was: 6.5h)

> Add batching support for HadoopOutputFormatIO
> -
>
> Key: BEAM-3912
> URL: https://issues.apache.org/jira/browse/BEAM-3912
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-hadoop
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 6h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4783) Spark SourceRDD Not Designed With Dynamic Allocation In Mind

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4783?focusedWorklogId=148910=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148910
 ]

ASF GitHub Bot logged work on BEAM-4783:


Author: ASF GitHub Bot
Created on: 27/Sep/18 21:21
Start Date: 27/Sep/18 21:21
Worklog Time Spent: 10m 
  Work Description: kyle-winkelman edited a comment on issue #6181: 
[BEAM-4783] Add bundleSize for splitting BoundedSources.
URL: https://github.com/apache/beam/pull/6181#issuecomment-425201311
 
 
   Looking further into the StreamingTransformTranslator, I would like to pose 
a question. Why do we do the groupByKey followed by the updateStateByKey? It 
appears to be a giant waste in which we convert everything to bytes and back 
unnecessarily.
   
   The only thing it does is gather all the values for a key into an Iterable, 
but the updateStateByKey would also do that if it were given the chance.
   
   If we were to update the UpdateStateByKeyFunction to expect 
WindowedValue's instead of Iterable>'s I believe we could 
eliminate the call to groupByKey. What is happening now is the updateStateByKey 
will wrap those values in a Seq and so currently we have either an empty Seq or 
a Seq with exactly 1 item and that item is itself an Iterable that contains 
multiple items.
   
   **UPDATE: I have created a separate jira to look into this. 
[BEAM-5519](https://jira.apache.org/jira/browse/BEAM-5519)**


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148910)
Time Spent: 3.5h  (was: 3h 20m)

> Spark SourceRDD Not Designed With Dynamic Allocation In Mind
> 
>
> Key: BEAM-4783
> URL: https://issues.apache.org/jira/browse/BEAM-4783
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Affects Versions: 2.5.0
>Reporter: Kyle Winkelman
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Labels: newbie
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> When the spark-runner is used along with the configuration 
> spark.dynamicAllocation.enabled=true the SourceRDD does not detect this. It 
> then falls back to the value calculated in this description:
>   // when running on YARN/SparkDeploy it's the result of max(totalCores, 
> 2).
>   // when running on Mesos it's 8.
>   // when running local it's the total number of cores (local = 1, 
> local[N] = N,
>   // local[*] = estimation of the machine's cores).
>   // ** the configuration "spark.default.parallelism" takes precedence 
> over all of the above **
> So in most cases this default is quite small. This is an issue when using a 
> very large input file as it will only get split in half.
> I believe that when Dynamic Allocation is enable the SourceRDD should use the 
> DEFAULT_BUNDLE_SIZE and possibly expose a SparkPipelineOptions that allows 
> you to change this DEFAULT_BUNDLE_SIZE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3912) Add batching support for HadoopOutputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3912?focusedWorklogId=148909=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148909
 ]

ASF GitHub Bot logged work on BEAM-3912:


Author: ASF GitHub Bot
Created on: 27/Sep/18 21:18
Start Date: 27/Sep/18 21:18
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on a change in pull request 
#6306: [BEAM-3912] Add HadoopOutputFormatIO support
URL: https://github.com/apache/beam/pull/6306#discussion_r221080282
 
 

 ##
 File path: 
sdks/java/io/hadoop-format/src/main/java/org/apache/beam/sdk/io/hadoop/format/HadoopFormatIO.java
 ##
 @@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for additional 
information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 
KIND, either express
+ * or implied. See the License for the specific language governing permissions 
and limitations under
+ * the License.
+ */
+package org.apache.beam.sdk.io.hadoop.format;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.io.IOException;
+import java.lang.reflect.InvocationTargetException;
+import java.util.concurrent.ExecutionException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.io.hadoop.SerializableConfiguration;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.apache.beam.sdk.values.TypeDescriptor;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapreduce.JobID;
+import org.apache.hadoop.mapreduce.OutputCommitter;
+import org.apache.hadoop.mapreduce.OutputFormat;
+import org.apache.hadoop.mapreduce.RecordWriter;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.JobContextImpl;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * TODO: For the purpose of unification of InputFormat and OutputFormat into 
one common
+ * HadoopFormatIO the code of old HadoopInputFormat should be moved to here 
and HadoopInputFormatIO
+ * becomes deprecated in "hadoop-input-format" module.
+ *
+ * A {@link HadoopFormatIO.Write} is a Transform for writing data to any 
sink which implements
+ * Hadoop {@link OutputFormat}. For example - Cassandra, Elasticsearch, HBase, 
Redis, Postgres etc.
+ * {@link HadoopFormatIO.Write} has to make several performance trade-offs in 
connecting to {@link
+ * OutputFormat}, so if there is another Beam IO Transform specifically for 
connecting to your data
+ * sink of choice, we would recommend using that one, but this IO Transform 
allows you to connect to
+ * many data sinks that do not yet have a Beam IO Transform.
+ *
+ * You will need to pass a Hadoop {@link Configuration} with parameters 
specifying how the write
+ * will occur. Many properties of the Configuration are optional, and some are 
required for certain
+ * {@link OutputFormat} classes, but the following properties must be set for 
all OutputFormats:
+ *
+ * 
+ *   {@code mapreduce.job.outputformat.class}: The {@link OutputFormat} 
class used to connect to
+ *   your data sink of choice.
+ *   {@code mapreduce.job.outputformat.key.class}: The key class passed to 
the {@link
+ *   OutputFormat} in {@code mapreduce.job.outputformat.class}.
+ *   {@code mapreduce.job.outputformat.value.class}: The value class 
passed to the {@link
+ *   OutputFormat} in {@code mapreduce.job.outputformat.class}.
+ * 
+ *
+ * For example:
+ *
+ * {@code
+ * Configuration myHadoopConfiguration = new Configuration(false);
+ * // Set Hadoop OutputFormat, key and value class in configuration
+ * myHadoopConfiguration.setClass(mapreduce.job.outputformat.class,
+ *MyDbOutputFormatClass, OutputFormat.class);
+ * 
myHadoopConfiguration.setClass(mapreduce.job.outputformat.key.class,
+ *MyDbOutputFormatKeyClass, Object.class);
+ * 

[jira] [Commented] (BEAM-5467) Python Flink ValidatesRunner job fixes

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631049#comment-16631049
 ] 

Ankur Goenka commented on BEAM-5467:


I verified that they get executed sequentially so that should not be a problem. 
 
:beam-sdks-python:flinkCompatibilityMatrixBatchFAILED
Started: 5m 27.699s
Duration: 4m 0.393s
 
:beam-sdks-python:flinkCompatibilityMatrixStreamingFAILED
Started: 9m 28.093s
Duration: 1m 13.280s

> Python Flink ValidatesRunner job fixes
> --
>
> Key: BEAM-5467
> URL: https://issues.apache.org/jira/browse/BEAM-5467
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Add status to README
> Rename script and job for consistency
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5519) Spark Streaming Duplicated Encoding/Decoding Effort

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5519?focusedWorklogId=148908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148908
 ]

ASF GitHub Bot logged work on BEAM-5519:


Author: ASF GitHub Bot
Created on: 27/Sep/18 21:07
Start Date: 27/Sep/18 21:07
Worklog Time Spent: 10m 
  Work Description: kyle-winkelman commented on issue #6511: [BEAM-5519] 
Remove call to groupByKey in Spark Streaming.
URL: https://github.com/apache/beam/pull/6511#issuecomment-425243571
 
 
   I am not extremely confident in this code I just wanted to put something 
together to demonstrate my point. I also think there is some additional 
refactoring required such as renaming 
SparkGroupAlsoByWindowViaWindowSet.groupAlsoByWindow, but I would like to get 
some feedback before continuing.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148908)
Time Spent: 20m  (was: 10m)

> Spark Streaming Duplicated Encoding/Decoding Effort
> ---
>
> Key: BEAM-5519
> URL: https://issues.apache.org/jira/browse/BEAM-5519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Kyle Winkelman
>Assignee: Kyle Winkelman
>Priority: Major
>  Labels: spark, spark-streaming
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When using the SparkRunner in streaming mode. There is a call to groupByKey 
> followed by a call to updateStateByKey. BEAM-1815 fixed an issue where this 
> used to cause 2 shuffles but it still causes 2 encode/decode cycles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5519) Spark Streaming Duplicated Encoding/Decoding Effort

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5519?focusedWorklogId=148902=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148902
 ]

ASF GitHub Bot logged work on BEAM-5519:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:55
Start Date: 27/Sep/18 20:55
Worklog Time Spent: 10m 
  Work Description: kyle-winkelman opened a new pull request #6511: 
[BEAM-5519] Remove call to groupByKey in Spark Streaming.
URL: https://github.com/apache/beam/pull/6511
 
 
   Prevent the duplication of encode/decode cycle in wasted call to groupByKey 
in spark streaming. It is a waste because after the groupByKey is a call to 
updateStateByKey which will ensure that the data is shuffled to the correct 
processing location.
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148902)
Time Spent: 10m
Remaining Estimate: 0h

> Spark Streaming Duplicated Encoding/Decoding Effort
> ---
>
> 

[jira] [Work logged] (BEAM-3912) Add batching support for HadoopOutputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3912?focusedWorklogId=148898=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148898
 ]

ASF GitHub Bot logged work on BEAM-3912:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:48
Start Date: 27/Sep/18 20:48
Worklog Time Spent: 10m 
  Work Description: timrobertson100 removed a comment on issue #6306: 
[BEAM-3912] Add HadoopOutputFormatIO support
URL: https://github.com/apache/beam/pull/6306#issuecomment-425236912
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148898)
Time Spent: 6h 20m  (was: 6h 10m)

> Add batching support for HadoopOutputFormatIO
> -
>
> Key: BEAM-3912
> URL: https://issues.apache.org/jira/browse/BEAM-3912
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-hadoop
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3912) Add batching support for HadoopOutputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3912?focusedWorklogId=148897=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148897
 ]

ASF GitHub Bot logged work on BEAM-3912:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:48
Start Date: 27/Sep/18 20:48
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #6306: [BEAM-3912] 
Add HadoopOutputFormatIO support
URL: https://github.com/apache/beam/pull/6306#issuecomment-425237668
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148897)
Time Spent: 6h 10m  (was: 6h)

> Add batching support for HadoopOutputFormatIO
> -
>
> Key: BEAM-3912
> URL: https://issues.apache.org/jira/browse/BEAM-3912
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-hadoop
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3912) Add batching support for HadoopOutputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3912?focusedWorklogId=148895=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148895
 ]

ASF GitHub Bot logged work on BEAM-3912:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:47
Start Date: 27/Sep/18 20:47
Worklog Time Spent: 10m 
  Work Description: timrobertson100 edited a comment on issue #6306: 
[BEAM-3912] Add HadoopOutputFormatIO support
URL: https://github.com/apache/beam/pull/6306#issuecomment-425236912
 
 
   Retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148895)
Time Spent: 5h 50m  (was: 5h 40m)

> Add batching support for HadoopOutputFormatIO
> -
>
> Key: BEAM-3912
> URL: https://issues.apache.org/jira/browse/BEAM-3912
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-hadoop
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3912) Add batching support for HadoopOutputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3912?focusedWorklogId=148896=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148896
 ]

ASF GitHub Bot logged work on BEAM-3912:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:47
Start Date: 27/Sep/18 20:47
Worklog Time Spent: 10m 
  Work Description: timrobertson100 edited a comment on issue #6306: 
[BEAM-3912] Add HadoopOutputFormatIO support
URL: https://github.com/apache/beam/pull/6306#issuecomment-425236912
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148896)
Time Spent: 6h  (was: 5h 50m)

> Add batching support for HadoopOutputFormatIO
> -
>
> Key: BEAM-3912
> URL: https://issues.apache.org/jira/browse/BEAM-3912
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-hadoop
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 6h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3912) Add batching support for HadoopOutputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3912?focusedWorklogId=148894=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148894
 ]

ASF GitHub Bot logged work on BEAM-3912:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:46
Start Date: 27/Sep/18 20:46
Worklog Time Spent: 10m 
  Work Description: timrobertson100 commented on issue #6306: [BEAM-3912] 
Add HadoopOutputFormatIO support
URL: https://github.com/apache/beam/pull/6306#issuecomment-425236912
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148894)
Time Spent: 5h 40m  (was: 5.5h)

> Add batching support for HadoopOutputFormatIO
> -
>
> Key: BEAM-3912
> URL: https://issues.apache.org/jira/browse/BEAM-3912
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-hadoop
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5519) Spark Streaming Duplicated Encoding/Decoding Effort

[Kyle Winkelman (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631017#comment-16631017
 ] 

Kyle Winkelman commented on BEAM-5519:
--

Proposed:

// SparkGroupAlsoByWindowViaWindowSet.buildPairDStream
JavaRDD>>
JavaRDD>>>
JavaRDD>>
JavaPairRDD

// UpdateStateByKeyOutputIterator.computeNext
gets the scala.collection.Seq the seq of values that have the same key
decoded to scala.collection.Seq> (convert to Iterable)


> Spark Streaming Duplicated Encoding/Decoding Effort
> ---
>
> Key: BEAM-5519
> URL: https://issues.apache.org/jira/browse/BEAM-5519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Kyle Winkelman
>Assignee: Kyle Winkelman
>Priority: Major
>  Labels: spark, spark-streaming
>
> When using the SparkRunner in streaming mode. There is a call to groupByKey 
> followed by a call to updateStateByKey. BEAM-1815 fixed an issue where this 
> used to cause 2 shuffles but it still causes 2 encode/decode cycles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5519) Spark Streaming Duplicated Encoding/Decoding Effort

[Kyle Winkelman (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631012#comment-16631012
 ] 

Kyle Winkelman commented on BEAM-5519:
--

Current:
// GroupCombineFunctions.groupByKeyOnly
JavaRDD>>
JavaRDD>>>
JavaRDD>>
JavaPairRDD>
JavaPairRDD
JavaPairRdd> // groupByKey
JavaPairRDD>>
JavaRDD>>>
JavaRDD

// SparkGroupAlsoByWindowViaWindowSet.buildPairDStream
JavaRDD>>>
JavaPairRDD>>
JavaPairRDD>>>
JavaPairRDD

// UpdateStateByKeyOutputIterator.computeNext
gets the scala.collection.Seq the seq of values that have the same key
decoded to scala.collection.Seq>>> (zero or 
one items because we have already grouped by key)
get the head of the Seq and pull out the Iterable>



> Spark Streaming Duplicated Encoding/Decoding Effort
> ---
>
> Key: BEAM-5519
> URL: https://issues.apache.org/jira/browse/BEAM-5519
> Project: Beam
>  Issue Type: Bug
>  Components: runner-spark
>Reporter: Kyle Winkelman
>Assignee: Kyle Winkelman
>Priority: Major
>  Labels: spark, spark-streaming
>
> When using the SparkRunner in streaming mode. There is a call to groupByKey 
> followed by a call to updateStateByKey. BEAM-1815 fixed an issue where this 
> used to cause 2 shuffles but it still causes 2 encode/decode cycles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5467) Python Flink ValidatesRunner job fixes

[Thomas Weise (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631010#comment-16631010
 ] 

Thomas Weise commented on BEAM-5467:


[~angoenka] should we try to turn off the parallel execution?

I also think we should move the following to a distinct task in 
sdks/python/build.gradle:
{code:java}
tasks(':beam-sdks-python:flinkCompatibilityMatrixBatch')
tasks(':beam-sdks-python:flinkCompatibilityMatrixStreaming'){code}
 

 

> Python Flink ValidatesRunner job fixes
> --
>
> Key: BEAM-5467
> URL: https://issues.apache.org/jira/browse/BEAM-5467
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Add status to README
> Rename script and job for consistency
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5519) Spark Streaming Duplicated Encoding/Decoding Effort

[Kyle Winkelman (JIRA)]

Kyle Winkelman created BEAM-5519:


 Summary: Spark Streaming Duplicated Encoding/Decoding Effort
 Key: BEAM-5519
 URL: https://issues.apache.org/jira/browse/BEAM-5519
 Project: Beam
  Issue Type: Bug
  Components: runner-spark
Reporter: Kyle Winkelman
Assignee: Kyle Winkelman


When using the SparkRunner in streaming mode. There is a call to groupByKey 
followed by a call to updateStateByKey. BEAM-1815 fixed an issue where this 
used to cause 2 shuffles but it still causes 2 encode/decode cycles.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5467) Python Flink ValidatesRunner job fixes

[Ankur Goenka (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631002#comment-16631002
 ] 

Ankur Goenka commented on BEAM-5467:


Anecdotally, tasks are failing because of segfault with following error 
04:18:38 Segmentation fault (core dumped) 04:18:38 04:18:38 > Task 
:beam-sdks-python:flinkCompatibilityMatrixStreaming FAILED 04:18:38 
:beam-sdks-python:flinkCompatibilityMatrixStreaming (Thread[Task worker for ':' 
Thread 6,5,main]) completed. Took 1 mins 13.28 secs. 04:18:38 04:18:38 FAILURE: 
Build completed with 2 failures. 04:18:38 04:18:38 1: Task failed with an 
exception. 04:18:38 --- 04:18:38 * Where: 04:18:38 Build file 
'/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_VR_Flink/src/sdks/python/build.gradle'
 line: 340 04:18:38 04:18:38 * What went wrong: 04:18:38 Execution failed for 
task ':beam-sdks-python:flinkCompatibilityMatrixBatch'. 04:18:38 > Process 
'command 'sh'' finished with non-zero exit value 139 04:18:38 04:18:38 * Try: 
04:18:38 Run with --stacktrace option to get the stack trace. Run with --debug 
option to get more log output. Run with --scan to get full insights. 04:18:38 
==

> Python Flink ValidatesRunner job fixes
> --
>
> Key: BEAM-5467
> URL: https://issues.apache.org/jira/browse/BEAM-5467
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Add status to README
> Rename script and job for consistency
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148890=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148890
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:29
Start Date: 27/Sep/18 20:29
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #566: [BEAM-5506] add 
whitespace to redirection in create_external_table
URL: https://github.com/apache/beam-site/pull/566
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/documentation/dsls/sql/create-external-table.md 
b/src/documentation/dsls/sql/create-external-table.md
index 57c03f1c21..354ebf9863 100644
--- a/src/documentation/dsls/sql/create-external-table.md
+++ b/src/documentation/dsls/sql/create-external-table.md
@@ -4,8 +4,8 @@ title: "Beam SQL: CREATE EXTERNAL TABLE Statement"
 section_menu: section-menu/sdks.html
 permalink: /documentation/dsls/sql/create-external-table/
 redirect_from:
-  -/documentation/dsls/sql/statements/create-table/
-  -/documentation/dsls/sql/create-table/
+  - /documentation/dsls/sql/statements/create-table/
+  - /documentation/dsls/sql/create-table/
 ---
 
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_GradleBuild #1565

[Apache Jenkins Server]

See 

[beam-site] 01/03: add whitespace to redirection in create_external_table

[melap]

This is an automated email from the ASF dual-hosted git repository.

melap pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 6d814e9306856d5df1e08d1b47f3881106c60b45
Author: amaliujia 
AuthorDate: Thu Sep 27 13:09:43 2018 -0700

add whitespace to redirection in create_external_table
---
 src/documentation/dsls/sql/create-external-table.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/documentation/dsls/sql/create-external-table.md 
b/src/documentation/dsls/sql/create-external-table.md
index 57c03f1..354ebf9 100644
--- a/src/documentation/dsls/sql/create-external-table.md
+++ b/src/documentation/dsls/sql/create-external-table.md
@@ -4,8 +4,8 @@ title: "Beam SQL: CREATE EXTERNAL TABLE Statement"
 section_menu: section-menu/sdks.html
 permalink: /documentation/dsls/sql/create-external-table/
 redirect_from:
-  -/documentation/dsls/sql/statements/create-table/
-  -/documentation/dsls/sql/create-table/
+  - /documentation/dsls/sql/statements/create-table/
+  - /documentation/dsls/sql/create-table/
 ---
 

[beam-site] branch asf-site updated (d52bf3e -> ac4f6be)

[melap]

This is an automated email from the ASF dual-hosted git repository.

melap pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from d52bf3e  This closes #562
 new 6d814e9  add whitespace to redirection in create_external_table
 new b7e59f5  Regenerate website
 new ac4f6be  This closes #566

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../ - => }/documentation/dsls/sql/create-table/index.html| 0
 .../sql => documentation/dsls/sql/statements}/create-table/index.html | 0
 src/documentation/dsls/sql/create-external-table.md   | 4 ++--
 3 files changed, 2 insertions(+), 2 deletions(-)
 copy content/{-/documentation/dsls/sql/statements/create-table/ - => 
}/documentation/dsls/sql/create-table/index.html (100%)
 rename content/{-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql => 
documentation/dsls/sql/statements}/create-table/index.html (100%)

[beam-site] 03/03: This closes #566

[melap]

This is an automated email from the ASF dual-hosted git repository.

melap pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit ac4f6be7ad8a9e943d03101f0f654c82af62d542
Merge: d52bf3e b7e59f5
Author: Melissa Pashniak 
AuthorDate: Thu Sep 27 13:27:22 2018 -0700

This closes #566

 .../ - => }/documentation/dsls/sql/create-table/index.html| 0
 .../sql => documentation/dsls/sql/statements}/create-table/index.html | 0
 src/documentation/dsls/sql/create-external-table.md   | 4 ++--
 3 files changed, 2 insertions(+), 2 deletions(-)

[beam-site] 02/03: Regenerate website

[melap]

This is an automated email from the ASF dual-hosted git repository.

melap pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit b7e59f5f77cc9dd805b05a9ab2dd538bdc417d55
Author: Melissa Pashniak 
AuthorDate: Thu Sep 27 13:27:22 2018 -0700

Regenerate website
---
 .../create-table/ - => }/documentation/dsls/sql/create-table/index.html   | 0
 .../sql => documentation/dsls/sql/statements}/create-table/index.html | 0
 2 files changed, 0 insertions(+), 0 deletions(-)

diff --git a/content/-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql/create-table/index.html 
b/content/documentation/dsls/sql/create-table/index.html
similarity index 100%
copy from content/-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql/create-table/index.html
copy to content/documentation/dsls/sql/create-table/index.html
diff --git a/content/-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql/create-table/index.html 
b/content/documentation/dsls/sql/statements/create-table/index.html
similarity index 100%
rename from content/-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql/create-table/index.html
rename to content/documentation/dsls/sql/statements/create-table/index.html

[jira] [Assigned] (BEAM-5511) Improve documentation when using AvroIO dynamic destination with user define objects

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov reassigned BEAM-5511:
--

Assignee: (was: Eugene Kirpichov)

> Improve documentation when using AvroIO dynamic destination with user define 
> objects
> 
>
> Key: BEAM-5511
> URL: https://issues.apache.org/jira/browse/BEAM-5511
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Affects Versions: 2.4.0, 2.5.0, 2.6.0
>Reporter: JC
>Priority: Trivial
>
> As described here: 
> [https://mail-archives.apache.org/mod_mbox/beam-user/201809.mbox/%3CCAMsy_NjC_95t+v=30kc9hg1gazhftfv-9yhmtaco85e+psu...@mail.gmail.com%3E]
> Title: AvroIO - failure using direct runner with 
> java.nio.file.FileAlreadyExistsException when moving from temp to destination
> Please improve the *org.apache.beam.sdk.io.DynamicAvroDestinations* JavaDoc 
> to state that when using a user define object as  hashCode and 
> equals needs to be implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5511) Improve documentation when using AvroIO dynamic destination with user define objects

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630998#comment-16630998
 ] 

Eugene Kirpichov commented on BEAM-5511:


This should equally apply to documentation of FileIO.write(). However I'm no 
longer full-time on Beam so someone else should feel free to pick this up.

> Improve documentation when using AvroIO dynamic destination with user define 
> objects
> 
>
> Key: BEAM-5511
> URL: https://issues.apache.org/jira/browse/BEAM-5511
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-avro
>Affects Versions: 2.4.0, 2.5.0, 2.6.0
>Reporter: JC
>Assignee: Eugene Kirpichov
>Priority: Trivial
>
> As described here: 
> [https://mail-archives.apache.org/mod_mbox/beam-user/201809.mbox/%3CCAMsy_NjC_95t+v=30kc9hg1gazhftfv-9yhmtaco85e+psu...@mail.gmail.com%3E]
> Title: AvroIO - failure using direct runner with 
> java.nio.file.FileAlreadyExistsException when moving from temp to destination
> Please improve the *org.apache.beam.sdk.io.DynamicAvroDestinations* JavaDoc 
> to state that when using a user define object as  hashCode and 
> equals needs to be implemented.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148882
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:22
Start Date: 27/Sep/18 20:22
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #566: [BEAM-5506] add 
whitespace to redirection in create_external_table
URL: https://github.com/apache/beam-site/pull/566#issuecomment-425230018
 
 
   @melap 
   
   I think this fix works. I can see the redirection behavior from 
`create-table` to `create-external-table` in the generated url.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148882)
Time Spent: 2h  (was: 1h 50m)

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (BEAM-5506) Update Beam documentation

[Rui Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Wang closed BEAM-5506.
--
   Resolution: Fixed
Fix Version/s: Not applicable

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Website_Publish #21

[Apache Jenkins Server]

See 


Changes:

[kevinsi] Randomize the reduced splits in BigtableIO so that multiple workers 
may

--
[...truncated 8.91 KB...]
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Daemon worker,5,main]) completed. Took 1.425 
secs.
:buildSrc:spotlessGroovyCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Daemon worker,5,main]) completed. Took 
0.0 secs.
:buildSrc:spotlessGroovyGradle (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Daemon worker,5,main]) completed. Took 
0.025 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Daemon worker,5,main]) completed. 
Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 0.001 
secs.
:buildSrc:compileTestGroovy (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Daemon worker,5,main]) completed. Took 
0.001 secs.
:buildSrc:processTestResources (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Daemon worker,5,main]) completed. Took 
0.001 secs.
:buildSrc:testClasses (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:test (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Daemon worker,5,main]) completed. Took 0.002 secs.
:buildSrc:check (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:buildSrc:check (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:build (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:build
Skipping task ':buildSrc:build' as it has no actions.
:buildSrc:build (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root project using build file 

[jira] [Work logged] (BEAM-5272) Randomize the reduced splits in BigtableIO so that multiple workers may not hit the same tablet server

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5272?focusedWorklogId=148879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148879
 ]

ASF GitHub Bot logged work on BEAM-5272:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:17
Start Date: 27/Sep/18 20:17
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #6503: [BEAM-5272] Randomize 
the reduced splits in BigtableIO so that multiple workers may not hit the same 
tablet server
URL: https://github.com/apache/beam/pull/6503
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index edad185323c..755d889b491 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -848,18 +848,25 @@ protected BigtableSource withEstimatedSizeBytes(Long 
estimatedSizeBytes) {
   // Delegate to testable helper.
   List splits =
   splitBasedOnSamples(desiredBundleSizeBytes, 
getSampleRowKeys(options));
-  return reduceSplits(splits, options, MAX_SPLIT_COUNT);
+
+  // Reduce the splits.
+  List reduced = reduceSplits(splits, options, 
MAX_SPLIT_COUNT);
+  // Randomize the result before returning an immutable copy of the 
splits, the default behavior
+  // may lead to multiple workers hitting the same tablet.
+  Collections.shuffle(reduced);
+  return ImmutableList.copyOf(reduced);
 }
 
+/** Returns a mutable list of reduced splits. */
 @VisibleForTesting
 protected List reduceSplits(
 List splits, PipelineOptions options, long 
maxSplitCounts)
 throws IOException {
   int numberToCombine = (int) ((splits.size() + maxSplitCounts - 1) / 
maxSplitCounts);
   if (splits.size() < maxSplitCounts || numberToCombine < 2) {
-return splits;
+return new ArrayList<>(splits);
   }
-  ImmutableList.Builder reducedSplits = 
ImmutableList.builder();
+  List reducedSplits = new ArrayList<>();
   List previousSourceRanges = new ArrayList();
   int counter = 0;
   long size = 0;
@@ -879,7 +886,7 @@ protected BigtableSource withEstimatedSizeBytes(Long 
estimatedSizeBytes) {
   if (size > 0) {
 reducedSplits.add(new BigtableSource(config, filter, 
previousSourceRanges, size));
   }
-  return reducedSplits.build();
+  return reducedSplits;
 }
 
 /**
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
index cadb908be5a..54a2fee99b0 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
@@ -102,7 +102,7 @@
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TypeDescriptor;
 import org.hamcrest.Matchers;
-import org.hamcrest.collection.IsIterableContainingInOrder;
+import org.hamcrest.collection.IsIterableContainingInAnyOrder;
 import org.junit.Before;
 import org.junit.Rule;
 import org.junit.Test;
@@ -736,10 +736,10 @@ public void testReduceSplitsWithSomeNonAdjacentRanges() 
throws Exception {
 keyRanges,
 null /*size*/);
 
-List splits =
-source.split(numRows * bytesPerRow / numSamples, null /* options */);
-
-assertThat(splits, hasSize(keyRanges.size()));
+List splits = new ArrayList<>();
+for (ByteKeyRange range : keyRanges) {
+  splits.add(source.withSingleRange(range));
+}
 
 List reducedSplits = source.reduceSplits(splits, null, 
maxSplit);
 
@@ -753,7 +753,8 @@ public void testReduceSplitsWithSomeNonAdjacentRanges() 
throws Exception {
 
 assertThat(
 actualRangesAfterSplit,
-
IsIterableContainingInOrder.contains(expectedKeyRangesAfterReducedSplits.toArray()));
+IsIterableContainingInAnyOrder.containsInAnyOrder(
+expectedKeyRangesAfterReducedSplits.toArray()));
   }
 
   /** Tests reduce split with all non adjacent ranges. */
@@ -786,10 +787,10 @@ public void testReduceSplitsWithAllNonAdjacentRange() 
throws Exception {
 keyRanges,
 null /*size*/);
 
-List splits =
-source.split(numRows * 

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148880
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:17
Start Date: 27/Sep/18 20:17
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #566: [BEAM-5506] add 
whitespace to redirection in create_external_table
URL: https://github.com/apache/beam-site/pull/566#issuecomment-425228551
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148880)
Time Spent: 1h 50m  (was: 1h 40m)

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (2bee441 -> 97bf456)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2bee441  Merge pull request #6501 from kevinsi4508/remove_displaydata
 add 2158c50  Randomize the reduced splits in BigtableIO so that multiple 
workers may not hit the same tablet server
 add 97bf456  Merge pull request #6503 from kevinsi4508/update_bigtable_io2

No new revisions were added by this update.

Summary of changes:
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   | 15 +--
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   | 50 ++
 2 files changed, 44 insertions(+), 21 deletions(-)

Build failed in Jenkins: beam_PostCommit_Website_Publish #20

[Apache Jenkins Server]

See 


Changes:

[kevinsi] When getting display data from a runtime parameter, don't call get().

--
[...truncated 8.57 KB...]

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 1.374 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.025 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for 
':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for 
':buildSrc',5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.0 secs.
:buildSrc:test (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.002 secs.
:buildSrc:check (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:buildSrc:check (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:build (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:build
Skipping task ':buildSrc:build' as it has no actions.
:buildSrc:build (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 

[jira] [Work logged] (BEAM-5420) BigtableIO tries to get runtime parameters when collecting display data at pipeline construction time

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5420?focusedWorklogId=148878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148878
 ]

ASF GitHub Bot logged work on BEAM-5420:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:15
Start Date: 27/Sep/18 20:15
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #6501: [BEAM-5420] When 
getting display data from a runtime parameter, don't call get()
URL: https://github.com/apache/beam/pull/6501
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
index edad185323c..45394cfdcc7 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java
@@ -1095,7 +1095,7 @@ public void validate() {
 public void populateDisplayData(DisplayData.Builder builder) {
   super.populateDisplayData(builder);
 
-  builder.add(DisplayData.item("tableId", 
config.getTableId().get()).withLabel("Table ID"));
+  builder.add(DisplayData.item("tableId", 
config.getTableId()).withLabel("Table ID"));
 
   if (filter != null) {
 builder.add(DisplayData.item("rowFilter", 
filter.toString()).withLabel("Table Row Filter"));
diff --git 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
index cadb908be5a..8fa98924f4a 100644
--- 
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
+++ 
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java
@@ -1060,6 +1060,39 @@ public void testReadingPrimitiveDisplayData() throws 
IOException, InterruptedExc
 Matchers.hasItem(hasDisplayItem("rowFilter")));
   }
 
+  @Test
+  public void testReadingDisplayDataFromRuntimeParameters() {
+ReadOptions options = 
PipelineOptionsFactory.fromArgs().withValidation().as(ReadOptions.class);
+BigtableIO.Read read =
+BigtableIO.read()
+.withBigtableOptions(BIGTABLE_OPTIONS)
+.withProjectId(options.getBigtableProject())
+.withInstanceId(options.getBigtableInstanceId())
+.withTableId(options.getBigtableTableId());
+DisplayData displayData = DisplayData.from(read);
+assertThat(
+displayData,
+hasDisplayItem(
+allOf(
+hasKey("projectId"),
+hasLabel("Bigtable Project Id"),
+hasValue("RuntimeValueProvider{propertyName=bigtableProject, 
default=null}";
+assertThat(
+displayData,
+hasDisplayItem(
+allOf(
+hasKey("instanceId"),
+hasLabel("Bigtable Instance Id"),
+
hasValue("RuntimeValueProvider{propertyName=bigtableInstanceId, 
default=null}";
+assertThat(
+displayData,
+hasDisplayItem(
+allOf(
+hasKey("tableId"),
+hasLabel("Bigtable Table Id"),
+hasValue("RuntimeValueProvider{propertyName=bigtableTableId, 
default=null}";
+  }
+
   @Test
   public void testReadWithoutValidate() {
 final String table = "fooTable";


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148878)
Time Spent: 40m  (was: 0.5h)

> BigtableIO tries to get runtime parameters when collecting display data at 
> pipeline construction time
> -
>
> Key: BEAM-5420
> URL: https://issues.apache.org/jira/browse/BEAM-5420
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Reporter: Kevin Si
>Assignee: Chamikara Jayalath
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> For example: 
> 

[beam] branch master updated (4a97ed7 -> 2bee441)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4a97ed7  Merge pull request #6507 from pabloem/new-v1b3
 add 0334933  When getting display data from a runtime parameter, don't 
call get().
 add 2bee441  Merge pull request #6501 from kevinsi4508/remove_displaydata

No new revisions were added by this update.

Summary of changes:
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java   |  2 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIOTest.java   | 33 ++
 2 files changed, 34 insertions(+), 1 deletion(-)

Build failed in Jenkins: beam_PostCommit_Website_Publish #19

[Apache Jenkins Server]

See 


Changes:

[pablo] Updating Dataflow API protocol buffers

--
[...truncated 8.24 KB...]
:buildSrc:spotlessGroovy (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Daemon worker,5,main]) completed. Took 1.512 
secs.
:buildSrc:spotlessGroovyCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Daemon worker,5,main]) completed. Took 
0.0 secs.
:buildSrc:spotlessGroovyGradle (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Daemon worker,5,main]) completed. Took 
0.041 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Daemon worker,5,main]) completed. 
Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Daemon worker,5,main]) completed. Took 0.002 
secs.
:buildSrc:compileTestGroovy (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Daemon worker,5,main]) completed. Took 
0.002 secs.
:buildSrc:processTestResources (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Daemon worker,5,main]) completed. Took 
0.001 secs.
:buildSrc:testClasses (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:test (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Daemon worker,5,main]) completed. Took 0.003 secs.
:buildSrc:check (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:buildSrc:check (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:build (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:build
Skipping task ':buildSrc:build' as it has no actions.
:buildSrc:build (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
Settings evaluated using settings file 
'
Using local directory build cache for the root build (location = 
/home/jenkins/.gradle/caches/build-cache-1, removeUnusedEntriesAfter = 7 days).
Projects loaded. Root 

[beam] 01/01: Merge pull request #6507 from pabloem/new-v1b3

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 4a97ed72259b5f16e3a1b178566bec94fc70a698
Merge: 7bc8bcc caee861
Author: Ahmet Altay 
AuthorDate: Thu Sep 27 13:12:13 2018 -0700

Merge pull request #6507 from pabloem/new-v1b3

Updating Dataflow API protocol buffers

 .../clients/dataflow/dataflow_v1b3_client.py   | 761 ++---
 .../clients/dataflow/dataflow_v1b3_messages.py | 253 +--
 2 files changed, 584 insertions(+), 430 deletions(-)

[beam] branch master updated (7bc8bcc -> 4a97ed7)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7bc8bcc  Merge pull request #6491 from 
amaliujia/rui_wang-update_create_table_comment
 add caee861  Updating Dataflow API protocol buffers
 new 4a97ed7  Merge pull request #6507 from pabloem/new-v1b3

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../clients/dataflow/dataflow_v1b3_client.py   | 761 ++---
 .../clients/dataflow/dataflow_v1b3_messages.py | 253 +--
 2 files changed, 584 insertions(+), 430 deletions(-)

[jira] [Work logged] (BEAM-3912) Add batching support for HadoopOutputFormatIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3912?focusedWorklogId=148877=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148877
 ]

ASF GitHub Bot logged work on BEAM-3912:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:11
Start Date: 27/Sep/18 20:11
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #6306: [BEAM-3912] Add 
HadoopOutputFormatIO support
URL: https://github.com/apache/beam/pull/6306#issuecomment-425226876
 
 
   What is the status of this PR? @echauchot would you be able to review it?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148877)
Time Spent: 5.5h  (was: 5h 20m)

> Add batching support for HadoopOutputFormatIO
> -
>
> Key: BEAM-3912
> URL: https://issues.apache.org/jira/browse/BEAM-3912
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-hadoop
>Reporter: Alexey Romanenko
>Assignee: Alexey Romanenko
>Priority: Minor
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5467) Python Flink ValidatesRunner job fixes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5467?focusedWorklogId=148876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148876
 ]

ASF GitHub Bot logged work on BEAM-5467:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:10
Start Date: 27/Sep/18 20:10
Worklog Time Spent: 10m 
  Work Description: angoenka closed pull request #6510: [BEAM-5467] Use 
dynamic port while running Flink PVR
URL: https://github.com/apache/beam/pull/6510
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/runners/flink/job-server/build.gradle 
b/runners/flink/job-server/build.gradle
index 541ede96916..b7dc0286ac1 100644
--- a/runners/flink/job-server/build.gradle
+++ b/runners/flink/job-server/build.gradle
@@ -74,6 +74,6 @@ runShadow {
 
 createPortableValidatesRunnerTask(
 jobServerDriver: "org.apache.beam.runners.flink.FlinkJobServerDriver",
-jobServerConfig: "--clean-artifacts-per-job,--job-host=localhost",
+jobServerConfig: 
"--clean-artifacts-per-job,--job-host=localhost,--job-port=0,--artifact-port=0",
 testClasspathConfiguration: configurations.validatesPortableRunner
 )


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148876)
Time Spent: 1h 20m  (was: 1h 10m)

> Python Flink ValidatesRunner job fixes
> --
>
> Key: BEAM-5467
> URL: https://issues.apache.org/jira/browse/BEAM-5467
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Add status to README
> Rename script and job for consistency
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Website_Publish #18

[Apache Jenkins Server]

See 


Changes:

[amaliujia] [BEAM-5506] Add reference link.

--
[...truncated 8.97 KB...]
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 1.447 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.024 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 0.001 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.001 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
3,5,main]) completed. Took 0.001 secs.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 0.0 secs.
:buildSrc:test (Thread[Task worker for ':buildSrc' Thread 3,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Task worker for ':buildSrc' Thread 3,5,main]) completed. 
Took 0.004 secs.
:buildSrc:check (Thread[Task worker for ':buildSrc' Thread 3,5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:buildSrc:check (Thread[Task worker for ':buildSrc' Thread 3,5,main]) 
completed. Took 0.0 secs.
:buildSrc:build (Thread[Task worker for ':buildSrc' Thread 3,5,main]) started.

> Task :buildSrc:build
Skipping 

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148874=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148874
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:05
Start Date: 27/Sep/18 20:05
Worklog Time Spent: 10m 
  Work Description: akedin closed pull request #6491: [BEAM-5506] Add 
reference link in CREATE TABLE exception
URL: https://github.com/apache/beam/pull/6491
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl 
b/sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl
index 0f9b2d1db5e..1fd6efda54f 100644
--- a/sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl
+++ b/sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl
@@ -197,8 +197,9 @@ SqlCreate SqlCreateTableNotSupportedMessage(Span s, boolean 
replace) :
 {
   
   {
-throw new ParseException("'CREATE TABLE' is not supported in BeamSQL. You 
can use "
-+ "'CREATE EXTERNAL TABLE' to register an external data source to 
BeamSQL");
+throw new ParseException("'CREATE TABLE' is not supported in SQL. You can 
use "
++ "'CREATE EXTERNAL TABLE' to register an external data source to SQL. For 
more details, "
++ "please check: 
https://beam.apache.org/documentation/dsls/sql/create-external-table;);
   }
 }
 


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148874)
Time Spent: 1h 40m  (was: 1.5h)

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #6491 from amaliujia/rui_wang-update_create_table_comment

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7bc8bcca9a05c196070ced804badc6ecbd3dc092
Merge: fc38698 1e60dd2
Author: Anton Kedin <33067037+ake...@users.noreply.github.com>
AuthorDate: Thu Sep 27 16:05:32 2018 -0400

Merge pull request #6491 from amaliujia/rui_wang-update_create_table_comment

[BEAM-5506] Add reference link in CREATE TABLE exception

 sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

[beam] branch master updated (fc38698 -> 7bc8bcc)

[anton]

This is an automated email from the ASF dual-hosted git repository.

anton pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from fc38698  Go SDK avroio Package - Read/Write Avro Files
 add 1e60dd2  [BEAM-5506] Add reference link.
 new 7bc8bcc  Merge pull request #6491 from 
amaliujia/rui_wang-update_create_table_comment

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/extensions/sql/src/main/codegen/includes/parserImpls.ftl | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148873
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:01
Start Date: 27/Sep/18 20:01
Worklog Time Spent: 10m 
  Work Description: amaliujia edited a comment on issue #6491: [BEAM-5506] 
Add reference link in CREATE TABLE exception
URL: https://github.com/apache/beam/pull/6491#issuecomment-425224044
 
 
   https://github.com/apache/beam-site/pull/562 has been merged and the doc 
change has been applied to Beam website.
   
   See: https://beam.apache.org/documentation/dsls/sql/create-external-table


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148873)
Time Spent: 1.5h  (was: 1h 20m)

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148871
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:01
Start Date: 27/Sep/18 20:01
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6491: [BEAM-5506] Add 
reference link in CREATE TABLE exception
URL: https://github.com/apache/beam/pull/6491#issuecomment-425223860
 
 
   Thanks @akedin!
   
   cc: @kennknowles @apilloud 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148871)
Time Spent: 1h 10m  (was: 1h)

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148872=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148872
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:01
Start Date: 27/Sep/18 20:01
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #6491: [BEAM-5506] Add 
reference link in CREATE TABLE exception
URL: https://github.com/apache/beam/pull/6491#issuecomment-425224044
 
 
   https://github.com/apache/beam-site/pull/562 has been merged and the doc 
change has been applied to Beam website.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148872)
Time Spent: 1h 20m  (was: 1h 10m)

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5467) Python Flink ValidatesRunner job fixes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5467?focusedWorklogId=148870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148870
 ]

ASF GitHub Bot logged work on BEAM-5467:


Author: ASF GitHub Bot
Created on: 27/Sep/18 20:00
Start Date: 27/Sep/18 20:00
Worklog Time Spent: 10m 
  Work Description: angoenka commented on issue #6510: [BEAM-5467] Use 
dynamic port while running Flink PVR
URL: https://github.com/apache/beam/pull/6510#issuecomment-425223695
 
 
   Run Python Flink ValidatesRunner


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148870)
Time Spent: 1h 10m  (was: 1h)

> Python Flink ValidatesRunner job fixes
> --
>
> Key: BEAM-5467
> URL: https://issues.apache.org/jira/browse/BEAM-5467
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise
>Priority: Minor
>  Labels: portability-flink
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Add status to README
> Rename script and job for consistency
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5467) Python Flink ValidatesRunner job fixes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5467?focusedWorklogId=148869=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148869
 ]

ASF GitHub Bot logged work on BEAM-5467:


Author: ASF GitHub Bot
Created on: 27/Sep/18 19:59
Start Date: 27/Sep/18 19:59
Worklog Time Spent: 10m 
  Work Description: angoenka opened a new pull request #6510: [BEAM-5467] 
Use dynamic port while running Flink PVR
URL: https://github.com/apache/beam/pull/6510
 
 
   **Please** add a meaningful description for your change here
   
   
   
   Follow this checklist to help us incorporate your contribution quickly and 
easily:
   
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue, if applicable. This will automatically link the pull request to the 
issue.
- [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   It will help us expedite review of your Pull Request if you tag someone 
(e.g. `@username`) to look at it.
   
   Post-Commit Tests Status (on master branch)
   

   
   Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
   --- | --- | --- | --- | --- | --- | --- | ---
   Go | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
 | --- | --- | --- | --- | --- | ---
   Java | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
   Python | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
 | --- | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
  [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
 | [![Build 
Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/)
 | --- | --- | ---
   
   
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148869)
Time Spent: 1h  (was: 50m)

> Python Flink ValidatesRunner job fixes
> --
>
> Key: BEAM-5467
> URL: https://issues.apache.org/jira/browse/BEAM-5467
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-flink
>Reporter: Thomas Weise
>Assignee: Thomas Weise

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148865
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 19:46
Start Date: 27/Sep/18 19:46
Worklog Time Spent: 10m 
  Work Description: asfgit closed pull request #562: [BEAM-5506] update 
create table to create external table
URL: https://github.com/apache/beam-site/pull/562
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/_includes/section-menu/sdks.html 
b/src/_includes/section-menu/sdks.html
index 0102b4bd9b..fc9603304d 100644
--- a/src/_includes/section-menu/sdks.html
+++ b/src/_includes/section-menu/sdks.html
@@ -64,7 +64,7 @@
   
 Data types
 Lexical structure
-CREATE TABLE
+CREATE EXTERNAL TABLE
 SELECT
 Windowing & 
Triggering
 Joins
diff --git a/src/documentation/dsls/sql/create-table.md 
b/src/documentation/dsls/sql/create-external-table.md
similarity index 89%
rename from src/documentation/dsls/sql/create-table.md
rename to src/documentation/dsls/sql/create-external-table.md
index cfa1d2d1ec..57c03f1c21 100644
--- a/src/documentation/dsls/sql/create-table.md
+++ b/src/documentation/dsls/sql/create-external-table.md
@@ -1,9 +1,11 @@
 ---
 layout: section
-title: "Beam SQL: CREATE TABLE Statement"
+title: "Beam SQL: CREATE EXTERNAL TABLE Statement"
 section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/create-table/
-redirect_from: /documentation/dsls/sql/statements/create-table/
+permalink: /documentation/dsls/sql/create-external-table/
+redirect_from:
+  -/documentation/dsls/sql/statements/create-table/
+  -/documentation/dsls/sql/create-table/
 ---
 
 
-# CREATE TABLE
+# CREATE EXTERNAL TABLE
 
-Beam SQL's `CREATE TABLE` statement registers a virtual table that maps to an
+Beam SQL's `CREATE EXTERNAL TABLE` statement registers a virtual table that 
maps to an
 [external storage system](https://beam.apache.org/documentation/io/built-in/).
-For some storage systems, `CREATE TABLE` does not create a physical table until
+For some storage systems, `CREATE EXTERNAL TABLE` does not create a physical 
table until
 a write occurs. After the physical table exists, you can access the table with
 the `SELECT`, `JOIN`, and `INSERT INTO` statements.
 
-The `CREATE TABLE` statement includes a schema and extended clauses.
+The `CREATE EXTERNAL TABLE` statement includes a schema and extended clauses.
 
 ## Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, 
tableElement ]*)
 TYPE type
 [LOCATION location]
 [TBLPROPERTIES tblProperties]
@@ -83,7 +85,7 @@ tableElement: columnName fieldType [ NOT NULL ]
 ### Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, 
tableElement ]*)
 TYPE bigquery
 LOCATION '[PROJECT_ID]:[DATASET].[TABLE]'
 ```
@@ -183,7 +185,7 @@ as follows:
 ### Example
 
 ```
-CREATE TABLE users (id INTEGER, username VARCHAR)
+CREATE EXTERNAL TABLE users (id INTEGER, username VARCHAR)
 TYPE bigquery
 LOCATION 'testing-integration:apache.users'
 ```
@@ -193,7 +195,7 @@ LOCATION 'testing-integration:apache.users'
 ### Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName
   (
event_timestamp TIMESTAMP,
attributes MAP,
@@ -263,7 +265,7 @@ declare a special set of columns, as shown below.
 ### Example
 
 ```
-CREATE TABLE locations (event_timestamp TIMESTAMP, attributes MAP, payload ROW)
+CREATE EXTERNAL TABLE locations (event_timestamp TIMESTAMP, attributes 
MAP, payload ROW)
 TYPE pubsub
 LOCATION 'projects/testing-integration/topics/user-location'
 ```
@@ -275,7 +277,7 @@ KafkaIO is experimental in Beam SQL.
 ### Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, 
tableElement ]*)
 TYPE kafka
 LOCATION 'kafka://localhost:2181/brokers'
 TBLPROPERTIES '{"bootstrap.servers":"localhost:9092", "topics": ["topic1", 
"topic2"]}'
@@ -313,7 +315,7 @@ access the same underlying data.
 ### Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, 
tableElement ]*)
 TYPE text
 LOCATION '/home/admin/orders'
 TBLPROPERTIES '{"format: "Excel"}'
@@ -345,7 +347,7 @@ Only simple types are supported.
 ### Example
 
 ```
-CREATE TABLE orders (id 

[beam-site] 01/03: update create table to create external table

[melap]

This is an automated email from the ASF dual-hosted git repository.

melap pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 93a09e4505a824a355d9381b8781c57869f9e333
Author: amaliujia 
AuthorDate: Tue Sep 25 14:05:33 2018 -0700

update create table to create external table
---
 src/_includes/section-menu/sdks.html   |  2 +-
 .../{create-table.md => create-external-table.md}  | 32 --
 src/documentation/dsls/sql/shell.md|  8 +++---
 3 files changed, 22 insertions(+), 20 deletions(-)

diff --git a/src/_includes/section-menu/sdks.html 
b/src/_includes/section-menu/sdks.html
index 0102b4b..fc96033 100644
--- a/src/_includes/section-menu/sdks.html
+++ b/src/_includes/section-menu/sdks.html
@@ -64,7 +64,7 @@
   
 Data types
 Lexical structure
-CREATE TABLE
+CREATE EXTERNAL TABLE
 SELECT
 Windowing & 
Triggering
 Joins
diff --git a/src/documentation/dsls/sql/create-table.md 
b/src/documentation/dsls/sql/create-external-table.md
similarity index 89%
rename from src/documentation/dsls/sql/create-table.md
rename to src/documentation/dsls/sql/create-external-table.md
index cfa1d2d..57c03f1 100644
--- a/src/documentation/dsls/sql/create-table.md
+++ b/src/documentation/dsls/sql/create-external-table.md
@@ -1,9 +1,11 @@
 ---
 layout: section
-title: "Beam SQL: CREATE TABLE Statement"
+title: "Beam SQL: CREATE EXTERNAL TABLE Statement"
 section_menu: section-menu/sdks.html
-permalink: /documentation/dsls/sql/create-table/
-redirect_from: /documentation/dsls/sql/statements/create-table/
+permalink: /documentation/dsls/sql/create-external-table/
+redirect_from:
+  -/documentation/dsls/sql/statements/create-table/
+  -/documentation/dsls/sql/create-table/
 ---
 
 
-# CREATE TABLE
+# CREATE EXTERNAL TABLE
 
-Beam SQL's `CREATE TABLE` statement registers a virtual table that maps to an
+Beam SQL's `CREATE EXTERNAL TABLE` statement registers a virtual table that 
maps to an
 [external storage system](https://beam.apache.org/documentation/io/built-in/).
-For some storage systems, `CREATE TABLE` does not create a physical table until
+For some storage systems, `CREATE EXTERNAL TABLE` does not create a physical 
table until
 a write occurs. After the physical table exists, you can access the table with
 the `SELECT`, `JOIN`, and `INSERT INTO` statements.
 
-The `CREATE TABLE` statement includes a schema and extended clauses.
+The `CREATE EXTERNAL TABLE` statement includes a schema and extended clauses.
 
 ## Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, 
tableElement ]*)
 TYPE type
 [LOCATION location]
 [TBLPROPERTIES tblProperties]
@@ -83,7 +85,7 @@ tableElement: columnName fieldType [ NOT NULL ]
 ### Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, 
tableElement ]*)
 TYPE bigquery
 LOCATION '[PROJECT_ID]:[DATASET].[TABLE]'
 ```
@@ -183,7 +185,7 @@ as follows:
 ### Example
 
 ```
-CREATE TABLE users (id INTEGER, username VARCHAR)
+CREATE EXTERNAL TABLE users (id INTEGER, username VARCHAR)
 TYPE bigquery
 LOCATION 'testing-integration:apache.users'
 ```
@@ -193,7 +195,7 @@ LOCATION 'testing-integration:apache.users'
 ### Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName
   (
event_timestamp TIMESTAMP,
attributes MAP,
@@ -263,7 +265,7 @@ declare a special set of columns, as shown below.
 ### Example
 
 ```
-CREATE TABLE locations (event_timestamp TIMESTAMP, attributes MAP, payload ROW)
+CREATE EXTERNAL TABLE locations (event_timestamp TIMESTAMP, attributes 
MAP, payload ROW)
 TYPE pubsub
 LOCATION 'projects/testing-integration/topics/user-location'
 ```
@@ -275,7 +277,7 @@ KafkaIO is experimental in Beam SQL.
 ### Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, 
tableElement ]*)
 TYPE kafka
 LOCATION 'kafka://localhost:2181/brokers'
 TBLPROPERTIES '{"bootstrap.servers":"localhost:9092", "topics": ["topic1", 
"topic2"]}'
@@ -313,7 +315,7 @@ access the same underlying data.
 ### Syntax
 
 ```
-CREATE TABLE [ IF NOT EXISTS ] tableName (tableElement [, tableElement ]*)
+CREATE EXTERNAL TABLE [ IF NOT EXISTS ] tableName (tableElement [, 
tableElement ]*)
 TYPE text
 LOCATION '/home/admin/orders'
 TBLPROPERTIES '{"format: "Excel"}'
@@ -345,7 +347,7 @@ Only simple types are supported.
 ### Example
 
 ```
-CREATE TABLE orders (id INTEGER, price INTEGER)
+CREATE EXTERNAL TABLE orders (id INTEGER, price INTEGER)
 TYPE text
 LOCATION '/home/admin/orders'
 ```
diff --git a/src/documentation/dsls/sql/shell.md 
b/src/documentation/dsls/sql/shell.md
index 6f9c32c..4ef670a 100644
--- 

[beam-site] 02/03: Regenerate website

[melap]

This is an automated email from the ASF dual-hosted git repository.

melap pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d9b8e36919d21e461c10355ae4b820f5d4c79ded
Author: Melissa Pashniak 
AuthorDate: Thu Sep 27 12:44:38 2018 -0700

Regenerate website
---
 .../dsls/sql}/create-table/index.html  |  8 +++---
 .../dsls/sql/aggregate-functions/index.html|  2 +-
 .../index.html | 30 +++---
 .../documentation/dsls/sql/data-types/index.html   |  2 +-
 content/documentation/dsls/sql/joins/index.html|  2 +-
 content/documentation/dsls/sql/lexical/index.html  |  2 +-
 content/documentation/dsls/sql/overview/index.html |  2 +-
 .../dsls/sql/scalar-functions/index.html   |  2 +-
 content/documentation/dsls/sql/select/index.html   |  2 +-
 content/documentation/dsls/sql/set/index.html  |  2 +-
 content/documentation/dsls/sql/shell/index.html| 10 
 .../dsls/sql/user-defined-functions/index.html |  2 +-
 .../documentation/dsls/sql/walkthrough/index.html  |  2 +-
 .../dsls/sql/windowing-and-triggering/index.html   |  2 +-
 .../sdks/feature-comparison/index.html |  2 +-
 content/documentation/sdks/go/index.html   |  2 +-
 .../documentation/sdks/java-extensions/index.html  |  2 +-
 .../documentation/sdks/java-thirdparty/index.html  |  2 +-
 .../documentation/sdks/java/euphoria/index.html|  2 +-
 content/documentation/sdks/java/index.html |  2 +-
 content/documentation/sdks/java/nexmark/index.html |  2 +-
 .../documentation/sdks/python-custom-io/index.html |  2 +-
 .../sdks/python-pipeline-dependencies/index.html   |  2 +-
 .../documentation/sdks/python-streaming/index.html |  2 +-
 .../sdks/python-type-safety/index.html |  2 +-
 content/documentation/sdks/python/index.html   |  2 +-
 26 files changed, 47 insertions(+), 47 deletions(-)

diff --git a/content/documentation/dsls/sql/statements/create-table/index.html 
b/content/-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql/create-table/index.html
similarity index 56%
rename from content/documentation/dsls/sql/statements/create-table/index.html
rename to content/-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql/create-table/index.html
index e603c02..d1ba0da 100644
--- a/content/documentation/dsls/sql/statements/create-table/index.html
+++ b/content/-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql/create-table/index.html
@@ -2,9 +2,9 @@
 
 
 Redirecting…
-https://beam.apache.org/documentation/dsls/sql/create-table/;>
-https://beam.apache.org/documentation/dsls/sql/create-table/;>
+https://beam.apache.org/documentation/dsls/sql/create-external-table/;>
+https://beam.apache.org/documentation/dsls/sql/create-external-table/;>
 Redirecting…
-https://beam.apache.org/documentation/dsls/sql/create-table/;>Click 
here if you are not redirected.
-location="https://beam.apache.org/documentation/dsls/sql/create-table/";
+https://beam.apache.org/documentation/dsls/sql/create-external-table/;>Click
 here if you are not redirected.
+location="https://beam.apache.org/documentation/dsls/sql/create-external-table/";
 
diff --git a/content/documentation/dsls/sql/aggregate-functions/index.html 
b/content/documentation/dsls/sql/aggregate-functions/index.html
index ce77ea5..4151697 100644
--- a/content/documentation/dsls/sql/aggregate-functions/index.html
+++ b/content/documentation/dsls/sql/aggregate-functions/index.html
@@ -190,7 +190,7 @@
   
 Data types
 Lexical 
structure
-CREATE 
TABLE
+CREATE 
EXTERNAL TABLE
 SELECT
 Windowing & 
Triggering
 Joins
diff --git a/content/documentation/dsls/sql/create-table/index.html 
b/content/documentation/dsls/sql/create-external-table/index.html
similarity index 96%
rename from content/documentation/dsls/sql/create-table/index.html
rename to content/documentation/dsls/sql/create-external-table/index.html
index c40550c..e2a4e9c 100644
--- a/content/documentation/dsls/sql/create-table/index.html
+++ b/content/documentation/dsls/sql/create-external-table/index.html
@@ -28,7 +28,7 @@
   
   
   
-  Beam SQL: CREATE TABLE Statement
+  Beam SQL: CREATE EXTERNAL TABLE Statement
   
   https://fonts.googleapis.com/css?family=Roboto:100,300,400; 
rel="stylesheet">
@@ -39,7 +39,7 @@
   
   
   
-  https://beam.apache.org/documentation/dsls/sql/create-table/; 
data-proofer-ignore>
+  https://beam.apache.org/documentation/dsls/sql/create-external-table/; 
data-proofer-ignore>
   
   https://beam.apache.org/feed.xml;>
   

[beam-site] 03/03: This closes #562

[melap]

This is an automated email from the ASF dual-hosted git repository.

melap pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d52bf3edfe5431f92709066b16c6a48b5b4f5588
Merge: 87fb41e d9b8e36
Author: Melissa Pashniak 
AuthorDate: Thu Sep 27 12:44:38 2018 -0700

This closes #562

 .../dsls/sql}/create-table/index.html  |  8 +++---
 .../dsls/sql/aggregate-functions/index.html|  2 +-
 .../index.html | 30 ++--
 .../documentation/dsls/sql/data-types/index.html   |  2 +-
 content/documentation/dsls/sql/joins/index.html|  2 +-
 content/documentation/dsls/sql/lexical/index.html  |  2 +-
 content/documentation/dsls/sql/overview/index.html |  2 +-
 .../dsls/sql/scalar-functions/index.html   |  2 +-
 content/documentation/dsls/sql/select/index.html   |  2 +-
 content/documentation/dsls/sql/set/index.html  |  2 +-
 content/documentation/dsls/sql/shell/index.html| 10 +++
 .../dsls/sql/user-defined-functions/index.html |  2 +-
 .../documentation/dsls/sql/walkthrough/index.html  |  2 +-
 .../dsls/sql/windowing-and-triggering/index.html   |  2 +-
 .../sdks/feature-comparison/index.html |  2 +-
 content/documentation/sdks/go/index.html   |  2 +-
 .../documentation/sdks/java-extensions/index.html  |  2 +-
 .../documentation/sdks/java-thirdparty/index.html  |  2 +-
 .../documentation/sdks/java/euphoria/index.html|  2 +-
 content/documentation/sdks/java/index.html |  2 +-
 content/documentation/sdks/java/nexmark/index.html |  2 +-
 .../documentation/sdks/python-custom-io/index.html |  2 +-
 .../sdks/python-pipeline-dependencies/index.html   |  2 +-
 .../documentation/sdks/python-streaming/index.html |  2 +-
 .../sdks/python-type-safety/index.html |  2 +-
 content/documentation/sdks/python/index.html   |  2 +-
 src/_includes/section-menu/sdks.html   |  2 +-
 .../{create-table.md => create-external-table.md}  | 32 --
 src/documentation/dsls/sql/shell.md|  8 +++---
 29 files changed, 69 insertions(+), 67 deletions(-)

[beam-site] branch asf-site updated (87fb41e -> d52bf3e)

[melap]

This is an automated email from the ASF dual-hosted git repository.

melap pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 87fb41e  Prepare repository for deployment.
 new 93a09e4  update create table to create external table
 new d9b8e36  Regenerate website
 new d52bf3e  This closes #562

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../dsls/sql}/create-table/index.html  |  8 +++---
 .../dsls/sql/aggregate-functions/index.html|  2 +-
 .../index.html | 30 ++--
 .../documentation/dsls/sql/data-types/index.html   |  2 +-
 content/documentation/dsls/sql/joins/index.html|  2 +-
 content/documentation/dsls/sql/lexical/index.html  |  2 +-
 content/documentation/dsls/sql/overview/index.html |  2 +-
 .../dsls/sql/scalar-functions/index.html   |  2 +-
 content/documentation/dsls/sql/select/index.html   |  2 +-
 content/documentation/dsls/sql/set/index.html  |  2 +-
 content/documentation/dsls/sql/shell/index.html| 10 +++
 .../dsls/sql/user-defined-functions/index.html |  2 +-
 .../documentation/dsls/sql/walkthrough/index.html  |  2 +-
 .../dsls/sql/windowing-and-triggering/index.html   |  2 +-
 .../sdks/feature-comparison/index.html |  2 +-
 content/documentation/sdks/go/index.html   |  2 +-
 .../documentation/sdks/java-extensions/index.html  |  2 +-
 .../documentation/sdks/java-thirdparty/index.html  |  2 +-
 .../documentation/sdks/java/euphoria/index.html|  2 +-
 content/documentation/sdks/java/index.html |  2 +-
 content/documentation/sdks/java/nexmark/index.html |  2 +-
 .../documentation/sdks/python-custom-io/index.html |  2 +-
 .../sdks/python-pipeline-dependencies/index.html   |  2 +-
 .../documentation/sdks/python-streaming/index.html |  2 +-
 .../sdks/python-type-safety/index.html |  2 +-
 content/documentation/sdks/python/index.html   |  2 +-
 src/_includes/section-menu/sdks.html   |  2 +-
 .../{create-table.md => create-external-table.md}  | 32 --
 src/documentation/dsls/sql/shell.md|  8 +++---
 29 files changed, 69 insertions(+), 67 deletions(-)
 rename content/{documentation/dsls/sql/statements => 
-/documentation/dsls/sql/statements/create-table/ 
-/documentation/dsls/sql}/create-table/index.html (56%)
 rename content/documentation/dsls/sql/{create-table => 
create-external-table}/index.html (96%)
 rename src/documentation/dsls/sql/{create-table.md => 
create-external-table.md} (89%)

Jenkins build is back to normal : beam_PostCommit_Python_Verify #6108

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148860
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 19:40
Start Date: 27/Sep/18 19:40
Worklog Time Spent: 10m 
  Work Description: amaliujia commented on issue #562: [BEAM-5506] update 
create table to create external table
URL: https://github.com/apache/beam-site/pull/562#issuecomment-425218132
 
 
   @melap 
   
   Done!
   
   Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148860)
Time Spent: 50m  (was: 40m)

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5407) [beam_PostCommit_Go_GradleBuild][testE2ETopWikiPages][RolledBack] Breaks post commit

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5407:

Summary: [beam_PostCommit_Go_GradleBuild][testE2ETopWikiPages][RolledBack] 
Breaks post commit  (was: [beam_PostCommit_Go_GradleBuild][testE2ETopWikiPages] 
Breaks post commit)

> [beam_PostCommit_Go_GradleBuild][testE2ETopWikiPages][RolledBack] Breaks post 
> commit
> 
>
> Key: BEAM-5407
> URL: https://issues.apache.org/jira/browse/BEAM-5407
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Pablo Estrada
>Priority: Major
>
> Failing job url: 
> https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1482/testReport/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-5506) Update Beam documentation

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5506?focusedWorklogId=148857=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148857
 ]

ASF GitHub Bot logged work on BEAM-5506:


Author: ASF GitHub Bot
Created on: 27/Sep/18 19:16
Start Date: 27/Sep/18 19:16
Worklog Time Spent: 10m 
  Work Description: melap commented on issue #562: [BEAM-5506] update 
create table to create external table
URL: https://github.com/apache/beam-site/pull/562#issuecomment-425211244
 
 
   @amaliujia could you please squash your commits into one, and then I will 
merge. thanks!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148857)
Time Spent: 40m  (was: 0.5h)

> Update Beam documentation
> -
>
> Key: BEAM-5506
> URL: https://issues.apache.org/jira/browse/BEAM-5506
> Project: Beam
>  Issue Type: Sub-task
>  Components: dsl-sql
>Reporter: Rui Wang
>Assignee: Rui Wang
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5294) [beam_Release_Gradle_NightlySnapshot] Failing due to website test.

[Alan Myrvold (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630889#comment-16630889
 ] 

Alan Myrvold commented on BEAM-5294:


Fixed by https://github.com/apache/beam/pull/6502

> [beam_Release_Gradle_NightlySnapshot] Failing due to website test.
> --
>
> Key: BEAM-5294
> URL: https://issues.apache.org/jira/browse/BEAM-5294
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Build link: 
> [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/]
> [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/185/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4783) Spark SourceRDD Not Designed With Dynamic Allocation In Mind

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4783?focusedWorklogId=148849=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-148849
 ]

ASF GitHub Bot logged work on BEAM-4783:


Author: ASF GitHub Bot
Created on: 27/Sep/18 18:44
Start Date: 27/Sep/18 18:44
Worklog Time Spent: 10m 
  Work Description: kyle-winkelman commented on issue #6181: [BEAM-4783] 
Add bundleSize for splitting BoundedSources.
URL: https://github.com/apache/beam/pull/6181#issuecomment-425201311
 
 
   Looking further into the StreamingTransformTranslator, I would like to pose 
a question. Why do we do the groupByKey followed by the updateStateByKey? It 
appears to be a giant waste in which we convert everything to bytes and back 
unnecessarily.
   
   The only thing it does is gather all the values for a key into an Iterable, 
but the updateStateByKey would also do that if it were given the chance.
   
   If we were to update the UpdateStateByKeyFunction to expect 
WindowedValue's instead of Iterable>'s I believe we could 
eliminate the call to groupByKey. What is happening now is the updateStateByKey 
will wrap those values in a Seq and so currently we have either an empty Seq or 
a Seq with exactly 1 item and that item is itself an Iterable that contains 
multiple items.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 148849)
Time Spent: 3h 20m  (was: 3h 10m)

> Spark SourceRDD Not Designed With Dynamic Allocation In Mind
> 
>
> Key: BEAM-4783
> URL: https://issues.apache.org/jira/browse/BEAM-4783
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-spark
>Affects Versions: 2.5.0
>Reporter: Kyle Winkelman
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Labels: newbie
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> When the spark-runner is used along with the configuration 
> spark.dynamicAllocation.enabled=true the SourceRDD does not detect this. It 
> then falls back to the value calculated in this description:
>   // when running on YARN/SparkDeploy it's the result of max(totalCores, 
> 2).
>   // when running on Mesos it's 8.
>   // when running local it's the total number of cores (local = 1, 
> local[N] = N,
>   // local[*] = estimation of the machine's cores).
>   // ** the configuration "spark.default.parallelism" takes precedence 
> over all of the above **
> So in most cases this default is quite small. This is an issue when using a 
> very large input file as it will only get split in half.
> I believe that when Dynamic Allocation is enable the SourceRDD should use the 
> DEFAULT_BUNDLE_SIZE and possibly expose a SparkPipelineOptions that allows 
> you to change this DEFAULT_BUNDLE_SIZE.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #6107

[Apache Jenkins Server]

See 


Changes:

[shaun] added avroio package

[shaun] updated read emits to support both string and custom type reflects

[shaun] added avro write support

[shaun] updated to be in-line with beam project specifications

[shaun] update package log prints

[shaun] added readavro example

[shaun] updated example package header

[shaun] removed output.avro file

--
[...truncated 1.28 MB...]
wrapper = lambda x: [fn(x)]
  File 
"
 line 185, in raise_error
raise RuntimeError('x')
RuntimeError: x

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File 
"
 line 131, in _execute
response = task()
  File 
"
 line 166, in 
self._execute(lambda: worker.do_instruction(work), work)
  File 
"
 line 212, in do_instruction
request.instruction_id)
  File 
"
 line 234, in process_bundle
processor.process_bundle(instruction_id)
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
self.gen.throw(type, value, traceback)
  File 
"
 line 349, in process_instruction_id
yield
  File 
"
 line 234, in process_bundle
processor.process_bundle(instruction_id)
  File 
"
 line 387, in process_bundle
input_op.process_encoded(data.data)
  File 
"
 line 123, in process_encoded
self.output(decoded_value)
  File 
"
 line 167, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 268, in process
self.output(windowed_value)
  File 
"
 line 167, in output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, consumer).process(windowed_value)
  File 
"
 line 414, in process
self.dofn_receiver.receive(o)
  File 
"
 line 673, in receive
self.process(windowed_value)
  File 
"
 line 679, in process
self._reraise_augmented(exn)
  File 
"
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"
 line 787, in process_outputs
self.main_receivers.receive(windowed_value)
  File 
"
 line 87, in receive
cython.cast(Operation, 

[jira] [Updated] (BEAM-5286) [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake] .sh script: text file busy.

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5286:

Summary: 
[beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake]
 .sh script: text file busy.  (was: Java PostCommit failed: 
org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline)

> [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake]
>  .sh script: text file busy.
> --
>
> Key: BEAM-5286
> URL: https://issues.apache.org/jira/browse/BEAM-5286
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Sample failure: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1375/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/]
> Sample relevant log:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.Exception: 
> java.io.IOException: Cannot run program 
> "/tmp/test-Echoo1519764280436328522/test-EchoAgain3143210610074994370.sh": 
> error=26, Text file busy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5286) Java PostCommit failed: org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin reassigned BEAM-5286:
---

Assignee: Alan Myrvold

> Java PostCommit failed: 
> org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline
> ---
>
> Key: BEAM-5286
> URL: https://issues.apache.org/jira/browse/BEAM-5286
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Sample failure: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1375/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/]
> Sample relevant log:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.Exception: 
> java.io.IOException: Cannot run program 
> "/tmp/test-Echoo1519764280436328522/test-EchoAgain3143210610074994370.sh": 
> error=26, Text file busy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5286) Java PostCommit failed: org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin updated BEAM-5286:

Description: 
Sample failure: 
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1375/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/]

Sample relevant log:

org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.Exception: 
java.io.IOException: Cannot run program 
"/tmp/test-Echoo1519764280436328522/test-EchoAgain3143210610074994370.sh": 
error=26, Text file busy

  
was:https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1375/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/


> Java PostCommit failed: 
> org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline
> ---
>
> Key: BEAM-5286
> URL: https://issues.apache.org/jira/browse/BEAM-5286
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Boyuan Zhang
>Priority: Major
>
> Sample failure: 
> [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1375/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/]
> Sample relevant log:
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.Exception: 
> java.io.IOException: Cannot run program 
> "/tmp/test-Echoo1519764280436328522/test-EchoAgain3143210610074994370.sh": 
> error=26, Text file busy



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5294) [beam_Release_Gradle_NightlySnapshot] Failing due to website test.

[Mikhail Gryzykhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Gryzykhin reassigned BEAM-5294:
---

   Assignee: Alan Myrvold
Description: 
Build link: [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/]

[https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/185/]

 

  was:Build link: 
https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/

Summary: [beam_Release_Gradle_NightlySnapshot] Failing due to website 
test.  (was: java snapshot build continues failed)

> [beam_Release_Gradle_NightlySnapshot] Failing due to website test.
> --
>
> Key: BEAM-5294
> URL: https://issues.apache.org/jira/browse/BEAM-5294
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Boyuan Zhang
>Assignee: Alan Myrvold
>Priority: Major
>
> Build link: 
> [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/]
> [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/185/]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Python_VR_Flink #161

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Website_Publish #17

[Apache Jenkins Server]

See 


--
[...truncated 8.18 KB...]
> Task :buildSrc:assemble
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Task worker for ':buildSrc',5,main]) completed. Took 
0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 1.75 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) completed. Took 0.001 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) completed. Took 0.036 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for 
':buildSrc',5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for 
':buildSrc',5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc',5,main]) completed. 
Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) 
started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc',5,main]) 
completed. Took 0.001 secs.
:buildSrc:processTestResources (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Daemon worker,5,main]) completed. Took 
0.001 secs.
:buildSrc:testClasses (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:test (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:test NO-SOURCE
Skipping task ':buildSrc:test' as it has no source files and no previous output 
files.
:buildSrc:test (Thread[Daemon worker,5,main]) completed. Took 0.002 secs.
:buildSrc:check (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:check
Skipping task ':buildSrc:check' as it has no actions.
:buildSrc:check (Thread[Daemon worker,5,main]) completed. Took 0.0 secs.
:buildSrc:build (Thread[Daemon worker,5,main]) started.

> Task :buildSrc:build
Skipping task ':buildSrc:build' as it has no actions.
:buildSrc:build 

Build failed in Jenkins: beam_PostCommit_Java_GradleBuild #1564

[Apache Jenkins Server]

See 


Changes:

[scott] [BEAM-5518] Ignore failing ssl validation of globenewswire (#6502)

--
[...truncated 39.28 MB...]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
at 
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at 
io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
... 3 more

Sep 27, 2018 6:05:22 PM 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn processElement
WARNING: Failed to submit the mutation group
com.google.cloud.spanner.SpannerException: FAILED_PRECONDITION: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
at 
com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:456)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.commit(GrpcSpannerRpc.java:404)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:797)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:794)
at 
com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:227)
at 
com.google.cloud.spanner.SpannerImpl$SessionImpl.writeAtLeastOnce(SpannerImpl.java:793)
at 
com.google.cloud.spanner.SessionPool$PooledSession.writeAtLeastOnce(SessionPool.java:319)
at 
com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:60)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn.processElement(SpannerIO.java:1108)
at 
org.apache.beam.sdk.io.gcp.spanner.SpannerIO$WriteToSpannerFn$DoFnInvoker.invokeProcessElement(Unknown
 Source)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:275)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:240)
at 
org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows(SimplePushbackSideInputDoFnRunner.java:78)
at 
org.apache.beam.runners.direct.ParDoEvaluator.processElement(ParDoEvaluator.java:207)
at 
org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement(DoFnLifecycleManagerRemovingTransformEvaluator.java:55)
at 
org.apache.beam.runners.direct.DirectTransformExecutor.processElements(DirectTransformExecutor.java:160)
at 
org.apache.beam.runners.direct.DirectTransformExecutor.run(DirectTransformExecutor.java:124)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: 
io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must not be NULL in 
table users.
at 
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
at 
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
at 
com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:450)
... 21 more
Caused by: io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Value must 
not be NULL in table users.
at io.grpc.Status.asRuntimeException(Status.java:526)
at 
io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:468)
at 
io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at 
io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at 
io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at 
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100)
at 

Jenkins build is back to normal : beam_PostCommit_Py_VR_Dataflow #1181

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PreCommit_Website_Cron #111

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_VR_Flink #160

[Apache Jenkins Server]

See 


Changes:

[shaun] added avroio package

[shaun] updated read emits to support both string and custom type reflects

[shaun] added avro write support

[shaun] updated to be in-line with beam project specifications

[shaun] update package log prints

[shaun] added readavro example

[shaun] updated example package header

[shaun] removed output.avro file

--
[...truncated 51.25 MB...]
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
c633cf15e30ed2275224a3354e8fd86c.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (1/16) 
(1f2b260c3b409534308db2a8f5d2eee8) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
a3e4eef03690e31a45011788d2431e82.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (2/16) 
(bf05d0b948b39e429170155db38e9a17) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(16/16) (2e74ffba7924f8876449d1eee7b17c06) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
c40b18006bfb098619ce15fe99192466.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem (8/16) 
(69ce17ab65312a85eddff6bf1bf7ad2b) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
c054ed4eea544cba2619c670f29fa532.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
72cf525daf848ff48de315f4b05ab4e1.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
cb6c713edcaf5b5643db1407910d999a.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - ToKeyedWorkItem 
(13/16) (3cb9a4fed799c743790b7a4f36893aac) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
0674be15c7e70082bf893d8569a3c10b.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
b5d075cdc08a5f6b8026ac65fc2a8fd4.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
d061bb6995f1769d399fd3235ffa17ac.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
e0b371c06cfd05de68338186776a6dc3.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 
f19bf3c54f91e1d3e00ea4dae6fc80d4.
[flink-akka.actor.default-dispatcher-5] INFO 
org.apache.flink.runtime.executiongraph.ExecutionGraph - GroupByKey -> 
24GroupByKey/GroupByWindow.None/beam:env:docker:v1:0 (12/16) 
(831b4b7cc163e734c69eea42a5b8aff0) switched from RUNNING to FINISHED.
[flink-akka.actor.default-dispatcher-2] INFO 
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task and 
sending final execution state FINISHED to JobManager for task ToKeyedWorkItem 
17e36560b2b3c027ddcc18a2efa7f69f.
[flink-akka.actor.default-dispatcher-2] INFO 

Build failed in Jenkins: beam_PostCommit_Website_Publish #16

[Apache Jenkins Server]

See 


Changes:

[shaun] added avroio package

[shaun] updated read emits to support both string and custom type reflects

[shaun] added avro write support

[shaun] updated to be in-line with beam project specifications

[shaun] update package log prints

[shaun] added readavro example

[shaun] updated example package header

[shaun] removed output.avro file

--
[...truncated 8.30 KB...]
> Task :buildSrc:assemble
Skipping task ':buildSrc:assemble' as it has no actions.
:buildSrc:assemble (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
completed. Took 0.0 secs.
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
started.

> Task :buildSrc:spotlessGroovy
file or directory 
'
 not found
file or directory 
'
 not found
file or directory 
'
 not found
Caching disabled for task ':buildSrc:spotlessGroovy': Caching has not been 
enabled for the task
Task ':buildSrc:spotlessGroovy' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovy'.
file or directory 
'
 not found
:buildSrc:spotlessGroovy (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
completed. Took 1.44 secs.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) started.

> Task :buildSrc:spotlessGroovyCheck
Skipping task ':buildSrc:spotlessGroovyCheck' as it has no actions.
:buildSrc:spotlessGroovyCheck (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) started.

> Task :buildSrc:spotlessGroovyGradle
Caching disabled for task ':buildSrc:spotlessGroovyGradle': Caching has not 
been enabled for the task
Task ':buildSrc:spotlessGroovyGradle' is not up-to-date because:
  No history is available.
All input files are considered out-of-date for incremental task 
':buildSrc:spotlessGroovyGradle'.
:buildSrc:spotlessGroovyGradle (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) completed. Took 0.023 secs.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) started.

> Task :buildSrc:spotlessGroovyGradleCheck
Skipping task ':buildSrc:spotlessGroovyGradleCheck' as it has no actions.
:buildSrc:spotlessGroovyGradleCheck (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) completed. Took 0.0 secs.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
started.

> Task :buildSrc:spotlessCheck
Skipping task ':buildSrc:spotlessCheck' as it has no actions.
:buildSrc:spotlessCheck (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
completed. Took 0.0 secs.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) started.

> Task :buildSrc:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestJava' as it has no source files and no 
previous output files.
:buildSrc:compileTestJava (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) completed. Took 0.001 secs.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) started.

> Task :buildSrc:compileTestGroovy NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:compileTestGroovy' as it has no source files and no 
previous output files.
:buildSrc:compileTestGroovy (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) completed. Took 0.001 secs.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) started.

> Task :buildSrc:processTestResources NO-SOURCE
file or directory 
'
 not found
Skipping task ':buildSrc:processTestResources' as it has no source files and no 
previous output files.
:buildSrc:processTestResources (Thread[Task worker for ':buildSrc' Thread 
11,5,main]) completed. Took 0.001 secs.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
started.

> Task :buildSrc:testClasses UP-TO-DATE
Skipping task ':buildSrc:testClasses' as it has no actions.
:buildSrc:testClasses (Thread[Task worker for ':buildSrc' Thread 11,5,main]) 
completed. Took 0.0 secs.
:buildSrc:test (Thread[Task worker for