Build failed in Jenkins: beam_PerformanceTests_Python #946

[Apache Jenkins Server]

See 


Changes:

[kenn] Explicitly exclude further optional deps from elasticsearch-hadoop

[lcwik] [BEAM-2573] Don't force importing filesystems, if they fail then give up

[kirpichov] Adds more logging of BigQuery jobs and makes load job naming more

[coheigea] Make sure there is a space between closing round bracket and opening

[altay] Fixing minor bugs:

[ankurgoenka] Making default thread count 12

--
[...truncated 550 B...]
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 817cf972ddce7e1c7682c3379c2cceadf1f96317 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 817cf972ddce7e1c7682c3379c2cceadf1f96317
Commit message: "Merge pull request #4735 from angoenka/change_default_thread"
 > git rev-list --no-walk f4b65d31af60107d78d23ff4dc5052aab16c5468 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8114221554133351690.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5037848381467106805.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4536238407753025656.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8737160457483219610.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3989534979452374020.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2874130366069814259.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #190

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3738) Enable Py3 linting in Jenkins

[holdenk (JIRA)]

holdenk created BEAM-3738:
-

 Summary: Enable Py3 linting in Jenkins
 Key: BEAM-3738
 URL: https://issues.apache.org/jira/browse/BEAM-3738
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core, testing
Reporter: holdenk
Assignee: Ahmet Altay


After BEAM-3671 is finished enable linting.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4735 from angoenka/change_default_thread

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 817cf972ddce7e1c7682c3379c2cceadf1f96317
Merge: 8a54e65 0762e19
Author: Ahmet Altay 
AuthorDate: Thu Feb 22 19:39:13 2018 -0800

Merge pull request #4735 from angoenka/change_default_thread

Making default thread count 12

 sdks/python/apache_beam/runners/worker/sdk_worker_main.py  | 2 +-
 sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (8a54e65 -> 817cf97)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8a54e65  Merge pull request #4733 from aaltay/bugs
 add 0762e19  Making default thread count 12
 new 817cf97  Merge pull request #4735 from angoenka/change_default_thread

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/worker/sdk_worker_main.py  | 2 +-
 sdks/python/apache_beam/runners/worker/sdk_worker_main_test.py | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] branch master updated (9c7c5d2 -> 8a54e65)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9c7c5d2  Merge pull request #4725: [BEAM-3715] Explicitly exclude 
further optional Cascading deps
 add 3dc643f  Fixing minor bugs:
 new 8a54e65  Merge pull request #4733 from aaltay/bugs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/options/value_provider.py | 3 +++
 sdks/python/apache_beam/utils/retry.py| 4 +++-
 2 files changed, 6 insertions(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Commented] (BEAM-3737) Key-aware batching function

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373867#comment-16373867
 ] 

Kenneth Knowles commented on BEAM-3737:
---

Would you be interested in contributing something here?

> Key-aware batching function
> ---
>
> Key: BEAM-3737
> URL: https://issues.apache.org/jira/browse/BEAM-3737
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Chuan Yu Foo
>Assignee: Kenneth Knowles
>Priority: Major
>
> I have a CombineFn for which add_input has very large overhead. I would like 
> to batch the incoming elements into a large batch before each call to 
> add_input to reduce this overhead. In other words, I would like to do 
> something like: 
> {{elements | GroupByKey() | BatchElements() | CombineValues(MyCombineFn())}}
> Unfortunately, BatchElements is not key-aware, and can't be used after a 
> GroupByKey to batch elements per key. I'm working around this by doing the 
> batching within CombineValues, which makes the CombineFn rather messy. It 
> would be nice if there were a key-aware BatchElements transform which could 
> be used in this context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3737) Key-aware batching function

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373866#comment-16373866
 ] 

Kenneth Knowles commented on BEAM-3737:
---

This is a very interesting scenario!

Can you build a {{BatchingCombineFn}} that has an accumulator that just buffers 
elements for a while and compacts them as needed? Then in {{extractOutput}} you 
can do the final conversion.

> Key-aware batching function
> ---
>
> Key: BEAM-3737
> URL: https://issues.apache.org/jira/browse/BEAM-3737
> Project: Beam
>  Issue Type: New Feature
>  Components: beam-model
>Reporter: Chuan Yu Foo
>Assignee: Kenneth Knowles
>Priority: Major
>
> I have a CombineFn for which add_input has very large overhead. I would like 
> to batch the incoming elements into a large batch before each call to 
> add_input to reduce this overhead. In other words, I would like to do 
> something like: 
> {{elements | GroupByKey() | BatchElements() | CombineValues(MyCombineFn())}}
> Unfortunately, BatchElements is not key-aware, and can't be used after a 
> GroupByKey to batch elements per key. I'm working around this by doing the 
> batching within CombineValues, which makes the CombineFn rather messy. It 
> would be nice if there were a key-aware BatchElements transform which could 
> be used in this context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (7bb4dd9 -> 9c7c5d2)

[kenn]

This is an automated email from the ASF dual-hosted git repository.

kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7bb4dd9  Merge pull request #4724: Adds more logging around BigQuery 
jobs
 add c3881f5  Explicitly exclude further optional deps from 
elasticsearch-hadoop
 new 9c7c5d2  Merge pull request #4725: [BEAM-3715] Explicitly exclude 
further optional Cascading deps

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/hadoop-input-format/build.gradle | 3 +++
 1 file changed, 3 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
k...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #973

[Apache Jenkins Server]

See 


Changes:

[coheigea] Make sure there is a space between closing round bracket and opening

--
[...truncated 121.35 KB...]
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert_that/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s11"
}, 
"serialized_fn": 
"%0AD%22B%0A%1Dref_Coder_GlobalWindowCoder_1%12%21%0A%1F%0A%1D%0A%1Bbeam%3Acoder%3Aglobal_window%3Av1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "assert_that/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s13", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert_that/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s12"
}, 
"serialized_fn": "", 
"user_name": "assert_that/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s14", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 

[beam] 01/01: Merge pull request #4724: Adds more logging around BigQuery jobs

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7bb4dd9cb8cc415d30d8865f22f6d5e6acaf9997
Merge: 701f531 4a25ce1
Author: Eugene Kirpichov 
AuthorDate: Thu Feb 22 18:00:42 2018 -0800

Merge pull request #4724: Adds more logging around BigQuery jobs

Adds more logging around BigQuery jobs

 .../beam/sdk/io/gcp/bigquery/BatchLoads.java   | 50 --
 .../sdk/io/gcp/bigquery/BigQueryServicesImpl.java  | 26 +--
 .../beam/sdk/io/gcp/bigquery/WriteTables.java  | 24 ---
 3 files changed, 56 insertions(+), 44 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[beam] branch master updated (701f531 -> 7bb4dd9)

[jkff]

This is an automated email from the ASF dual-hosted git repository.

jkff pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 701f531  Make sure there is a space between closing round bracket and 
opening …
 add 4a25ce1  Adds more logging of BigQuery jobs and makes load job naming 
more structured
 new 7bb4dd9  Merge pull request #4724: Adds more logging around BigQuery 
jobs

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/gcp/bigquery/BatchLoads.java   | 50 --
 .../sdk/io/gcp/bigquery/BigQueryServicesImpl.java  | 26 +--
 .../beam/sdk/io/gcp/bigquery/WriteTables.java  | 24 ---
 3 files changed, 56 insertions(+), 44 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
j...@apache.org.

[jira] [Resolved] (BEAM-3724) Make the coders package compatible with Python 3

[Luke Zhu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Zhu resolved BEAM-3724.

   Resolution: Duplicate
Fix Version/s: Not applicable

> Make the coders package compatible with Python 3
> 
>
> Key: BEAM-3724
> URL: https://issues.apache.org/jira/browse/BEAM-3724
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Luke Zhu
>Assignee: Luke Zhu
>Priority: Major
> Fix For: Not applicable
>
>
> The coders package is affect a lot by the fact that Strings are unicode in 
> Python 3.
>  
> The planned approach is to
>  * Prefix bytestrings with 'b' where appropriate
>  * Replace uses of 'str' with 'bytes' where appropriate
>  * Use python-modernize to solve syntax and import errors
> The goal of this subtask is not to make the coders package completely 
> compatible with Python 3.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (5518664 -> 701f531)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5518664  [BEAM-2573] Don't force importing filesystems, if they fail 
then give up and continue.
 add 50a8432  Make sure there is a space between closing round bracket and 
opening curly bracket
 new 701f531  Make sure there is a space between closing round bracket and 
opening …

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/examples/cookbook/BigQueryTornadoes.java |  2 +-
 .../apache/beam/examples/cookbook/CombinePerKeyExamples.java |  2 +-
 .../org/apache/beam/examples/cookbook/FilterExamples.java|  6 +++---
 .../org/apache/beam/examples/cookbook/TriggerExample.java|  6 +++---
 .../java/org/apache/beam/runners/flink/ReadSourceITCase.java |  2 +-
 .../apache/beam/runners/flink/ReadSourceStreamingITCase.java |  2 +-
 .../beam/runners/flink/streaming/GroupByNullKeyTest.java |  2 +-
 .../runners/flink/streaming/TopWikipediaSessionsITCase.java  |  2 +-
 .../beam/runners/spark/translation/MultiDoFnFunction.java|  2 +-
 .../beam/runners/spark/translation/SparkContextFactory.java  |  2 +-
 .../apache/beam/runners/spark/ProvidedSparkContextTest.java  |  2 +-
 .../core/src/main/java/org/apache/beam/sdk/coders/Coder.java |  2 +-
 .../main/java/org/apache/beam/sdk/coders/CoderProviders.java |  2 +-
 .../src/main/java/org/apache/beam/sdk/coders/RowCoder.java   |  4 ++--
 .../java/org/apache/beam/sdk/testing/WindowFnTestUtils.java  |  6 +++---
 .../org/apache/beam/sdk/transforms/GroupIntoBatches.java |  2 +-
 .../beam/sdk/transforms/reflect/OnTimerMethodSpecifier.java  |  2 +-
 .../src/main/java/org/apache/beam/sdk/values/RowType.java| 12 ++--
 .../java/org/apache/beam/sdk/coders/StructuredCoderTest.java |  2 +-
 .../src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java |  2 +-
 .../org/apache/beam/sdk/transforms/GroupIntoBatchesTest.java |  2 +-
 .../test/java/org/apache/beam/sdk/transforms/ParDoTest.java  |  2 +-
 .../beam/sdk/transforms/display/DisplayDataMatchersTest.java |  2 +-
 .../beam/sdk/util/ExposedByteArrayOutputStreamTest.java  |  2 +-
 .../beam/sdk/util/RetryHttpRequestInitializerTest.java   |  2 +-
 .../java/org/apache/beam/sdk/extensions/sql/BeamSql.java |  1 +
 .../org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java  |  2 +-
 .../sql/impl/interpreter/operator/BeamSqlPrimitive.java  |  2 +-
 .../operator/comparison/BeamSqlIsNotNullExpression.java  |  2 +-
 .../operator/comparison/BeamSqlIsNullExpression.java |  2 +-
 .../sdk/extensions/sql/impl/schema/BeamPCollectionTable.java |  2 +-
 .../sql/impl/transform/BeamAggregationTransforms.java|  2 +-
 .../extensions/sql/impl/transform/BeamJoinTransforms.java|  2 +-
 .../sdk/extensions/sql/impl/transform/agg/CovarianceFn.java  |  2 +-
 .../sdk/extensions/sql/impl/transform/agg/VarianceFn.java|  2 +-
 .../org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java   |  2 +-
 .../beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java   |  2 +-
 .../interpreter/operator/BeamSqlCompareExpressionTest.java   | 10 +-
 .../interpreter/operator/BeamSqlInputRefExpressionTest.java  |  4 ++--
 .../sql/impl/interpreter/operator/BeamSqlPrimitiveTest.java  | 10 +-
 .../beam/sdk/extensions/sql/mock/MockedBoundedTable.java |  2 +-
 .../beam/sdk/extensions/sql/mock/MockedUnboundedTable.java   |  2 +-
 .../beam/sdk/io/elasticsearch/ElasticsearchIOTest.java   |  8 
 .../beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java  |  4 ++--
 .../beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java |  2 +-
 .../apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java| 12 ++--
 .../src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java  |  4 ++--
 .../java/org/apache/beam/sdk/io/gcp/spanner/OrderedCode.java |  2 +-
 .../org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java  |  2 +-
 .../beam/sdk/io/gcp/bigtable/BigtableServiceImplTest.java|  2 +-
 .../beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java  |  2 +-
 .../sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java   |  2 +-
 .../main/java/org/apache/beam/sdk/nexmark/model/Event.java   |  2 +-
 53 files changed, 84 insertions(+), 83 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: Make sure there is a space between closing round bracket and opening …

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 701f531466d93bb82229e856444781627256a0e9
Merge: 5518664 50a8432
Author: Lukasz Cwik 
AuthorDate: Thu Feb 22 17:18:31 2018 -0800

Make sure there is a space between closing round bracket and opening …

 .../org/apache/beam/examples/cookbook/BigQueryTornadoes.java |  2 +-
 .../apache/beam/examples/cookbook/CombinePerKeyExamples.java |  2 +-
 .../org/apache/beam/examples/cookbook/FilterExamples.java|  6 +++---
 .../org/apache/beam/examples/cookbook/TriggerExample.java|  6 +++---
 .../java/org/apache/beam/runners/flink/ReadSourceITCase.java |  2 +-
 .../apache/beam/runners/flink/ReadSourceStreamingITCase.java |  2 +-
 .../beam/runners/flink/streaming/GroupByNullKeyTest.java |  2 +-
 .../runners/flink/streaming/TopWikipediaSessionsITCase.java  |  2 +-
 .../beam/runners/spark/translation/MultiDoFnFunction.java|  2 +-
 .../beam/runners/spark/translation/SparkContextFactory.java  |  2 +-
 .../apache/beam/runners/spark/ProvidedSparkContextTest.java  |  2 +-
 .../core/src/main/java/org/apache/beam/sdk/coders/Coder.java |  2 +-
 .../main/java/org/apache/beam/sdk/coders/CoderProviders.java |  2 +-
 .../src/main/java/org/apache/beam/sdk/coders/RowCoder.java   |  4 ++--
 .../java/org/apache/beam/sdk/testing/WindowFnTestUtils.java  |  6 +++---
 .../org/apache/beam/sdk/transforms/GroupIntoBatches.java |  2 +-
 .../beam/sdk/transforms/reflect/OnTimerMethodSpecifier.java  |  2 +-
 .../src/main/java/org/apache/beam/sdk/values/RowType.java| 12 ++--
 .../java/org/apache/beam/sdk/coders/StructuredCoderTest.java |  2 +-
 .../src/test/java/org/apache/beam/sdk/io/TextIOReadTest.java |  2 +-
 .../org/apache/beam/sdk/transforms/GroupIntoBatchesTest.java |  2 +-
 .../test/java/org/apache/beam/sdk/transforms/ParDoTest.java  |  2 +-
 .../beam/sdk/transforms/display/DisplayDataMatchersTest.java |  2 +-
 .../beam/sdk/util/ExposedByteArrayOutputStreamTest.java  |  2 +-
 .../beam/sdk/util/RetryHttpRequestInitializerTest.java   |  2 +-
 .../java/org/apache/beam/sdk/extensions/sql/BeamSql.java |  1 +
 .../org/apache/beam/sdk/extensions/sql/impl/BeamSqlEnv.java  |  2 +-
 .../sql/impl/interpreter/operator/BeamSqlPrimitive.java  |  2 +-
 .../operator/comparison/BeamSqlIsNotNullExpression.java  |  2 +-
 .../operator/comparison/BeamSqlIsNullExpression.java |  2 +-
 .../sdk/extensions/sql/impl/schema/BeamPCollectionTable.java |  2 +-
 .../sql/impl/transform/BeamAggregationTransforms.java|  2 +-
 .../extensions/sql/impl/transform/BeamJoinTransforms.java|  2 +-
 .../sdk/extensions/sql/impl/transform/agg/CovarianceFn.java  |  2 +-
 .../sdk/extensions/sql/impl/transform/agg/VarianceFn.java|  2 +-
 .../org/apache/beam/sdk/extensions/sql/BeamSqlDslBase.java   |  2 +-
 .../beam/sdk/extensions/sql/BeamSqlDslUdfUdafTest.java   |  2 +-
 .../interpreter/operator/BeamSqlCompareExpressionTest.java   | 10 +-
 .../interpreter/operator/BeamSqlInputRefExpressionTest.java  |  4 ++--
 .../sql/impl/interpreter/operator/BeamSqlPrimitiveTest.java  | 10 +-
 .../beam/sdk/extensions/sql/mock/MockedBoundedTable.java |  2 +-
 .../beam/sdk/extensions/sql/mock/MockedUnboundedTable.java   |  2 +-
 .../beam/sdk/io/elasticsearch/ElasticsearchIOTest.java   |  8 
 .../beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java  |  4 ++--
 .../beam/sdk/io/elasticsearch/ElasticsearchIOTestCommon.java |  2 +-
 .../apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java| 12 ++--
 .../src/test/java/org/apache/beam/sdk/io/avro/AvroIOIT.java  |  4 ++--
 .../java/org/apache/beam/sdk/io/gcp/spanner/OrderedCode.java |  2 +-
 .../org/apache/beam/sdk/io/gcp/bigtable/BigtableIOTest.java  |  2 +-
 .../beam/sdk/io/gcp/bigtable/BigtableServiceImplTest.java|  2 +-
 .../beam/sdk/io/hadoop/inputformat/HadoopInputFormatIO.java  |  2 +-
 .../sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java   |  2 +-
 .../main/java/org/apache/beam/sdk/nexmark/model/Event.java   |  2 +-
 53 files changed, 84 insertions(+), 83 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Updated] (BEAM-3735) Beam 2.3.0 release archetypes missing mobile gaming examples

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3735:

Affects Version/s: 2.3.0

> Beam 2.3.0 release archetypes missing mobile gaming examples
> 
>
> Key: BEAM-3735
> URL: https://issues.apache.org/jira/browse/BEAM-3735
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Affects Versions: 2.3.0
>Reporter: yifan zou
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.4.0
>
>
> We stopped copying the mobile gaming examples after merging Java8 examples to 
> "mian" Java examples. 
> [Here|https://github.com/apache/beam/pull/4479/files#diff-3e5600cc4b04a4a7f27d7ce10ac2994aL51]
>  is the PR. So, we're not able to run those pipelines via mvn archepyte: 
> generate.
> We need bring those examples back.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3735) Beam 2.3.0 release archetypes missing mobile gaming examples

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-3735:
---

Assignee: Jean-Baptiste Onofré  (was: Reuven Lax)

> Beam 2.3.0 release archetypes missing mobile gaming examples
> 
>
> Key: BEAM-3735
> URL: https://issues.apache.org/jira/browse/BEAM-3735
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Affects Versions: 2.3.0
>Reporter: yifan zou
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.4.0
>
>
> We stopped copying the mobile gaming examples after merging Java8 examples to 
> "mian" Java examples. 
> [Here|https://github.com/apache/beam/pull/4479/files#diff-3e5600cc4b04a4a7f27d7ce10ac2994aL51]
>  is the PR. So, we're not able to run those pipelines via mvn archepyte: 
> generate.
> We need bring those examples back.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3735) Beam 2.3.0 release archetypes missing mobile gaming examples

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-3735:

Fix Version/s: 2.4.0

> Beam 2.3.0 release archetypes missing mobile gaming examples
> 
>
> Key: BEAM-3735
> URL: https://issues.apache.org/jira/browse/BEAM-3735
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Affects Versions: 2.3.0
>Reporter: yifan zou
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.4.0
>
>
> We stopped copying the mobile gaming examples after merging Java8 examples to 
> "mian" Java examples. 
> [Here|https://github.com/apache/beam/pull/4479/files#diff-3e5600cc4b04a4a7f27d7ce10ac2994aL51]
>  is the PR. So, we're not able to run those pipelines via mvn archepyte: 
> generate.
> We need bring those examples back.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3736) Add SetUp() and TearDown() for CombineFns

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay updated BEAM-3736:
--
Component/s: beam-model

> Add SetUp() and TearDown() for CombineFns
> -
>
> Key: BEAM-3736
> URL: https://issues.apache.org/jira/browse/BEAM-3736
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model, sdk-py-core
>Reporter: Chuan Yu Foo
>Assignee: Ahmet Altay
>Priority: Major
>
> I have a CombineFn that has a large amount of state that needs to be loaded 
> once before it can add_input or merge_combiners (for example, the CombineFn 
> might load up a large lookup table used for combining). 
> Right now, to initialise this state, for each of the methods, I check if the 
> state has already been initialised, and if not, I initialise it. It would be 
> nice if CombineFn provided a SetUp() method that is called once to initialise 
> this state (and a corresponding TearDown() method to clean up this state if 
> necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1389

[Apache Jenkins Server]

See 


Changes:

[boyuanz] Add distribution counter implementation

[tgroh] Use Conccurrent Constructs in InMemoryArtifactStagerService

[tgroh] Add Environment Manager Interfaces

[iemejia] [BEAM-3632] Add missing partitioning parameter in WriteTables

[iemejia] [BEAM-3632] Add TableDestination.withTableReference and fix 
WriteTables

[robertwb] Add MultiMap side inputs to Python SDK.

--
[...truncated 70.98 KB...]
2018-02-23 00:16:18,493 fc9b3b55 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 00:16:36,866 fc9b3b55 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 00:16:40,745 fc9b3b55 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.86s,  CPU:0.44s,  MaxMemory:31620kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r57b093172857262f_0161c004d3c7_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r57b093172857262f_0161c004d3c7_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r57b093172857262f_0161c004d3c7_1 ... (0s) Current status: DONE   
2018-02-23 00:16:40,746 fc9b3b55 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 00:17:10,576 fc9b3b55 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 00:17:14,081 fc9b3b55 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.49s,  CPU:0.41s,  MaxMemory:31636kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r2bd61c5979eba14a_0161c0055772_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r2bd61c5979eba14a_0161c0055772_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r2bd61c5979eba14a_0161c0055772_1 ... (0s) Current status: DONE   
2018-02-23 00:17:14,081 fc9b3b55 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 00:17:31,443 fc9b3b55 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 00:17:34,828 fc9b3b55 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.37s,  CPU:0.38s,  MaxMemory:31596kb 
STDOUT: 

BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r350c82e40b36d4db_0161c005a8ec_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: Upload complete.Waiting on bqjob_r350c82e40b36d4db_0161c005a8ec_1 
... (0s) Current status: RUNNING
  Waiting on 
bqjob_r350c82e40b36d4db_0161c005a8ec_1 ... (0s) Current status: DONE   
2018-02-23 00:17:34,828 fc9b3b55 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-23 00:17:59,820 fc9b3b55 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-23 00:18:03,226 fc9b3b55 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:03.39s,  CPU:0.41s,  MaxMemory:31640kb 
STDOUT: 

BigQuery error in load operation: Error 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #189

[Apache Jenkins Server]

See 


Changes:

[boyuanz] Add distribution counter implementation

[tgroh] Use Conccurrent Constructs in InMemoryArtifactStagerService

[tgroh] Add Environment Manager Interfaces

[iemejia] [BEAM-3632] Add missing partitioning parameter in WriteTables

[iemejia] [BEAM-3632] Add TableDestination.withTableReference and fix 
WriteTables

[robertwb] Add MultiMap side inputs to Python SDK.

--
[...truncated 16.98 KB...]
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 
proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: hdfs, apache-beam
  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins8199194438469078266.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.textioit_pkb_results 
--temp_dir= 
--official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TextIOIT/189/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-23 00:00:48,517 271f8785 MainThread INFO Verbose logging to: 

2018-02-23 00:00:48,517 271f8785 MainThread INFO PerfKitBenchmarker 
version: v1.12.0-362-g15db1b8
2018-02-23 00:00:48,518 271f8785 MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_it_module=sdks/java/io/file-based-io-tests
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TextIOIT/189/]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.textioit_pkb_results
--official
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-02-23 00:00:48,865 271f8785 MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-02-23 00:00:48,865 271f8785 MainThread INFO Initializing the edw 
service decoder
2018-02-23 00:00:48,972 271f8785 MainThread beam_integration_benchmark(1/1) 
INFO Provisioning resources for benchmark beam_integration_benchmark
2018-02-23 00:00:48,974 271f8785 MainThread beam_integration_benchmark(1/1) 
INFO Preparing benchmark beam_integration_benchmark
2018-02-23 00:00:48,974 271f8785 MainThread beam_integration_benchmark(1/1) 
INFO Running: git clone https://github.com/apache/beam.git
2018-02-23 00:00:57,094 271f8785 MainThread beam_integration_benchmark(1/1) 
INFO Running benchmark beam_integration_benchmark
2018-02-23 00:00:57,099 271f8785 MainThread beam_integration_benchmark(1/1) 
INFO Running: /home/jenkins/tools/maven/latest/bin/mvn -e verify 

Jenkins build is back to normal : beam_PerformanceTests_JDBC #248

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #945

[Apache Jenkins Server]

See 


Changes:

[boyuanz] Add distribution counter implementation

[tgroh] Use Conccurrent Constructs in InMemoryArtifactStagerService

[tgroh] Add Environment Manager Interfaces

[iemejia] [BEAM-3632] Add missing partitioning parameter in WriteTables

[iemejia] [BEAM-3632] Add TableDestination.withTableReference and fix 
WriteTables

[robertwb] Add MultiMap side inputs to Python SDK.

--
[...truncated 896 B...]
Checking out Revision f4b65d31af60107d78d23ff4dc5052aab16c5468 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f f4b65d31af60107d78d23ff4dc5052aab16c5468
Commit message: "Merge pull request #4613 Add distribution counter 
implementation"
 > git rev-list --no-walk bd49a61611ccfc1887aef1ac1c2367e836849ab7 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins43704951523790261.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1226866990066254839.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins235986849346728818.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1289940183043012216.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6187917068112202898.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3324836302582336063.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 

[jira] [Created] (BEAM-3737) Key-aware batching function

[Chuan Yu Foo (JIRA)]

Chuan Yu Foo created BEAM-3737:
--

 Summary: Key-aware batching function
 Key: BEAM-3737
 URL: https://issues.apache.org/jira/browse/BEAM-3737
 Project: Beam
  Issue Type: New Feature
  Components: beam-model
Reporter: Chuan Yu Foo
Assignee: Kenneth Knowles


I have a CombineFn for which add_input has very large overhead. I would like to 
batch the incoming elements into a large batch before each call to add_input to 
reduce this overhead. In other words, I would like to do something like: 

{{elements | GroupByKey() | BatchElements() | CombineValues(MyCombineFn())}}

Unfortunately, BatchElements is not key-aware, and can't be used after a 
GroupByKey to batch elements per key. I'm working around this by doing the 
batching within CombineValues, which makes the CombineFn rather messy. It would 
be nice if there were a key-aware BatchElements transform which could be used 
in this context.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PerformanceTests_Compressed_TextIOIT #173

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3736) Add SetUp() and TearDown() for CombineFns

[Chuan Yu Foo (JIRA)]

Chuan Yu Foo created BEAM-3736:
--

 Summary: Add SetUp() and TearDown() for CombineFns
 Key: BEAM-3736
 URL: https://issues.apache.org/jira/browse/BEAM-3736
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Chuan Yu Foo
Assignee: Ahmet Altay


I have a CombineFn that has a large amount of state that needs to be loaded 
once before it can add_input or merge_combiners (for example, the CombineFn 
might load up a large lookup table used for combining). 

Right now, to initialise this state, for each of the methods, I check if the 
state has already been initialised, and if not, I initialise it. It would be 
nice if CombineFn provided a SetUp() method that is called once to initialise 
this state (and a corresponding TearDown() method to clean up this state if 
necessary).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3735) Beam 2.3.0 release archetypes missing mobile gaming examples

[yifan zou (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

yifan zou updated BEAM-3735:

Summary: Beam 2.3.0 release archetypes missing mobile gaming examples  
(was: Mobile gaming examples are not copied to maven archetype)

> Beam 2.3.0 release archetypes missing mobile gaming examples
> 
>
> Key: BEAM-3735
> URL: https://issues.apache.org/jira/browse/BEAM-3735
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java
>Reporter: yifan zou
>Assignee: Reuven Lax
>Priority: Major
>
> We stopped copying the mobile gaming examples after merging Java8 examples to 
> "mian" Java examples. 
> [Here|https://github.com/apache/beam/pull/4479/files#diff-3e5600cc4b04a4a7f27d7ce10ac2994aL51]
>  is the PR. So, we're not able to run those pipelines via mvn archepyte: 
> generate.
> We need bring those examples back.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 10ec0c0d89487106100d1273d49003c329d49bed
Author: Mergebot 
AuthorDate: Thu Feb 22 14:19:11 2018 -0800

Prepare repository for deployment.
---
 content/documentation/runners/dataflow/index.html | 50 +++
 1 file changed, 50 insertions(+)

diff --git a/content/documentation/runners/dataflow/index.html 
b/content/documentation/runners/dataflow/index.html
index 527b9b1..1ad92b1 100644
--- a/content/documentation/runners/dataflow/index.html
+++ b/content/documentation/runners/dataflow/index.html
@@ -105,6 +105,7 @@
   Cloud Dataflow 
Runner prerequisites and setup
 
   Specify your dependency
+  Self executing JAR
   Authentication
 
   
@@ -196,6 +197,55 @@ Pub/Sub, or Cloud Datastore) if you use them in your 
pipeline code.
 
 This section is not applicable to the Beam SDK 
for Python.
 
+Self executing JAR
+
+This section is not applicable to the Beam SDK for 
Python.
+
+In some cases, such as starting a pipeline using a 
scheduler such as https://airflow.apache.org;>Apache AirFlow, you 
must have a self-contained application. You can pack a self-executing JAR by 
explicitly adding the following dependency on the Project section of your 
pom.xml, in addition to the adding existing dependency shown in the previous 
section.
+
+dependency
+groupIdorg.apache.beam/groupId
+artifactIdbeam-runners-google-cloud-dataflow-java/artifactId
+version${beam.version}/version
+scoperuntime/scope
+/dependency
+
+
+
+Then, add the mainClass name in the Maven JAR 
plugin.
+
+plugin
+  groupIdorg.apache.maven.plugins/groupId
+  artifactIdmaven-jar-plugin/artifactId
+  version${maven-jar-plugin.version}/version
+  configuration
+archive
+  manifest
+addClasspathtrue/addClasspath
+classpathPrefixlib//classpathPrefix
+mainClassYOUR_MAIN_CLASS_NAME/mainClass
+  /manifest
+/archive
+  /configuration
+/plugin
+
+
+
+After running mvn package, run ls 
target and you should see (assuming your artifactId is beam-examples and the version is 1.0.0) the 
following output.
+
+beam-examples-bundled-1.0.0.jar
+
+
+
+To run the self-executing JAR on Cloud Dataflow, use 
the following command.
+
+java -jar target/beam-examples-bundled-1.0.0.jar--runner=DataflowRunner \
+  --project=YOUR_GCP_PROJECT_ID \
+  --tempLocation=gs://YOUR_GCS_BUCKET/temp/
+
+
+
 Authentication
 
 Before running your pipeline, you must authenticate with the Google Cloud 
Platform. Run the following command to get https://developers.google.com/identity/protocols/application-default-credentials;>Application
 Default Credentials.

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch asf-site updated (6ec9c95 -> 10ec0c0)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 6ec9c95  Prepare repository for deployment.
 add cb9cbf7  Add procedure to create a self executing Jar for DataFlow
 add 283f624  This closes #387
 new 10ec0c0  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/runners/dataflow/index.html | 50 +
 src/documentation/runners/dataflow.md | 54 +++
 2 files changed, 104 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Add procedure to create a self executing Jar for DataFlow

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit cb9cbf7cc870b366366ef5e3ad46517a36500823
Author: Lorenzo Caggioni 
AuthorDate: Tue Jan 30 14:48:04 2018 +0100

Add procedure to create a self executing Jar for DataFlow
---
 src/documentation/runners/dataflow.md | 54 +++
 1 file changed, 54 insertions(+)

diff --git a/src/documentation/runners/dataflow.md 
b/src/documentation/runners/dataflow.md
index 9be4ce3..2ff8198 100644
--- a/src/documentation/runners/dataflow.md
+++ b/src/documentation/runners/dataflow.md
@@ -64,6 +64,60 @@ For more information, see the *Before you begin* section of 
the [Cloud Dataflow
 
 This section is not applicable to the Beam SDK for 
Python.
 
+### Self executing JAR
+
+{:.language-py}
+This section is not applicable to the Beam SDK for Python.
+
+{:.language-java}
+In some cases, such as starting a pipeline using a scheduler such as [Apache 
AirFlow](https://airflow.apache.org), you must have a self-contained 
application. You can pack a self-executing JAR by explicitly adding the 
following dependency on the Project section of your pom.xml, in addition to the 
adding existing dependency shown in the previous section.
+
+```java
+
+org.apache.beam
+beam-runners-google-cloud-dataflow-java
+${beam.version}
+runtime
+
+```
+
+{:.language-java}
+Then, add the mainClass name in the Maven JAR plugin.
+
+```java
+
+  org.apache.maven.plugins
+  maven-jar-plugin
+  ${maven-jar-plugin.version}
+  
+
+  
+true
+lib/
+YOUR_MAIN_CLASS_NAME
+  
+
+  
+
+```
+
+{:.language-java}
+After running mvn package, run ls target and you 
should see (assuming your artifactId is `beam-examples` and the version is 
1.0.0) the following output.
+
+```java
+beam-examples-bundled-1.0.0.jar
+```
+
+{:.language-java}
+To run the self-executing JAR on Cloud Dataflow, use the following command.
+
+```java
+java -jar target/beam-examples-bundled-1.0.0.jar \
+  --runner=DataflowRunner \
+  --project= \
+  --tempLocation=gs:///temp/
+```
+
 ### Authentication
 
 Before running your pipeline, you must authenticate with the Google Cloud 
Platform. Run the following command to get [Application Default 
Credentials](https://developers.google.com/identity/protocols/application-default-credentials).

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #387

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 283f624c1d4ae87da686e9b45800a7c26a166985
Merge: 6ec9c95 cb9cbf7
Author: Mergebot 
AuthorDate: Thu Feb 22 14:12:51 2018 -0800

This closes #387

 src/documentation/runners/dataflow.md | 54 +++
 1 file changed, 54 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Created] (BEAM-3735) Mobile gaming examples are not copied to maven archetype

[yifan zou (JIRA)]

yifan zou created BEAM-3735:
---

 Summary: Mobile gaming examples are not copied to maven archetype
 Key: BEAM-3735
 URL: https://issues.apache.org/jira/browse/BEAM-3735
 Project: Beam
  Issue Type: Bug
  Components: examples-java
Reporter: yifan zou
Assignee: Reuven Lax


We stopped copying the mobile gaming examples after merging Java8 examples to 
"mian" Java examples. 
[Here|https://github.com/apache/beam/pull/4479/files#diff-3e5600cc4b04a4a7f27d7ce10ac2994aL51]
 is the PR. So, we're not able to run those pipelines via mvn archepyte: 
generate.

We need bring those examples back.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4722 from Add MultiMap side inputs to Python SDK.

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 67109452fed9bd48155c2858b9065d0ce6d1da43
Merge: a53968d 769351f
Author: Robert Bradshaw 
AuthorDate: Thu Feb 22 13:10:00 2018 -0800

Merge pull request #4722 from Add MultiMap side inputs to Python SDK.

Add MultiMap side inputs to Python SDK.

 sdks/python/apache_beam/coders/coder_impl.pxd  |  2 +
 sdks/python/apache_beam/coders/coder_impl.py   |  8 
 sdks/python/apache_beam/pvalue.py  | 28 ++
 .../runners/portability/fn_api_runner.py   | 31 +++
 .../runners/portability/fn_api_runner_test.py  |  9 +
 .../apache_beam/runners/worker/bundle_processor.py | 44 +++---
 6 files changed, 109 insertions(+), 13 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch master updated (a53968d -> 6710945)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a53968d  Merge pull request #4728: [BEAM-3632] Table partioning in 
DynamicDestination is lost with project is not set in Table Destination
 add 769351f  Add MultiMap side inputs to Python SDK.
 new 6710945  Merge pull request #4722 from Add MultiMap side inputs to 
Python SDK.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/coders/coder_impl.pxd  |  2 +
 sdks/python/apache_beam/coders/coder_impl.py   |  8 
 sdks/python/apache_beam/pvalue.py  | 28 ++
 .../runners/portability/fn_api_runner.py   | 31 +++
 .../runners/portability/fn_api_runner_test.py  |  9 +
 .../apache_beam/runners/worker/bundle_processor.py | 44 +++---
 6 files changed, 109 insertions(+), 13 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #4253

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3632) Table partioning in DynamicDestination is lost with project is not set in Table Destination

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3632.

Resolution: Fixed

> Table partioning in DynamicDestination is lost with project is not set in 
> Table Destination
> ---
>
> Key: BEAM-3632
> URL: https://issues.apache.org/jira/browse/BEAM-3632
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-gcp
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Blaquiere Guillaume
>Assignee: Blaquiere Guillaume
>Priority: Major
> Fix For: 2.4.0
>
>   Original Estimate: 10m
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> In the following class 
> https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/*WriteTables.java*
> line 140
> When the project is not set in table name, a new TableDestination is created 
> but all the time partitioning parameter is lost.
> Fix is very easy. Add a get time partitioning on the new creation:
>  tableDestination = new TableDestination(tableReference, 
> tableDestination.getTableDescription()*{color:#ff},tableDestination{color}**{color:#ff}.getTimePartitioning(){color}*);
>   
>  Should be enought :P



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (2fbc195 -> a53968d)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2fbc195  Merge pull request #4589: Add Environment Manager Interfaces
 add ce3147d  [BEAM-3632] Add missing partitioning parameter in WriteTables
 add 223770b  [BEAM-3632] Add TableDestination.withTableReference and fix 
WriteTables
 new a53968d  Merge pull request #4728: [BEAM-3632] Table partioning in 
DynamicDestination is lost with project is not set in Table Destination

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java |  3 +--
 .../org/apache/beam/sdk/io/gcp/bigquery/TableDestination.java | 11 ++-
 .../java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java |  3 +--
 3 files changed, 12 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] 01/01: Merge pull request #4728: [BEAM-3632] Table partioning in DynamicDestination is lost with project is not set in Table Destination

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a53968d7bf7542f556f1da90ec1a683d1cd5d954
Merge: 2fbc195 223770b
Author: Ismaël Mejía 
AuthorDate: Thu Feb 22 21:17:13 2018 +0100

Merge pull request #4728: [BEAM-3632] Table partioning in 
DynamicDestination is lost with project is not set in Table Destination

 .../org/apache/beam/sdk/io/gcp/bigquery/CreateTables.java |  3 +--
 .../org/apache/beam/sdk/io/gcp/bigquery/TableDestination.java | 11 ++-
 .../java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java |  3 +--
 3 files changed, 12 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #6029

[Apache Jenkins Server]

See 

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 6ec9c9583ae84f4735d7f44f4ef487e6c0b5b8a2
Author: Mergebot 
AuthorDate: Thu Feb 22 11:12:39 2018 -0800

Prepare repository for deployment.
---
 content/get-started/wordcount-example/index.html | 43 +++-
 1 file changed, 19 insertions(+), 24 deletions(-)

diff --git a/content/get-started/wordcount-example/index.html 
b/content/get-started/wordcount-example/index.html
index bbe6fad..eddefd4 100644
--- a/content/get-started/wordcount-example/index.html
+++ b/content/get-started/wordcount-example/index.html
@@ -298,7 +298,11 @@ often, command-line arguments are used to set Pi
 example, your pipeline executes locally using the DirectRunner. In the next
 sections, we will specify the pipeline’s runner.
 
- 
PipelineOptions options = PipelineOptionsFactory.create();
+ 
// Create a PipelineOptions object. This object lets us set 
various execution
+ // options for our pipeline, such as the runner you wish to 
use. This example
+ // will run with the DirectRunner by default, based on the 
class path configured
+ // in its dependencies.
+ PipelineOptions options = PipelineOptionsFactory.create();
 
 // In order to run your pipeline, you need to make 
following runner specific changes:
 //
@@ -372,25 +376,19 @@ data stored in a publicly accessible Google Cloud Storage 
bucket (“gs://”).<
 
   
   
-A ParDo
-transform that invokes a DoFn (defined 
in-line as an anonymous class) on
+This transform splits the lines in PCollection, where each 
element
+is an individual word in Shakespeare's collected texts.
+As an alternative, it would have been possible to use a 
+[ParDo](/documentation/programming-guide/#pardo)
+transform that invokes a `DoFn` (defined in-line as an anonymous class) on
 each element that tokenizes the text lines into individual words. The input
-for this transform is the PCollection 
of text lines generated by the
-previous TextIO.Read transform. The 
ParDo transform outputs a new
-PCollection, where each element 
represents an individual word in the text.
+for this transform is the `PCollection` of text lines generated by the
+previous `TextIO.Read` transform. The `ParDo` transform outputs a new
+`PCollection`, where each element represents an individual word in the 
text.
 
-.apply("ExtractWords", ParDo.of(new DoFnString, 
String@ProcessElement
-public void processElement(ProcessContext c) {
-// \p{L} denotes the category of Unicode 
letters,
-// so this pattern will match on everything that is 
not a letter.
-for (String word : 
c.element().split("[^\\p{L}]+")) {
-if (!word.isEmpty()) {
-c.output(word);
-}
-}
-}
-}))
+ 
   .apply("ExtractWords", 
FlatMapElements
+.into(TypeDescriptors.strings())
+.via((String word) - 
Arrays.asList(word.split("[^\\p{L}]+"
 
 
 
@@ -428,12 +426,9 @@ file.
 simple ParDo. For each element in the 
input PCollection, the map
 transform applies a function that produces exactly one output element.
 
-.apply("FormatResults", MapElements.via(new SimpleFunctionKVString [...]
-@Override
-public String apply(KVString, 
Long input) {
-return input.getKey() + ": " + input.getValue();
-}
-}))
+.apply("FormatResults", MapElements
+.into(TypeDescriptors.strings())
+.via((KVString, Long wordCount) - 
wordCount.getKey() + 
": "  [...]
 
 
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch asf-site updated (d85231c -> 6ec9c95)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from d85231c  Prepare repository for deployment.
 add f09f363  [BEAM-3430] Update workdcount example for Java8 for Java SDK
 add 9132245  This closes #379
 new 6ec9c95  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/get-started/wordcount-example/index.html | 43 +++-
 src/get-started/wordcount-example.md | 33 --
 2 files changed, 33 insertions(+), 43 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #6028

[Apache Jenkins Server]

See 

[beam-site] 02/02: This closes #379

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 9132245c286ce4e5cd45fe55c1ccf17094a93699
Merge: d85231c f09f363
Author: Mergebot 
AuthorDate: Thu Feb 22 11:07:03 2018 -0800

This closes #379

 src/get-started/wordcount-example.md | 33 ++---
 1 file changed, 14 insertions(+), 19 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: [BEAM-3430] Update workdcount example for Java8 for Java SDK

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit f09f363e744b4fdfb5472a2d2c4d27e2b9338e0c
Author: Jean-Baptiste Onofré 
AuthorDate: Sun Jan 28 14:50:20 2018 +0100

[BEAM-3430] Update workdcount example for Java8 for Java SDK
---
 src/get-started/wordcount-example.md | 33 ++---
 1 file changed, 14 insertions(+), 19 deletions(-)

diff --git a/src/get-started/wordcount-example.md 
b/src/get-started/wordcount-example.md
index 408ce5b..9947330 100644
--- a/src/get-started/wordcount-example.md
+++ b/src/get-started/wordcount-example.md
@@ -126,6 +126,10 @@ example, your pipeline executes locally using the 
`DirectRunner`. In the next
 sections, we will specify the pipeline's runner.
 
 ```java
+ // Create a PipelineOptions object. This object lets us set various execution
+ // options for our pipeline, such as the runner you wish to use. This example
+ // will run with the DirectRunner by default, based on the class path 
configured
+ // in its dependencies.
  PipelineOptions options = PipelineOptionsFactory.create();
 
 // In order to run your pipeline, you need to make following runner 
specific changes:
@@ -190,7 +194,10 @@ The MinimalWordCount pipeline contains five transforms:
 {% github_sample 
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets.py 
tag:examples_wordcount_minimal_read
 %}```
 
-2.  A [ParDo]({{ site.baseurl }}/documentation/programming-guide/#pardo)
+2.  This transform splits the lines in PCollection, where each element
+is an individual word in Shakespeare's collected texts.
+As an alternative, it would have been possible to use a 
+[ParDo]({{ site.baseurl }}/documentation/programming-guide/#pardo)
 transform that invokes a `DoFn` (defined in-line as an anonymous class) on
 each element that tokenizes the text lines into individual words. The input
 for this transform is the `PCollection` of text lines generated by the
@@ -198,18 +205,9 @@ The MinimalWordCount pipeline contains five transforms:
 `PCollection`, where each element represents an individual word in the 
text.
 
 ```java
-.apply("ExtractWords", ParDo.of(new DoFn() {
-@ProcessElement
-public void processElement(ProcessContext c) {
-// \p{L} denotes the category of Unicode letters,
-// so this pattern will match on everything that is not a letter.
-for (String word : c.element().split("[^\\p{L}]+")) {
-if (!word.isEmpty()) {
-c.output(word);
-}
-}
-}
-}))
+.apply("ExtractWords", FlatMapElements
+.into(TypeDescriptors.strings())
+.via((String word) -> Arrays.asList(word.split("[^\\p{L}]+"
 ```
 
 ```py
@@ -245,12 +243,9 @@ The MinimalWordCount pipeline contains five transforms:
 transform applies a function that produces exactly one output element.
 
 ```java
-.apply("FormatResults", MapElements.via(new SimpleFunction, String>() {
-@Override
-public String apply(KV input) {
-return input.getKey() + ": " + input.getValue();
-}
-}))
+.apply("FormatResults", MapElements
+.into(TypeDescriptors.strings())
+.via((KV wordCount) -> wordCount.getKey() + ": " + 
wordCount.getValue()))
 ```
 
 ```py

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (2eabb2d -> 9132245)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 2eabb2d  This closes #382
 add d85231c  Prepare repository for deployment.
 new f09f363  [BEAM-3430] Update workdcount example for Java8 for Java SDK
 new 9132245  This closes #379

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/release-guide/index.html | 10 +
 src/get-started/wordcount-example.md| 33 -
 2 files changed, 24 insertions(+), 19 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Updated] (BEAM-3734) Add performance tests for XmlIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Łukasz Gajowy updated BEAM-3734:

Issue Type: Test  (was: Improvement)

> Add performance tests for XmlIO
> ---
>
> Key: BEAM-3734
> URL: https://issues.apache.org/jira/browse/BEAM-3734
> Project: Beam
>  Issue Type: Test
>  Components: testing
>Reporter: Łukasz Gajowy
>Assignee: Łukasz Gajowy
>Priority: Major
>
> We should add a large scale performance test for XmlIO as described in 
> documentation [1].
> [1] 
> https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] branch asf-site updated (c8413e3 -> d85231c)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from c8413e3  Prepare repository for deployment.
 add ba39536  Add nightly snapshot build to the release guide
 add 2eabb2d  This closes #382
 new d85231c  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/release-guide/index.html | 10 ++
 src/contribute/release-guide.md | 28 ++--
 2 files changed, 28 insertions(+), 10 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit d85231c7ef34d53834121670c8936abed8bc248e
Author: Mergebot 
AuthorDate: Thu Feb 22 11:02:38 2018 -0800

Prepare repository for deployment.
---
 content/contribute/release-guide/index.html | 10 ++
 1 file changed, 10 insertions(+)

diff --git a/content/contribute/release-guide/index.html 
b/content/contribute/release-guide/index.html
index 2bcaf75..8ba599b 100644
--- a/content/contribute/release-guide/index.html
+++ b/content/contribute/release-guide/index.html
@@ -163,6 +163,7 @@
   Create a release branch
   Update the Python SDK 
version
   Update release 
specific configurations
+  Start a snapshot build
   Checklist to 
proceed to the next step
 
   
@@ -236,6 +237,7 @@
   Create a release branch
   Update the Python SDK 
version
   Update release 
specific configurations
+  Start a snapshot build
   Checklist to proceed 
to the next step
 
   
@@ -541,6 +543,13 @@ DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
 https://github.com/apache/beam/commit/f572328ce23e70adee8001e3d10f1479bd9a380d;>example
 
 
+Start a snapshot build
+
+Start a build of https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_NightlySnapshot/;>the
 nightly snapshot.
+Some processes, including our archetype tests, rely on having a live SNAPSHOT 
of the current version
+from the master branch. Once the 
release branch is cut, these SNAPSHOT versions are no longer found,
+so builds will be broken until a new snapshot is available.
+
 Checklist to proceed to the 
next step
 
 
@@ -554,6 +563,7 @@ DEVELOPMENT_VERSION="${NEXT_VERSION}-SNAPSHOT"
   Combined javadoc has the appropriate contents.
   Release branch has been created
   Originating branch has the version information updated to the new 
version
+  Nightly snapshot is in progress (do revisit it continually)
 
 
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #382

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 2eabb2d9c5d6befd85c264c0ce5619ed5c955acb
Merge: c8413e3 ba39536
Author: Mergebot 
AuthorDate: Thu Feb 22 10:57:11 2018 -0800

This closes #382

 src/contribute/release-guide.md | 28 ++--
 1 file changed, 18 insertions(+), 10 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (9488a81 -> 2eabb2d)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 9488a81  This closes #385
 add c8413e3  Prepare repository for deployment.
 new ba39536  Add nightly snapshot build to the release guide
 new 2eabb2d  This closes #382

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../sdks/python-type-safety/index.html |  6 +++--
 src/contribute/release-guide.md| 28 ++
 2 files changed, 22 insertions(+), 12 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Add nightly snapshot build to the release guide

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit ba395361f1be06d26eb70d5c4423c83f803ea11f
Author: Kenn Knowles 
AuthorDate: Tue Jan 30 19:57:45 2018 -0800

Add nightly snapshot build to the release guide
---
 src/contribute/release-guide.md | 28 ++--
 1 file changed, 18 insertions(+), 10 deletions(-)

diff --git a/src/contribute/release-guide.md b/src/contribute/release-guide.md
index ea6..7feeba7 100644
--- a/src/contribute/release-guide.md
+++ b/src/contribute/release-guide.md
@@ -226,21 +226,29 @@ In the release branch, update the Python SDK version to 
the release version (e.g
 
 1. Update archetypes:

[example](https://github.com/apache/beam/commit/d375cfa126fd7be9c34f39c2b9b856f324bf)
-1. Update runner specific configurations:
+2. Update runner specific configurations:

[example](https://github.com/apache/beam/commit/f572328ce23e70adee8001e3d10f1479bd9a380d)
 
+### Start a snapshot build
+
+Start a build of [the nightly 
snapshot](https://builds.apache.org/view/A-D/view/Beam/job/beam_Release_NightlySnapshot/).
+Some processes, including our archetype tests, rely on having a live SNAPSHOT 
of the current version
+from the `master` branch. Once the release branch is cut, these SNAPSHOT 
versions are no longer found,
+so builds will be broken until a new snapshot is available.
+
 ### Checklist to proceed to the next step
 
 1. Release Manager’s GPG key is published to `dist.apache.org`
-1. Release Manager’s GPG key is configured in `git` configuration
-1. Release Manager has `org.apache.beam` listed under `Staging Profiles` in 
Nexus
-1. Release Manager’s Nexus User Token is configured in `settings.xml`
-1. JIRA release item for the subsequent release has been created
-1. There are no release blocking JIRA issues
-1. Release Notes in JIRA have been audited and adjusted
-1. Combined javadoc has the appropriate contents.
-1. Release branch has been created
-1. Originating branch has the version information updated to the new version
+2. Release Manager’s GPG key is configured in `git` configuration
+3. Release Manager has `org.apache.beam` listed under `Staging Profiles` in 
Nexus
+4. Release Manager’s Nexus User Token is configured in `settings.xml`
+5. JIRA release item for the subsequent release has been created
+6. There are no release blocking JIRA issues
+7. Release Notes in JIRA have been audited and adjusted
+8. Combined javadoc has the appropriate contents.
+9. Release branch has been created
+10. Originating branch has the version information updated to the new version
+11. Nightly snapshot is in progress (do revisit it continually)
 
 **
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c8413e3afc7ea65c6e9c8baf01557dc15b6e7089
Author: Mergebot 
AuthorDate: Thu Feb 22 10:53:44 2018 -0800

Prepare repository for deployment.
---
 content/documentation/sdks/python-type-safety/index.html | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/content/documentation/sdks/python-type-safety/index.html 
b/content/documentation/sdks/python-type-safety/index.html
index e116b24..1f3ef16 100644
--- a/content/documentation/sdks/python-type-safety/index.html
+++ b/content/documentation/sdks/python-type-safety/index.html
@@ -281,19 +281,21 @@ words_with_lens = words | MyTransform()
 
 In addition to using type hints for type checking at pipeline construction, 
you can enable runtime type checking to check that actual elements satisfy the 
declared type constraints during pipeline execution.
 
-For example, the following code would pass at both pipeline construction 
and runtime.
+For example, the following pipeline emits elements of the wrong type. 
Depending on the runner implementation, its execution may or may not fail at 
runtime.
 
 p | 
beam.Create(['a']) | beam.Map(lambda x: 3).with_output_types(str)
 
 
 
-However, if you enable runtime type checking, the code passes at pipeline 
construction and fails at runtime. To enable runtime type checking, set the 
pipeline option runtime_type_check to 
True.
+However, if you enable runtime type checking, the code is guaranteed to 
fail at runtime. To enable runtime type checking, set the pipeline option runtime_type_check to True.
 
 p | 
beam.Create(['a']) | beam.Map(lambda x: 3).with_output_types(str)
 p.run()
 
 
 
+Note that because runtime type checks are done for each PCollection element, enabling this feature may 
incur a significant performance penalty. It is therefore recommended that 
runtime type checks are disabled for production pipelines.
+
 Use of Type Hints in Coders
 
 When your pipeline reads, writes, or otherwise materializes its data, the 
elements in your PCollection need to be 
encoded and decoded to and from byte strings. Byte strings are used for 
intermediate storage, for comparing keys in GroupByKey operations, and for reading from 
sources and writing to sinks.

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch asf-site updated (4ffc4bc -> c8413e3)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 4ffc4bc  Prepare repository for deployment.
 add 1781b86  Clarify runtime type checking documentation
 add 9488a81  This closes #385
 new c8413e3  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/sdks/python-type-safety/index.html | 6 --
 src/documentation/sdks/python-type-safety.md | 6 --
 2 files changed, 8 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Created] (BEAM-3734) Add performance tests for XmlIO

[JIRA]

Łukasz Gajowy created BEAM-3734:
---

 Summary: Add performance tests for XmlIO
 Key: BEAM-3734
 URL: https://issues.apache.org/jira/browse/BEAM-3734
 Project: Beam
  Issue Type: Improvement
  Components: testing
Reporter: Łukasz Gajowy
Assignee: Łukasz Gajowy


We should add a large scale performance test for XmlIO as described in 
documentation [1].

[1] 
https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 01/02: Clarify runtime type checking documentation

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 1781b86778bed6fb9bfb72465128b8b09785c861
Author: Charles Chen 
AuthorDate: Mon Feb 5 16:28:03 2018 -0800

Clarify runtime type checking documentation
---
 src/documentation/sdks/python-type-safety.md | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/src/documentation/sdks/python-type-safety.md 
b/src/documentation/sdks/python-type-safety.md
index ae8fc09..3962fe1 100644
--- a/src/documentation/sdks/python-type-safety.md
+++ b/src/documentation/sdks/python-type-safety.md
@@ -106,16 +106,18 @@ The following are special type hints that don't 
correspond to a class, but rathe
 
 In addition to using type hints for type checking at pipeline construction, 
you can enable runtime type checking to check that actual elements satisfy the 
declared type constraints during pipeline execution.
 
-For example, the following code would pass at both pipeline construction and 
runtime.
+For example, the following pipeline emits elements of the wrong type. 
Depending on the runner implementation, its execution may or may not fail at 
runtime.
 
 ```
 {% github_sample 
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py
 tag:type_hints_runtime_off %}```
 
-However, if you enable runtime type checking, the code passes at pipeline 
construction and fails at runtime. To enable runtime type checking, set the 
pipeline option `runtime_type_check` to `True`.
+However, if you enable runtime type checking, the code is guaranteed to fail 
at runtime. To enable runtime type checking, set the pipeline option 
`runtime_type_check` to `True`.
 
 ```
 {% github_sample 
/apache/beam/blob/master/sdks/python/apache_beam/examples/snippets/snippets_test.py
 tag:type_hints_runtime_on %}```
 
+Note that because runtime type checks are done for each `PCollection` element, 
enabling this feature may incur a significant performance penalty. It is 
therefore recommended that runtime type checks are disabled for production 
pipelines.
+
 ## Use of Type Hints in Coders
 
 When your pipeline reads, writes, or otherwise materializes its data, the 
elements in your `PCollection` need to be encoded and decoded to and from byte 
strings. Byte strings are used for intermediate storage, for comparing keys in 
`GroupByKey` operations, and for reading from sources and writing to sinks.

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (f238648 -> 9488a81)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from f238648  This closes #392
 add 4ffc4bc  Prepare repository for deployment.
 new 1781b86  Clarify runtime type checking documentation
 new 9488a81  This closes #385

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/programming-guide/index.html | 10 +-
 src/documentation/sdks/python-type-safety.md   |  6 --
 2 files changed, 9 insertions(+), 7 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #385

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 9488a813a1beec84b94cbffaf0f51b2f892548cf
Merge: 4ffc4bc 1781b86
Author: Mergebot 
AuthorDate: Thu Feb 22 10:46:20 2018 -0800

This closes #385

 src/documentation/sdks/python-type-safety.md | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[jira] [Resolved] (BEAM-3273) ArtifactServiceStagerTest flaky

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-3273.
---
   Resolution: Fixed
Fix Version/s: Not applicable

> ArtifactServiceStagerTest flaky
> ---
>
> Key: BEAM-3273
> URL: https://issues.apache.org/jira/browse/BEAM-3273
> Project: Beam
>  Issue Type: Bug
>  Components: runner-core
>Reporter: Eugene Kirpichov
>Assignee: Thomas Groh
>Priority: Major
>  Labels: flake
> Fix For: Not applicable
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Example failing build: 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/2891/consoleFull
> {code}
> 2017-11-29T04:21:25.890 [ERROR] Tests run: 2, Failures: 1, Errors: 0, 
> Skipped: 0, Time elapsed: 0.169 s <<< FAILURE! - in 
> org.apache.beam.runners.core.construction.ArtifactServiceStagerTest
> 2017-11-29T04:21:25.891 [ERROR] 
> testStagingMultipleFiles(org.apache.beam.runners.core.construction.ArtifactServiceStagerTest)
>   Time elapsed: 0.027 s  <<< FAILURE!
> java.lang.AssertionError: 
> All of the files contents should be staged
> Expected: a collection with size <3>
>  but: collection size was <2>
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.junit.Assert.assertThat(Assert.java:956)
>   at 
> org.apache.beam.runners.core.construction.ArtifactServiceStagerTest.testStagingMultipleFiles(ArtifactServiceStagerTest.java:139)
> {code}
> I didn't try repeatedly running this test myself, just bringing to your 
> attention.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4252

[Apache Jenkins Server]

See 

[beam-site] branch asf-site updated (1f4014c -> 4ffc4bc)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 1f4014c  Prepare repository for deployment.
 add 8e33942  Java variable names should be in CamelCase
 add f238648  This closes #392
 new 4ffc4bc  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/documentation/programming-guide/index.html | 10 +-
 src/documentation/programming-guide.md | 10 +-
 2 files changed, 10 insertions(+), 10 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 4ffc4bc0fc39b25711e9e004bb278900aea8f026
Author: Mergebot 
AuthorDate: Thu Feb 22 10:30:35 2018 -0800

Prepare repository for deployment.
---
 content/documentation/programming-guide/index.html | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/content/documentation/programming-guide/index.html 
b/content/documentation/programming-guide/index.html
index 23d0a20..0b6a64c 100644
--- a/content/documentation/programming-guide/index.html
+++ b/content/documentation/programming-guide/index.html
@@ -2877,7 +2877,7 @@ for more information.
 into fixed windows, each one minute in length:
 
 
PCollectionString items 
= ...;
-PCollectionString fixed_windowed_items = items.apply(
+PCollectionString fixedWindowedItems = items.apply(
 Window.Stringinto(FixedWindows.of(Duration.standardMinutes(1;
 
 
@@ -2895,7 +2895,7 @@ into sliding time windows. Each window is 30 minutes in 
length, and a new window
 begins every five seconds:
 
 
PCollectionString items 
= ...;
-PCollectionString sliding_windowed_items = items.apply(
+PCollectionString slidingWindowedItems = items.apply(
 Window.Stringinto(SlidingWindows.of(Duration.standardMinutes(30)).every
 
@@ -2913,7 +2913,7 @@ into session windows, where each session must be 
separated by a time gap of at
 least 10 minutes:
 
 
PCollectionString items 
= ...;
-PCollectionString session_windowed_items = items.apply(
+PCollectionString sessionWindowedItems = items.apply(
 Window.Stringinto(Sessions.withGapDuration(Duration.standardMinutes(10;
 
 
@@ -2934,7 +2934,7 @@ elements to a single global window. The following example 
code shows how to set
 a single global window for a PCollection:
 
 
PCollectionString items 
= ...;
-PCollectionString batch_items = items.apply(
+PCollectionString batchItems = items.apply(
 Window.Stringinto(new GlobalWindows()));
 
 
@@ -2997,7 +2997,7 @@ demonstrates a windowing strategy that will allow late 
data up to two days after
 the end of a window.
 
 
PCollectionString items 
= ...;
-PCollectionString fixed_windowed_items = items.apply(
+PCollectionString fixedWindowedItems = items.apply(
 Window.Stringinto(FixedWindows.of(Duration.standardMinutes(1)))
   .withAllowedLateness(Duration.standardDays(2)));
 

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Spark #1388

[Apache Jenkins Server]

See 


Changes:

[rmannibucau] [BEAM-3728][BEAM-3729] fixing the classloader lookup for pipeline

--
[...truncated 87.62 KB...]
'apache-beam-testing:bqjob_r2c4c1094d59a9d6d_0161bebdb332_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-22 18:19:20,521 91999e72 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-22 18:19:37,499 91999e72 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-22 18:19:39,675 91999e72 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.17s,  CPU:0.30s,  MaxMemory:25360kb 
STDOUT: Upload complete.
Waiting on bqjob_r43df1e81914a6bbb_0161bebdfdc9_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r43df1e81914a6bbb_0161bebdfdc9_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r43df1e81914a6bbb_0161bebdfdc9_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-22 18:19:39,676 91999e72 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-22 18:20:07,484 91999e72 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-22 18:20:10,459 91999e72 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.96s,  CPU:0.28s,  MaxMemory:25268kb 
STDOUT: Upload complete.
Waiting on bqjob_r76bca5beaf454daf_0161bebe72e6_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r76bca5beaf454daf_0161bebe72e6_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r76bca5beaf454daf_0161bebe72e6_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-02-22 18:20:10,459 91999e72 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-02-22 18:20:40,329 91999e72 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-02-22 18:20:42,791 91999e72 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.45s,  CPU:0.30s,  MaxMemory:25288kb 
STDOUT: Upload complete.
Waiting on bqjob_r58406092d5f01429_0161bebef341_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r58406092d5f01429_0161bebef341_1 ... (0s) 
Current status: 

[beam-site] 01/02: Java variable names should be in CamelCase

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 8e3394211174367fe1411131914e9e9ff3b89fe7
Author: Taro Murao 
AuthorDate: Thu Feb 22 15:29:08 2018 +0100

Java variable names should be in CamelCase
---
 src/documentation/programming-guide.md | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/documentation/programming-guide.md 
b/src/documentation/programming-guide.md
index 48b7939..7f6aea5 100644
--- a/src/documentation/programming-guide.md
+++ b/src/documentation/programming-guide.md
@@ -2233,7 +2233,7 @@ into fixed windows, each one minute in length:
 
 ```java
 PCollection items = ...;
-PCollection fixed_windowed_items = items.apply(
+PCollection fixedWindowedItems = items.apply(
 Window.into(FixedWindows.of(Duration.standardMinutes(1;
 ```
 ```py
@@ -2249,7 +2249,7 @@ begins every five seconds:
 
 ```java
 PCollection items = ...;
-PCollection sliding_windowed_items = items.apply(
+PCollection slidingWindowedItems = items.apply(
 
Window.into(SlidingWindows.of(Duration.standardMinutes(30)).every(Duration.standardSeconds(5;
 ```
 ```py
@@ -2265,7 +2265,7 @@ least 10 minutes:
 
 ```java
 PCollection items = ...;
-PCollection session_windowed_items = items.apply(
+PCollection sessionWindowedItems = items.apply(
 
Window.into(Sessions.withGapDuration(Duration.standardMinutes(10;
 ```
 ```py
@@ -2284,7 +2284,7 @@ a single global window for a `PCollection`:
 
 ```java
 PCollection items = ...;
-PCollection batch_items = items.apply(
+PCollection batchItems = items.apply(
 Window.into(new GlobalWindows()));
 ```
 ```py
@@ -2343,7 +2343,7 @@ the end of a window.
 
 ```java
 PCollection items = ...;
-PCollection fixed_windowed_items = items.apply(
+PCollection fixedWindowedItems = items.apply(
 Window.into(FixedWindows.of(Duration.standardMinutes(1)))
   .withAllowedLateness(Duration.standardDays(2)));
 ```

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (2d1f948 -> f238648)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from 2d1f948  This closes #391
 add 1f4014c  Prepare repository for deployment.
 new 8e33942  Java variable names should be in CamelCase
 new f238648  This closes #392

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/blog/2018/02/19/beam-2.3.0.html |  4 ++--
 content/feed.xml|  4 ++--
 src/documentation/programming-guide.md  | 10 +-
 3 files changed, 9 insertions(+), 9 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 02/02: This closes #392

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit f238648d9825e81f9817a7fc7dcb4c04f8f49041
Merge: 1f4014c 8e33942
Author: Mergebot 
AuthorDate: Thu Feb 22 10:25:17 2018 -0800

This closes #392

 src/documentation/programming-guide.md | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_JDBC #247

[Apache Jenkins Server]

See 


Changes:

[rmannibucau] [BEAM-3728][BEAM-3729] fixing the classloader lookup for pipeline

--
[...truncated 37.65 KB...]
[INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing 

 with 

[INFO] Replacing original test artifact with shaded test artifact.
[INFO] Replacing 

 with 

[INFO] Dependency-reduced POM written at: 

[INFO] 
[INFO] --- maven-failsafe-plugin:2.20.1:integration-test (default) @ 
beam-sdks-java-io-jdbc ---
[INFO] Failsafe report directory: 

[INFO] parallel='all', perCoreThreadCount=true, threadCount=4, 
useUnlimitedThreads=false, threadCountSuites=0, threadCountClasses=0, 
threadCountMethods=0, parallelOptimized=true
[INFO] 
[INFO] ---
[INFO]  T E S T S
[INFO] ---
[INFO] Running org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
1,104.985 s <<< FAILURE! - in org.apache.beam.sdk.io.jdbc.JdbcIOIT
[ERROR] testWriteThenRead(org.apache.beam.sdk.io.jdbc.JdbcIOIT)  Time elapsed: 
1,104.985 s  <<< ERROR!
java.lang.RuntimeException: 
(8e96aa2c63363caa): java.lang.RuntimeException: 
org.apache.beam.sdk.util.UserCodeException: org.postgresql.util.PSQLException: 
The connection attempt failed.
at 
com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:404)
at 
com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:374)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
at 
com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
at 
com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
at 
com.google.cloud.dataflow.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:158)
at 
com.google.cloud.dataflow.worker.DataflowWorker.doWork(DataflowWorker.java:308)
at 
com.google.cloud.dataflow.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:264)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:133)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:113)
at 
com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:100)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: 
org.postgresql.util.PSQLException: The connection attempt failed.
at 
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at 
org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn$DoFnInvoker.invokeSetup(Unknown 
Source)
at 
com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:63)
at 
com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:45)
at 
com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:94)
at 
com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:74)
at 
com.google.cloud.dataflow.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:481)
at 
com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:392)
... 14 more
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
at 

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #6027

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #944

[Apache Jenkins Server]

See 


Changes:

[rmannibucau] [BEAM-3728][BEAM-3729] fixing the classloader lookup for pipeline

--
[...truncated 696 B...]
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision bd49a61611ccfc1887aef1ac1c2367e836849ab7 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bd49a61611ccfc1887aef1ac1c2367e836849ab7
Commit message: "[BEAM-3728][BEAM-3729] fixing the classloader lookup for 
pipeline options factory creation"
 > git rev-list --no-walk 385f82def9d56c56c2770466d3395c977296d057 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8659433129892478665.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8422449508169249829.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2583026488135407858.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7877419370288425215.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6632177425724179202.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6135431430850463740.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #172

[Apache Jenkins Server]

See 


Changes:

[rmannibucau] [BEAM-3728][BEAM-3729] fixing the classloader lookup for pipeline

--
[...truncated 29.92 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.4.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.4.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0-beta from the 
shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded 

[beam] branch master updated (962adc5 -> 2fbc195)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 962adc5  Merge pull request #4601: Use Concurrent Constructs in 
InMemoryArtifactStagerService
 add 9feb238  Add Environment Manager Interfaces
 new 2fbc195  Merge pull request #4589: Add Environment Manager Interfaces

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../environment/EnvironmentManager.java| 27 +---
 .../environment/RemoteEnvironment.java}| 29 +-
 .../fnexecution/environment}/package-info.java |  4 +--
 3 files changed, 26 insertions(+), 34 deletions(-)
 copy 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Materialization.java
 => 
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/EnvironmentManager.java
 (55%)
 copy 
runners/{local-java/src/main/java/org/apache/beam/runners/local/PipelineMessageReceiver.java
 => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/RemoteEnvironment.java}
 (56%)
 copy 
runners/{apex/src/main/java/org/apache/beam/runners/apex/translation/utils => 
java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment}/package-info.java
 (86%)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #4589: Add Environment Manager Interfaces

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2fbc195672ba1daf70b9964be3a95f545bf7c81a
Merge: 962adc5 9feb238
Author: Thomas Groh 
AuthorDate: Thu Feb 22 10:05:52 2018 -0800

Merge pull request #4589: Add Environment Manager Interfaces

[BEAM-3327] Add Environment Manager Interfaces

 .../environment/EnvironmentManager.java| 37 
 .../fnexecution/environment/RemoteEnvironment.java | 39 ++
 .../fnexecution/environment/package-info.java  | 22 
 3 files changed, 98 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] branch master updated (bd49a61 -> 962adc5)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bd49a61  [BEAM-3728][BEAM-3729] fixing the classloader lookup for 
pipeline options factory creation
 add b1767db  Use Conccurrent Constructs in InMemoryArtifactStagerService
 add 962adc5  Merge pull request #4601: Use Concurrent Constructs in 
InMemoryArtifactStagerService

No new revisions were added by this update.

Summary of changes:
 .../construction/InMemoryArtifactStagerService.java   | 19 +--
 1 file changed, 13 insertions(+), 6 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #6026

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-3730) typehints.TypeVariable issues with __hash__

[Ahmet Altay (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmet Altay reassigned BEAM-3730:
-

Assignee: (was: Ahmet Altay)

> typehints.TypeVariable issues with __hash__
> ---
>
> Key: BEAM-3730
> URL: https://issues.apache.org/jira/browse/BEAM-3730
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Luke Zhu
>Priority: Major
>
> This class currently does not define a __hash__ function. This defies the 
> assumption made by sets and dicts which contain TypeConstraint objects.
> In Python 3, classes which define ___eq___ also need to explicitly define 
> ___hash___ to be hashable. However, this causes 
> _combiners_test.CombineTest.test_to_list_and_to_dict_ to throw an error in 
> slow_stream.py (adding NoneType and long).
> There are multiple TypeVariable instances with name 'K' or 'V' throughout the 
> codebase. The equality of these instances may possibly cause issues.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3581) [SQL] Support for Non-ASCII chars is flaky

[Anton Kedin (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373091#comment-16373091
 ] 

Anton Kedin commented on BEAM-3581:
---

We're on Calcite 1.15 now, created the Jira to put the settings into the 
properties file instead of build.gradle: 
https://issues.apache.org/jira/browse/BEAM-3733

> [SQL] Support for Non-ASCII chars is flaky
> --
>
> Key: BEAM-3581
> URL: https://issues.apache.org/jira/browse/BEAM-3581
> Project: Beam
>  Issue Type: Bug
>  Components: dsl-sql
>Reporter: Anton Kedin
>Assignee: Anton Kedin
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Beam SQL overrides default charset that Calcite uses and sets it to UTF16. It 
> is done via system properties.
> Problem is that we do this only when it hasn't been set yet. So if system 
> property has been set to ISO-8859-1 (Calcite's default), then test runs will 
> fail when trying to encode characters not supported in that encoding.
> Solution:
>  - because it's a system property, we don't want to force override it;
>  - for the same reason we cannot set it for a specific query execution;
>  - we can expose a static method on BeamSql to override these properties if 
> explicitly requested;
>  - affected tests will explicitly override it;
>  - otherwise behavior will stay unchanged and we will respect defaults and 
> user settings;



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3733) [SQL] Set encoding in saffron.properties instead of build.gradle

[Anton Kedin (JIRA)]

Anton Kedin created BEAM-3733:
-

 Summary: [SQL] Set encoding in saffron.properties instead of 
build.gradle
 Key: BEAM-3733
 URL: https://issues.apache.org/jira/browse/BEAM-3733
 Project: Beam
  Issue Type: Improvement
  Components: dsl-sql
Reporter: Anton Kedin


Before Calcite 1.15 we had to set encoding in build.gradle: 
https://issues.apache.org/jira/browse/BEAM-3581

 

Now that we're on Calcite 1.15 we should set the encoding in saffron.properties 
file instead



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3726) Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move backwards

[Pawel Bartoszek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pawel Bartoszek updated BEAM-3726:
--
Description: 
When the job is restored from savepoint Kinesis Reader throws almost always 
{{java.lang.IllegalArgumentException: Attempting to move backwards}}

After a few job restarts caused again by the same exception, job finally starts 
up and continues to run with no further problems.

Beam job is reading from 32 shards with parallelism set to 32. Using Flink 1.3.2

 
{code:java}
java.lang.IllegalArgumentException: Attempting to move backwards
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.apache.beam.sdk.util.MovingFunction.flush(MovingFunction.java:97)
at org.apache.beam.sdk.util.MovingFunction.add(MovingFunction.java:114)
at org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:137)
at 
org.apache.beam.runners.flink.metrics.ReaderInvocationUtil.invokeAdvance(ReaderInvocationUtil.java:67)
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:264)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
at 
org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:39)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702){code}
 

Kinesis Reader transformation configuration:
{code:java}
pipeline.apply("KINESIS READER", KinesisIO.read()
.withStreamName(streamName)
.withInitialPositionInStream(InitialPositionInStream.LATEST)
.withAWSClientsProvider(awsAccessKey, awsSecretKey, EU_WEST_1)){code}
 

When testing locally I managed to catch this exception. Just before executing 
this 
[link|https://github.com/apache/beam/blob/6c93105c2cb7be709c6b3e2e6cdcd09df2b48308/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java#L97]
 that threw exception I captured the state of the class so that you can 
replicate the issue
{code:java}
org.apache.beam.sdk.util.MovingFunction@71781a[sampleUpdateMs=5000,numSignificantBuckets=2,numSignificantSamples=10,function=org.apache.beam.sdk.transforms.Min$MinLongFn@7909d8d3,buckets={9223372036854775807,9223372036854775807,1519315344334,1519315343759,1519315343770,1519315344086,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807},numSamples={0,0,1,158,156,146,0,0,0,0,144,0},currentMsSinceEpoch=1519315585000,currentIndex=2]{code}
 

the add function of MovingFunction was called with nowMsSinceEpoch = 
1519315583591

 

  was:
When the job is restored from savepoint Kinesis Reader throws sometimes 
{{java.lang.IllegalArgumentException: Attempting to move backwards}}

After a few job restarts caused again by the same exception, job finally starts 
up and continues to run with no further problems.

 
{code:java}
java.lang.IllegalArgumentException: Attempting to move backwards
at 
org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
at org.apache.beam.sdk.util.MovingFunction.flush(MovingFunction.java:97)
at org.apache.beam.sdk.util.MovingFunction.add(MovingFunction.java:114)
at org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:137)
at 
org.apache.beam.runners.flink.metrics.ReaderInvocationUtil.invokeAdvance(ReaderInvocationUtil.java:67)
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:264)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
at 
org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:39)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702){code}
 

Kinesis Reader transformation configuration:
{code:java}
pipeline.apply("KINESIS READER", KinesisIO.read()
.withStreamName(streamName)
.withInitialPositionInStream(InitialPositionInStream.LATEST)
.withAWSClientsProvider(awsAccessKey, awsSecretKey, EU_WEST_1)){code}
 


> Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move 
> backwards
> 
>
> Key: BEAM-3726
> URL: https://issues.apache.org/jira/browse/BEAM-3726
> Project: 

[jira] [Resolved] (BEAM-3729) Spark ValidatesRunner broken with "org.apache.beam.sdk.options.$Proxy72 cannot access its superinterface"

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3729.
-
Resolution: Fixed

> Spark ValidatesRunner broken with "org.apache.beam.sdk.options.$Proxy72 
> cannot access its superinterface"
> -
>
> Key: BEAM-3729
> URL: https://issues.apache.org/jira/browse/BEAM-3729
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Romain Manni-Bucau
>Priority: Blocker
> Fix For: 2.4.0
>
>
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/4243/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3700) PipelineOptionsFactory leaks memory

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373065#comment-16373065
 ] 

Luke Cwik commented on BEAM-3700:
-

Because of the misses in unit testing, consider adding a unit test that would 
have caught the failures in BEAM-3728 and BEAM-3729.

> PipelineOptionsFactory leaks memory
> ---
>
> Key: BEAM-3700
> URL: https://issues.apache.org/jira/browse/BEAM-3700
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Romain Manni-Bucau
>Assignee: Romain Manni-Bucau
>Priority: Major
> Fix For: 2.4.0
>
>
> PipelineOptionsFactory has a lot of cache but no way to reset it. This task 
> is about adding a public method to be able to control it in integrations 
> (runners likely).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3728) Failing ParDoTest for Flink Runner

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik reassigned BEAM-3728:
---

Assignee: Romain Manni-Bucau

> Failing ParDoTest for Flink Runner
> --
>
> Key: BEAM-3728
> URL: https://issues.apache.org/jira/browse/BEAM-3728
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Romain Manni-Bucau
>Priority: Blocker
> Fix For: 2.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> It seems the changes for BEAM-3700/BEAM-3701 broke some tests in ParDoTest 
> for the Flink Runner: 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/5054/
> I don't currently understand what is causing this. [~lcwik] Do you have any 
> idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3728) Failing ParDoTest for Flink Runner

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik resolved BEAM-3728.
-
Resolution: Fixed

> Failing ParDoTest for Flink Runner
> --
>
> Key: BEAM-3728
> URL: https://issues.apache.org/jira/browse/BEAM-3728
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Romain Manni-Bucau
>Priority: Blocker
> Fix For: 2.4.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> It seems the changes for BEAM-3700/BEAM-3701 broke some tests in ParDoTest 
> for the Flink Runner: 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/5054/
> I don't currently understand what is causing this. [~lcwik] Do you have any 
> idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (385f82d -> bd49a61)

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 385f82d  Merge pull request #4708: [BEAM-3538] Remove (or merge) Java 
8 specific tests module into the main one
 add a58377d  [BEAM-3728][BEAM-3729] fixing the classloader lookup for 
pipeline options factory creation
 new bd49a61  [BEAM-3728][BEAM-3729] fixing the classloader lookup for 
pipeline options factory creation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/options/PipelineOptionsFactory.java   |  8 ++--
 .../beam/sdk/util/common/ReflectHelpers.java   | 48 ++
 2 files changed, 52 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[beam] 01/01: [BEAM-3728][BEAM-3729] fixing the classloader lookup for pipeline options factory creation

[lcwik]

This is an automated email from the ASF dual-hosted git repository.

lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit bd49a61611ccfc1887aef1ac1c2367e836849ab7
Merge: 385f82d a58377d
Author: Lukasz Cwik 
AuthorDate: Thu Feb 22 08:51:57 2018 -0800

[BEAM-3728][BEAM-3729] fixing the classloader lookup for pipeline options 
factory creation

 .../beam/sdk/options/PipelineOptionsFactory.java   |  8 ++--
 .../beam/sdk/util/common/ReflectHelpers.java   | 48 ++
 2 files changed, 52 insertions(+), 4 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
lc...@apache.org.

[jira] [Commented] (BEAM-3440) Generic IOIT profiles configuration in io's parent pom.xml

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373010#comment-16373010
 ] 

Łukasz Gajowy commented on BEAM-3440:
-

I agree about the fact that it may be hard to live with such inheritance. But 
in my opinion, living with lots of profiles whit the same purpose can be a 
burden too (duplication). Please see BEAM-3732 - I noticed that it's super easy 
to break them now. 

I didn't take a look at Gradle configuration yet. Will this be fine if I 
propose a PR for maven only and then take a look at Gradle?

> Generic IOIT profiles configuration in io's parent pom.xml
> --
>
> Key: BEAM-3440
> URL: https://issues.apache.org/jira/browse/BEAM-3440
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Łukasz Gajowy
>Assignee: Łukasz Gajowy
>Priority: Major
>
> Currently developing IOITs requires copying maven profile configuration to 
> every's IO pom for profiles like: io-it-suite, runner profiles etc. This can 
> be avoided by moving all the common profiles to sdks/java/io/pom.xml parent 
> pom and leave only some test specific argument settings in concrete child 
> modules. This can be done using maven properties ( section in 
> appropriate child pom's profile section). This would reduce lots of xml code 
> duplication.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3440) Generic IOIT profiles configuration in io's parent pom.xml

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373010#comment-16373010
 ] 

Łukasz Gajowy edited comment on BEAM-3440 at 2/22/18 4:19 PM:
--

[~kenn]

I agree about the fact that it may be hard to live with such inheritance. But 
in my opinion, living with lots of profiles whit the same purpose can be a 
burden too (duplication). Please see BEAM-3732 - I noticed that it's super easy 
to break them now. 

I didn't take a look at Gradle configuration yet. Will this be fine if I 
propose a PR for maven only and then take a look at Gradle?


was (Author: łukaszg):
I agree about the fact that it may be hard to live with such inheritance. But 
in my opinion, living with lots of profiles whit the same purpose can be a 
burden too (duplication). Please see BEAM-3732 - I noticed that it's super easy 
to break them now. 

I didn't take a look at Gradle configuration yet. Will this be fine if I 
propose a PR for maven only and then take a look at Gradle?

> Generic IOIT profiles configuration in io's parent pom.xml
> --
>
> Key: BEAM-3440
> URL: https://issues.apache.org/jira/browse/BEAM-3440
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Łukasz Gajowy
>Assignee: Łukasz Gajowy
>Priority: Major
>
> Currently developing IOITs requires copying maven profile configuration to 
> every's IO pom for profiles like: io-it-suite, runner profiles etc. This can 
> be avoided by moving all the common profiles to sdks/java/io/pom.xml parent 
> pom and leave only some test specific argument settings in concrete child 
> modules. This can be done using maven properties ( section in 
> appropriate child pom's profile section). This would reduce lots of xml code 
> duplication.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3732) Building with profiles io-it-suite and io-it-suite-local fails

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373000#comment-16373000
 ] 

Łukasz Gajowy commented on BEAM-3732:
-

Solving 3440 would avoid such issues or at least reduce them. Having those 
profiles in only one place would get rid of the duplication. Such issues would 
be much easier to find.

> Building with profiles io-it-suite and io-it-suite-local fails
> --
>
> Key: BEAM-3732
> URL: https://issues.apache.org/jira/browse/BEAM-3732
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-hadoop, io-java-jdbc, sdk-java-core, testing
>Reporter: Łukasz Gajowy
>Assignee: Łukasz Gajowy
>Priority: Major
>
> Running the following command:
> {noformat}
> mvn clean verify -Pio-it-suite{noformat}
> results in:
> {noformat}
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [ERROR] 'build.plugins.plugin.version' for 
> org.apache.maven.plugins:maven-surefire-plugin must be a valid version but is 
> '${surefire-plugin.version}'. @ 
> org.apache.beam:beam-sdks-java-io-file-based-io-tests:[unknown-version], 
> /Users/lukasz/Projects/apache-beam/beam/sdks/java/io/file-based-io-tests/pom.xml,
>  line 139, column 34
> [ERROR] 'build.plugins.plugin.version' for 
> org.apache.maven.plugins:maven-surefire-plugin must be a valid version but is 
> '${surefire-plugin.version}'. @ 
> org.apache.beam:beam-sdks-java-io-hadoop-input-format:[unknown-version], 
> /Users/lukasz/Projects/apache-beam/beam/sdks/java/io/hadoop-input-format/pom.xml,
>  line 185, column 22
> @
> [ERROR] The build could not read 2 projects -> [Help 1]
> [ERROR]
> [ERROR] The project 
> org.apache.beam:beam-sdks-java-io-file-based-io-tests:2.4.0-SNAPSHOT 
> (/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/file-based-io-tests/pom.xml)
>  has 1 error
> [ERROR] 'build.plugins.plugin.version' for 
> org.apache.maven.plugins:maven-surefire-plugin must be a valid version but is 
> '${surefire-plugin.version}'. @ 
> org.apache.beam:beam-sdks-java-io-file-based-io-tests:[unknown-version], 
> /Users/lukasz/Projects/apache-beam/beam/sdks/java/io/file-based-io-tests/pom.xml,
>  line 139, column 34
> [ERROR]
> [ERROR] The project 
> org.apache.beam:beam-sdks-java-io-hadoop-input-format:2.4.0-SNAPSHOT 
> (/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/hadoop-input-format/pom.xml)
>  has 1 error
> [ERROR] 'build.plugins.plugin.version' for 
> org.apache.maven.plugins:maven-surefire-plugin must be a valid version but is 
> '${surefire-plugin.version}'. @ 
> org.apache.beam:beam-sdks-java-io-hadoop-input-format:[unknown-version], 
> /Users/lukasz/Projects/apache-beam/beam/sdks/java/io/hadoop-input-format/pom.xml,
>  line 185, column 22
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please 
> read the following articles:
> [ERROR] [Help 1] 
> http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException{noformat}
>  
> This is broken since commit: 0ec27cd4fcc28350860e303786ad31a07aa87bf2. 
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3732) Building with profiles io-it-suite and io-it-suite-local fails

[JIRA]

Łukasz Gajowy created BEAM-3732:
---

 Summary: Building with profiles io-it-suite and io-it-suite-local 
fails
 Key: BEAM-3732
 URL: https://issues.apache.org/jira/browse/BEAM-3732
 Project: Beam
  Issue Type: Bug
  Components: io-java-hadoop, io-java-jdbc, sdk-java-core, testing
Reporter: Łukasz Gajowy
Assignee: Łukasz Gajowy


Running the following command:
{noformat}
mvn clean verify -Pio-it-suite{noformat}
results in:
{noformat}
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] 'build.plugins.plugin.version' for 
org.apache.maven.plugins:maven-surefire-plugin must be a valid version but is 
'${surefire-plugin.version}'. @ 
org.apache.beam:beam-sdks-java-io-file-based-io-tests:[unknown-version], 
/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/file-based-io-tests/pom.xml,
 line 139, column 34
[ERROR] 'build.plugins.plugin.version' for 
org.apache.maven.plugins:maven-surefire-plugin must be a valid version but is 
'${surefire-plugin.version}'. @ 
org.apache.beam:beam-sdks-java-io-hadoop-input-format:[unknown-version], 
/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/hadoop-input-format/pom.xml,
 line 185, column 22
@
[ERROR] The build could not read 2 projects -> [Help 1]
[ERROR]
[ERROR] The project 
org.apache.beam:beam-sdks-java-io-file-based-io-tests:2.4.0-SNAPSHOT 
(/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/file-based-io-tests/pom.xml)
 has 1 error
[ERROR] 'build.plugins.plugin.version' for 
org.apache.maven.plugins:maven-surefire-plugin must be a valid version but is 
'${surefire-plugin.version}'. @ 
org.apache.beam:beam-sdks-java-io-file-based-io-tests:[unknown-version], 
/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/file-based-io-tests/pom.xml,
 line 139, column 34
[ERROR]
[ERROR] The project 
org.apache.beam:beam-sdks-java-io-hadoop-input-format:2.4.0-SNAPSHOT 
(/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/hadoop-input-format/pom.xml)
 has 1 error
[ERROR] 'build.plugins.plugin.version' for 
org.apache.maven.plugins:maven-surefire-plugin must be a valid version but is 
'${surefire-plugin.version}'. @ 
org.apache.beam:beam-sdks-java-io-hadoop-input-format:[unknown-version], 
/Users/lukasz/Projects/apache-beam/beam/sdks/java/io/hadoop-input-format/pom.xml,
 line 185, column 22
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException{noformat}
 

This is broken since commit: 0ec27cd4fcc28350860e303786ad31a07aa87bf2. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Spark #4250

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3731) Enable tests to run in Python 3

[Luke Zhu (JIRA)]

Luke Zhu created BEAM-3731:
--

 Summary: Enable tests to run in Python 3
 Key: BEAM-3731
 URL: https://issues.apache.org/jira/browse/BEAM-3731
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Luke Zhu
Assignee: Luke Zhu


Currently the Python3 tests fail to run. This makes it difficult to continue 
with Python 3 compatibility.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Flink #5062

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3730) typehints.TypeVariable issues with __hash__

[Luke Zhu (JIRA)]

Luke Zhu created BEAM-3730:
--

 Summary: typehints.TypeVariable issues with __hash__
 Key: BEAM-3730
 URL: https://issues.apache.org/jira/browse/BEAM-3730
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Luke Zhu
Assignee: Ahmet Altay


This class currently does not define a __hash__ function. This defies the 
assumption made by sets and dicts which contain TypeConstraint objects.

In Python 3, classes which define ___eq___ also need to explicitly define 
___hash___ to be hashable. However, this causes 
_combiners_test.CombineTest.test_to_list_and_to_dict_ to throw an error in 
slow_stream.py (adding NoneType and long).

There are multiple TypeVariable instances with name 'K' or 'V' throughout the 
codebase. The equality of these instances may possibly cause issues.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3729) Spark ValidatesRunner broken with "org.apache.beam.sdk.options.$Proxy72 cannot access its superinterface"

[Romain Manni-Bucau (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372945#comment-16372945
 ] 

Romain Manni-Bucau commented on BEAM-3729:
--

https://github.com/apache/beam/pull/4729

> Spark ValidatesRunner broken with "org.apache.beam.sdk.options.$Proxy72 
> cannot access its superinterface"
> -
>
> Key: BEAM-3729
> URL: https://issues.apache.org/jira/browse/BEAM-3729
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Romain Manni-Bucau
>Priority: Blocker
> Fix For: 2.4.0
>
>
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/4243/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3728) Failing ParDoTest for Flink Runner

[Romain Manni-Bucau (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372944#comment-16372944
 ] 

Romain Manni-Bucau commented on BEAM-3728:
--

https://github.com/apache/beam/pull/4729

> Failing ParDoTest for Flink Runner
> --
>
> Key: BEAM-3728
> URL: https://issues.apache.org/jira/browse/BEAM-3728
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 2.4.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> It seems the changes for BEAM-3700/BEAM-3701 broke some tests in ParDoTest 
> for the Flink Runner: 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/5054/
> I don't currently understand what is causing this. [~lcwik] Do you have any 
> idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372917#comment-16372917
 ] 

Ismaël Mejía commented on BEAM-3664:


Jump in just to say that the ElasticsearchIO tests use exactly the same pattern 
that Solr, so if you fix Solr you can easily solve the Elasticsearch ticket for 
free (BEAM-3674). Feel free to take it if you are interested.

> Port SolrIOTest off DoFnTester
> --
>
> Key: BEAM-3664
> URL: https://issues.apache.org/jira/browse/BEAM-3664
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-solr
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3728) Failing ParDoTest for Flink Runner

[Aljoscha Krettek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aljoscha Krettek reassigned BEAM-3728:
--

Assignee: (was: Aljoscha Krettek)

> Failing ParDoTest for Flink Runner
> --
>
> Key: BEAM-3728
> URL: https://issues.apache.org/jira/browse/BEAM-3728
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Priority: Blocker
> Fix For: 2.4.0
>
>
> It seems the changes for BEAM-3700/BEAM-3701 broke some tests in ParDoTest 
> for the Flink Runner: 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/5054/
> I don't currently understand what is causing this. [~lcwik] Do you have any 
> idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3728) Failing ParDoTest for Flink Runner

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372906#comment-16372906
 ] 

Aljoscha Krettek commented on BEAM-3728:


Also cc [~romain.manni-bucau] as the implementor. I thought it was Luke but I 
was confused by the git history.

> Failing ParDoTest for Flink Runner
> --
>
> Key: BEAM-3728
> URL: https://issues.apache.org/jira/browse/BEAM-3728
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 2.4.0
>
>
> It seems the changes for BEAM-3700/BEAM-3701 broke some tests in ParDoTest 
> for the Flink Runner: 
> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/5054/
> I don't currently understand what is causing this. [~lcwik] Do you have any 
> idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3664) Port SolrIOTest off DoFnTester

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372869#comment-16372869
 ] 

Kenneth Knowles commented on BEAM-3664:
---

[~wslulciuc] I just read through the test. We have a few tests like this. I 
think they are a bit overspecified. I would suggest:

1. Eliminate the for loop and just pass the input docs to 
{{TestStream.addElements}}.
2. Verify that the output is as expected.
3. Determine a way to track insert requests to Solr. Could be via a fake 
{{solrClient}} (probably best) or by tweaking the data you insert with request 
ids.
4. Verify that no request exceeded the max size.

> Port SolrIOTest off DoFnTester
> --
>
> Key: BEAM-3664
> URL: https://issues.apache.org/jira/browse/BEAM-3664
> Project: Beam
>  Issue Type: Sub-task
>  Components: io-java-solr
>Reporter: Kenneth Knowles
>Priority: Major
>  Labels: beginner, newbie, starter
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3685) It should be an error to run a Pipeline without ever specifying options

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles reassigned BEAM-3685:
-

Assignee: Willy Lulciuc

> It should be an error to run a Pipeline without ever specifying options
> ---
>
> Key: BEAM-3685
> URL: https://issues.apache.org/jira/browse/BEAM-3685
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Willy Lulciuc
>Priority: Major
>  Labels: beginner, newbie, starter
>
> Doing so lets users run some pipelines without specifying any configuration, 
> which is dangerous.
>  
> At minimum, it should log a very obvious warning.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3729) Spark ValidatesRunner broken with "org.apache.beam.sdk.options.$Proxy72 cannot access its superinterface"

[Kenneth Knowles (JIRA)]

Kenneth Knowles created BEAM-3729:
-

 Summary: Spark ValidatesRunner broken with 
"org.apache.beam.sdk.options.$Proxy72 cannot access its superinterface"
 Key: BEAM-3729
 URL: https://issues.apache.org/jira/browse/BEAM-3729
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Kenneth Knowles
Assignee: Romain Manni-Bucau
 Fix For: 2.4.0


https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/4243/console



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-3728) Failing ParDoTest for Flink Runner

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created BEAM-3728:
--

 Summary: Failing ParDoTest for Flink Runner
 Key: BEAM-3728
 URL: https://issues.apache.org/jira/browse/BEAM-3728
 Project: Beam
  Issue Type: Bug
  Components: runner-flink
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
 Fix For: 2.4.0


It seems the changes for BEAM-3700/BEAM-3701 broke some tests in ParDoTest for 
the Flink Runner: 
https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/5054/

I don't currently understand what is causing this. [~lcwik] Do you have any 
idea?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)