Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #2514

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-831) ParDo Chaining

[Chinmay Kolhatkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Kolhatkar reassigned BEAM-831:
--

Assignee: Chinmay Kolhatkar

> ParDo Chaining
> --
>
> Key: BEAM-831
> URL: https://issues.apache.org/jira/browse/BEAM-831
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Thomas Weise
>Assignee: Chinmay Kolhatkar
>
> Current state of Apex runner creates a plan that will place each operator in 
> a separate container (which would be processes when running on a YARN 
> cluster). Often the ParDo operators can be collocated in same thread or 
> container. Use Apex affinity/stream locality attributes for more efficient 
> execution plan.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-831) ParDo Chaining

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904592#comment-15904592
 ] 

ASF GitHub Bot commented on BEAM-831:
-

GitHub user chinmaykolhatkar opened a pull request:

https://github.com/apache/beam/pull/2216

[BEAM-831] ParDo Chaining of Apex Runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

R: @tweise, @kennknowles 
---

This is currently Work in Progress PR. The tests are being developed. This 
PR was created to have a convenient environment for discussing this code. Here 
are the TODOS:
1. Create the test cases for ParDo fusion for Apex runner


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chinmaykolhatkar/beam BEAM-831_ParDoChaining

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2216.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2216


commit d33532e607a5853f664602b3aae49c41a30bf046
Author: chinmaykolhatkar 
Date:   2017-03-01T11:29:46Z

[BEAM-831] ParDo Fusion of Apex Runner




> ParDo Chaining
> --
>
> Key: BEAM-831
> URL: https://issues.apache.org/jira/browse/BEAM-831
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-apex
>Reporter: Thomas Weise
>
> Current state of Apex runner creates a plan that will place each operator in 
> a separate container (which would be processes when running on a YARN 
> cluster). Often the ParDo operators can be collocated in same thread or 
> container. Use Apex affinity/stream locality attributes for more efficient 
> execution plan.  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2216: [BEAM-831] ParDo Chaining of Apex Runner

[chinmaykolhatkar]

GitHub user chinmaykolhatkar opened a pull request:

https://github.com/apache/beam/pull/2216

[BEAM-831] ParDo Chaining of Apex Runner

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [x] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [x] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

R: @tweise, @kennknowles 
---

This is currently Work in Progress PR. The tests are being developed. This 
PR was created to have a convenient environment for discussing this code. Here 
are the TODOS:
1. Create the test cases for ParDo fusion for Apex runner


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/chinmaykolhatkar/beam BEAM-831_ParDoChaining

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2216.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2216


commit d33532e607a5853f664602b3aae49c41a30bf046
Author: chinmaykolhatkar 
Date:   2017-03-01T11:29:46Z

[BEAM-831] ParDo Fusion of Apex Runner




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is unstable: beam_PostCommit_Java_MavenInstall #2862

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Apex #721

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Spark #1188

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_RunnableOnService_Flink #1878

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-1650) Add code style xml to the project repo and website

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903560#comment-15903560
 ] 

Aviem Zur edited comment on BEAM-1650 at 3/10/17 5:30 AM:
--

Changed to use Eclipse code formatter profile xml. These can be generated from 
checkstyle xml using Eclipse and they work in both Eclipse and IntelliJ.

We might be able to leverage https://github.com/revelc/formatter-maven-plugin 
to format a test class before checkstyle runs so if the formatting does not 
match checkstyle it will fail.


was (Author: aviemzur):
Changed to use Eclipse code formatter profile xml, it works in both Eclipse and 
IntelliJ.
We might be able to leverage https://github.com/revelc/formatter-maven-plugin 
to format a test class before checkstyle runs so if the formatting does not 
match checkstyle it will fail.

> Add code style xml to the project repo and website 
> ---
>
> Key: BEAM-1650
> URL: https://issues.apache.org/jira/browse/BEAM-1650
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Contributors who develop using IntelliJ IDEa or Eclipse could benefit from 
> having an Eclipse code formatter xml which matches the checkstyle 
> enforcements of Beam.
> Add such a code style xml to the repository and link to it from the website 
> under the IDE setup sections: 
> https://beam.apache.org/contribute/contribution-guide/#intellij
> https://beam.apache.org/contribute/contribution-guide/#eclipse



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1650) Add code style xml to the project repo and website

[Aviem Zur (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903560#comment-15903560
 ] 

Aviem Zur edited comment on BEAM-1650 at 3/10/17 5:28 AM:
--

Changed to use Eclipse code formatter profile xml, it works in both Eclipse and 
IntelliJ.
We might be able to leverage https://github.com/revelc/formatter-maven-plugin 
to format a test class before checkstyle runs so if the formatting does not 
match checkstyle it will fail.


was (Author: aviemzur):
We might be able to leverage https://github.com/revelc/formatter-maven-plugin 
(If we change to Eclipse code style xml, which could be a good idea since it 
would work in both Eclipse and IntelliJ) or 
https://confluence.jetbrains.com/display/IDEADEV/Command-Line+Source+Code+Formatter
We can use one of these to format a test class before checkstyle runs so if the 
formatting does not match checkstyle it will fail.

> Add code style xml to the project repo and website 
> ---
>
> Key: BEAM-1650
> URL: https://issues.apache.org/jira/browse/BEAM-1650
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Contributors who develop using IntelliJ IDEa or Eclipse could benefit from 
> having an Eclipse code formatter xml which matches the checkstyle 
> enforcements of Beam.
> Add such a code style xml to the repository and link to it from the website 
> under the IDE setup sections: 
> https://beam.apache.org/contribute/contribution-guide/#intellij
> https://beam.apache.org/contribute/contribution-guide/#eclipse



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1651) Add code style xml to the project repository

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1651:

Summary: Add code style xml to the project repository  (was: Add IntelliJ 
code style xml to the project repository)

> Add code style xml to the project repository
> 
>
> Key: BEAM-1651
> URL: https://issues.apache.org/jira/browse/BEAM-1651
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1650) Add code style xml to the project repo and website

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1650:

Description: 
Contributors who develop using IntelliJ IDEa or Eclipse could benefit from 
having an Eclipse code formatter xml which matches the checkstyle enforcements 
of Beam.

Add such a code style xml to the repository and link to it from the website 
under the IDE setup sections: 
https://beam.apache.org/contribute/contribution-guide/#intellij
https://beam.apache.org/contribute/contribution-guide/#eclipse

  was:
Contributors who develop using IntelliJ IDEa could benefit from having a code 
style xml file which matches the checkstyle enforcements of Beam.

Add such a code style xml to the repository and link to it from the website 
under the IntelliJ setup section: 
https://beam.apache.org/contribute/contribution-guide/#intellij


> Add code style xml to the project repo and website 
> ---
>
> Key: BEAM-1650
> URL: https://issues.apache.org/jira/browse/BEAM-1650
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Contributors who develop using IntelliJ IDEa or Eclipse could benefit from 
> having an Eclipse code formatter xml which matches the checkstyle 
> enforcements of Beam.
> Add such a code style xml to the repository and link to it from the website 
> under the IDE setup sections: 
> https://beam.apache.org/contribute/contribution-guide/#intellij
> https://beam.apache.org/contribute/contribution-guide/#eclipse



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1650) Add code style xml to the project repo and website

[Aviem Zur (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aviem Zur updated BEAM-1650:

Summary: Add code style xml to the project repo and website   (was: Add 
IntelliJ code style xml to the project repo and website )

> Add code style xml to the project repo and website 
> ---
>
> Key: BEAM-1650
> URL: https://issues.apache.org/jira/browse/BEAM-1650
> Project: Beam
>  Issue Type: New Feature
>  Components: website
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>
> Contributors who develop using IntelliJ IDEa could benefit from having a code 
> style xml file which matches the checkstyle enforcements of Beam.
> Add such a code style xml to the repository and link to it from the website 
> under the IntelliJ setup section: 
> https://beam.apache.org/contribute/contribution-guide/#intellij



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1877

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_MavenInstall #2860

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-115) Beam Runner API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904419#comment-15904419
 ] 

ASF GitHub Bot commented on BEAM-115:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2190


> Beam Runner API
> ---
>
> Key: BEAM-115
> URL: https://issues.apache.org/jira/browse/BEAM-115
> Project: Beam
>  Issue Type: Improvement
>  Components: beam-model-runner-api
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>
> The PipelineRunner API from the SDK is not ideal for the Beam technical 
> vision.
> It has technical limitations:
>  - The user's DAG (even including library expansions) is never explicitly 
> represented, so it cannot be analyzed except incrementally, and cannot 
> necessarily be reconstructed (for example, to display it!).
>  - The flattened DAG of just primitive transforms isn't well-suited for 
> display or transform override.
>  - The TransformHierarchy isn't well-suited for optimizations.
>  - The user must realistically pre-commit to a runner, and its configuration 
> (batch vs streaming) prior to graph construction, since the runner will be 
> modifying the graph as it is built.
>  - It is fairly language- and SDK-specific.
> It has usability issues (these are not from intuition, but derived from 
> actual cases of failure to use according to the design)
>  - The interleaving of apply() methods in PTransform/Pipeline/PipelineRunner 
> is confusing.
>  - The TransformHierarchy, accessible only via visitor traversals, is 
> cumbersome.
>  - The staging of construction-time vs run-time is not always obvious.
> These are just examples. This ticket tracks designing, coming to consensus, 
> and building an API that more simply and directly supports the technical 
> vision.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2190: [BEAM-115] Runner API representation of windowing s...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2190


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[7/9] beam git commit: Runner API encoding of WindowFns.

[robertwb]

Runner API encoding of WindowFns.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/aad32b7a
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/aad32b7a
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/aad32b7a

Branch: refs/heads/master
Commit: aad32b7a00d1aea1e7e51b68ff609d2fb3b82a8f
Parents: bc76a18
Author: Robert Bradshaw 
Authored: Tue Mar 7 12:21:02 2017 -0800
Committer: Robert Bradshaw 
Committed: Thu Mar 9 20:29:01 2017 -0800

--
 sdks/python/apache_beam/transforms/window.py| 117 +++
 .../apache_beam/transforms/window_test.py   |  11 ++
 sdks/python/apache_beam/utils/proto_utils.py|  37 ++
 sdks/python/apache_beam/utils/urns.py   |   7 ++
 4 files changed, 172 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/aad32b7a/sdks/python/apache_beam/transforms/window.py
--
diff --git a/sdks/python/apache_beam/transforms/window.py 
b/sdks/python/apache_beam/transforms/window.py
index 14cf2f6..a562bcf 100644
--- a/sdks/python/apache_beam/transforms/window.py
+++ b/sdks/python/apache_beam/transforms/window.py
@@ -49,13 +49,20 @@ WindowFn.
 
 from __future__ import absolute_import
 
+from google.protobuf import struct_pb2
+from google.protobuf import wrappers_pb2
+
 from apache_beam import coders
+from apache_beam.internal import pickler
+from apache_beam.runners.api import beam_runner_api_pb2
 from apache_beam.transforms import timeutil
 from apache_beam.transforms.timeutil import Duration
 from apache_beam.transforms.timeutil import MAX_TIMESTAMP
 from apache_beam.transforms.timeutil import MIN_TIMESTAMP
 from apache_beam.transforms.timeutil import Timestamp
 from apache_beam.utils.windowed_value import WindowedValue
+from apache_beam.utils import proto_utils
+from apache_beam.utils import urns
 
 
 # TODO(ccy): revisit naming and semantics once Java Apache Beam finalizes their
@@ -131,6 +138,41 @@ class WindowFn(object):
 # By default, just return the input timestamp.
 return input_timestamp
 
+  _known_urns = {}
+
+  @classmethod
+  def register_urn(cls, urn, parameter_type, constructor):
+cls._known_urns[urn] = parameter_type, constructor
+
+  @classmethod
+  def from_runner_api(cls, fn_proto, context):
+parameter_type, constructor = cls._known_urns[fn_proto.spec.urn]
+return constructor(
+proto_utils.unpack_Any(fn_proto.spec.parameter, parameter_type),
+context)
+
+  def to_runner_api(self, context):
+urn, typed_param = self.to_runner_api_parameter(context)
+return beam_runner_api_pb2.FunctionSpec(
+spec=beam_runner_api_pb2.UrnWithParameter(
+urn=urn,
+parameter=proto_utils.pack_Any(typed_param)))
+
+  @staticmethod
+  def from_runner_api_parameter(fn_parameter, unused_context):
+return pickler.loads(fn_parameter.value)
+
+  def to_runner_api_parameter(self, context):
+raise TypeError(self)
+return (urns.PICKLED_WINDOW_FN,
+wrappers_pb2.BytesValue(value=pickler.dumps(self)))
+
+
+WindowFn.register_urn(
+urns.PICKLED_WINDOW_FN,
+wrappers_pb2.BytesValue,
+WindowFn.from_runner_api_parameter)
+
 
 class BoundedWindow(object):
   """A window for timestamps in range (-infinity, end).
@@ -251,6 +293,16 @@ class GlobalWindows(WindowFn):
   def __ne__(self, other):
 return not self == other
 
+  @staticmethod
+  def from_runner_api_parameter(unused_fn_parameter, unused_context):
+return GlobalWindows()
+
+  def to_runner_api_parameter(self, context):
+return urns.GLOBAL_WINDOWS_FN, None
+
+WindowFn.register_urn(
+urns.GLOBAL_WINDOWS_FN, None, GlobalWindows.from_runner_api_parameter)
+
 
 class FixedWindows(WindowFn):
   """A windowing function that assigns each element to one time interval.
@@ -280,6 +332,29 @@ class FixedWindows(WindowFn):
   def merge(self, merge_context):
 pass  # No merging.
 
+  def __eq__(self, other):
+if type(self) == type(other) == FixedWindows:
+  return self.size == other.size and self.offset == other.offset
+
+  def __ne__(self, other):
+return not self == other
+
+  @staticmethod
+  def from_runner_api_parameter(fn_parameter, unused_context):
+return FixedWindows(
+size=Duration(micros=fn_parameter['size']),
+offset=Timestamp(micros=fn_parameter['offset']))
+
+  def to_runner_api_parameter(self, context):
+return (urns.FIXED_WINDOWS_FN,
+proto_utils.pack_Struct(size=self.size.micros,
+offset=self.offset.micros))
+
+WindowFn.register_urn(
+urns.FIXED_WINDOWS_FN,
+struct_pb2.Struct,
+FixedWindows.from_runner_api_parameter)
+
 
 class SlidingWindows(WindowFn):
   """A windowing function 

[8/9] beam git commit: Move pipeline context and add more tests.

[robertwb]

Move pipeline context and add more tests.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/deff128f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/deff128f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/deff128f

Branch: refs/heads/master
Commit: deff128ff07ccc67956e1d5a94a64f2a31b224c8
Parents: b2da21e
Author: Robert Bradshaw 
Authored: Thu Mar 9 09:21:33 2017 -0800
Committer: Robert Bradshaw 
Committed: Thu Mar 9 20:29:02 2017 -0800

--
 sdks/python/apache_beam/coders/coders.py| 93 
 sdks/python/apache_beam/pipeline.py | 62 -
 .../apache_beam/runners/pipeline_context.py | 88 ++
 .../runners/pipeline_context_test.py| 49 +++
 sdks/python/apache_beam/transforms/core.py  |  1 +
 .../apache_beam/transforms/trigger_test.py  | 18 +---
 sdks/python/apache_beam/transforms/window.py|  2 +-
 .../apache_beam/transforms/window_test.py   |  6 +-
 sdks/python/apache_beam/utils/urns.py   |  2 +-
 9 files changed, 238 insertions(+), 83 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/deff128f/sdks/python/apache_beam/coders/coders.py
--
diff --git a/sdks/python/apache_beam/coders/coders.py 
b/sdks/python/apache_beam/coders/coders.py
index fd72af8..9f5a97a 100644
--- a/sdks/python/apache_beam/coders/coders.py
+++ b/sdks/python/apache_beam/coders/coders.py
@@ -266,6 +266,12 @@ class BytesCoder(FastCoder):
   def is_deterministic(self):
 return True
 
+  def __eq__(self, other):
+return type(self) == type(other)
+
+  def __hash__(self):
+return hash(type(self))
+
 
 class VarIntCoder(FastCoder):
   """Variable-length integer coder."""
@@ -276,6 +282,12 @@ class VarIntCoder(FastCoder):
   def is_deterministic(self):
 return True
 
+  def __eq__(self, other):
+return type(self) == type(other)
+
+  def __hash__(self):
+return hash(type(self))
+
 
 class FloatCoder(FastCoder):
   """A coder used for floating-point values."""
@@ -286,6 +298,12 @@ class FloatCoder(FastCoder):
   def is_deterministic(self):
 return True
 
+  def __eq__(self, other):
+return type(self) == type(other)
+
+  def __hash__(self):
+return hash(type(self))
+
 
 class TimestampCoder(FastCoder):
   """A coder used for timeutil.Timestamp values."""
@@ -296,6 +314,12 @@ class TimestampCoder(FastCoder):
   def is_deterministic(self):
 return True
 
+  def __eq__(self, other):
+return type(self) == type(other)
+
+  def __hash__(self):
+return hash(type(self))
+
 
 class SingletonCoder(FastCoder):
   """A coder that always encodes exactly one value."""
@@ -309,6 +333,12 @@ class SingletonCoder(FastCoder):
   def is_deterministic(self):
 return True
 
+  def __eq__(self, other):
+return type(self) == type(other) and self._value == other._value
+
+  def __hash__(self):
+return hash(self._value)
+
 
 def maybe_dill_dumps(o):
   """Pickle using cPickle or the Dill pickler as a fallback."""
@@ -365,6 +395,12 @@ class _PickleCoderBase(FastCoder):
   def value_coder(self):
 return self
 
+  def __eq__(self, other):
+return type(self) == type(other)
+
+  def __hash__(self):
+return hash(type(self))
+
 
 class PickleCoder(_PickleCoderBase):
   """Coder using Python's pickle functionality."""
@@ -446,6 +482,12 @@ class FastPrimitivesCoder(FastCoder):
   def value_coder(self):
 return self
 
+  def __eq__(self, other):
+return type(self) == type(other)
+
+  def __hash__(self):
+return hash(type(self))
+
 
 class Base64PickleCoder(Coder):
   """Coder of objects by Python pickle, then base64 encoding."""
@@ -503,6 +545,13 @@ class ProtoCoder(FastCoder):
 # a Map.
 return False
 
+  def __eq__(self, other):
+return (type(self) == type(other)
+and self.proto_message_type == other.proto_message_type)
+
+  def __hash__(self):
+return hash(self.proto_message_type)
+
   @staticmethod
   def from_type_hint(typehint, unused_registry):
 if issubclass(typehint, google.protobuf.message.Message):
@@ -563,6 +612,13 @@ class TupleCoder(FastCoder):
   def __repr__(self):
 return 'TupleCoder[%s]' % ', '.join(str(c) for c in self._coders)
 
+  def __eq__(self, other):
+return (type(self) == type(other)
+and self._coders == self._coders)
+
+  def __hash__(self):
+return hash(self._coders)
+
 
 class TupleSequenceCoder(FastCoder):
   """Coder of homogeneous tuple objects."""
@@ -586,6 +642,13 @@ class TupleSequenceCoder(FastCoder):
   def __repr__(self):
 return 'TupleSequenceCoder[%r]' % self._elem_coder
 
+  def __eq__(self, other):
+return (type(self) == type(other)
+and 

[3/9] beam git commit: Auto-generated runner api proto bindings.

[robertwb]

http://git-wip-us.apache.org/repos/asf/beam/blob/3bb125e1/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py 
b/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
new file mode 100644
index 000..66c331b
--- /dev/null
+++ b/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
@@ -0,0 +1,2755 @@
+# Generated by the protocol buffer compiler.  DO NOT EDIT!
+# source: beam_runner_api.proto
+
+import sys
+_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
+from google.protobuf.internal import enum_type_wrapper
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from google.protobuf import reflection as _reflection
+from google.protobuf import symbol_database as _symbol_database
+from google.protobuf import descriptor_pb2
+# @@protoc_insertion_point(imports)
+
+_sym_db = _symbol_database.Default()
+
+
+from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2
+
+
+DESCRIPTOR = _descriptor.FileDescriptor(
+  name='beam_runner_api.proto',
+  package='org.apache.beam.runner_api.v1',
+  syntax='proto3',
+  
serialized_pb=_b('\n\x15\x62\x65\x61m_runner_api.proto\x12\x1dorg.apache.beam.runner_api.v1\x1a\x19google/protobuf/any.proto\"\x8d\x07\n\nComponents\x12M\n\ntransforms\x18\x01
 
\x03(\x0b\x32\x39.org.apache.beam.runner_api.v1.Components.TransformsEntry\x12Q\n\x0cpcollections\x18\x02
 
\x03(\x0b\x32;.org.apache.beam.runner_api.v1.Components.PcollectionsEntry\x12`\n\x14windowing_strategies\x18\x03
 
\x03(\x0b\x32\x42.org.apache.beam.runner_api.v1.Components.WindowingStrategiesEntry\x12\x45\n\x06\x63oders\x18\x04
 
\x03(\x0b\x32\x35.org.apache.beam.runner_api.v1.Components.CodersEntry\x12Q\n\x0c\x65nvironments\x18\x05
 
\x03(\x0b\x32;.org.apache.beam.runner_api.v1.Components.EnvironmentsEntry\x1a\\\n\x0fTransformsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x38\n\x05value\x18\x02 
\x01(\x0b\x32).org.apache.beam.runner_api.v1.PTransform:\x02\x38\x01\x1a_\n\x11PcollectionsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x39\n\x05value\x18\x02 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.PCollection:\x02\
 x38\x01\x1al\n\x18WindowingStrategiesEntry\x12\x0b\n\x03key\x18\x01 
\x01(\t\x12?\n\x05value\x18\x02 
\x01(\x0b\x32\x30.org.apache.beam.runner_api.v1.WindowingStrategy:\x02\x38\x01\x1aS\n\x0b\x43odersEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x33\n\x05value\x18\x02 
\x01(\x0b\x32$.org.apache.beam.runner_api.v1.Coder:\x02\x38\x01\x1a_\n\x11\x45nvironmentsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\x39\n\x05value\x18\x02 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.Environment:\x02\x38\x01\"\xe4\x06\n\x15MessageWithComponents\x12=\n\ncomponents\x18\x01
 
\x01(\x0b\x32).org.apache.beam.runner_api.v1.Components\x12\x35\n\x05\x63oder\x18\x02
 
\x01(\x0b\x32$.org.apache.beam.runner_api.v1.CoderH\x00\x12H\n\x0f\x63ombine_payload\x18\x03
 
\x01(\x0b\x32-.org.apache.beam.runner_api.v1.CombinePayloadH\x00\x12\x44\n\rfunction_spec\x18\x04
 
\x01(\x0b\x32+.org.apache.beam.runner_api.v1.FunctionSpecH\x00\x12\x45\n\x0epar_do_payload\x18\x06
 \x01(\x0b\x32+.org.apache.beam.runner_api.v1.ParDoPayloadH\x00\x12?\
 n\nptransform\x18\x07 
\x01(\x0b\x32).org.apache.beam.runner_api.v1.PTransformH\x00\x12\x41\n\x0bpcollection\x18\x08
 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.PCollectionH\x00\x12\x42\n\x0cread_payload\x18\t
 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.ReadPayloadH\x00\x12>\n\nside_input\x18\x0b
 
\x01(\x0b\x32(.org.apache.beam.runner_api.v1.SideInputH\x00\x12O\n\x13window_into_payload\x18\x0c
 
\x01(\x0b\x32\x30.org.apache.beam.runner_api.v1.WindowIntoPayloadH\x00\x12N\n\x12windowing_strategy\x18\r
 
\x01(\x0b\x32\x30.org.apache.beam.runner_api.v1.WindowingStrategyH\x00\x12M\n\x12urn_with_parameter\x18\x0e
 
\x01(\x0b\x32/.org.apache.beam.runner_api.v1.UrnWithParameterH\x00\x42\x06\n\x04root\"\xa6\x01\n\x08Pipeline\x12=\n\ncomponents\x18\x01
 
\x01(\x0b\x32).org.apache.beam.runner_api.v1.Components\x12\x19\n\x11root_transform_id\x18\x02
 \x01(\t\x12@\n\x0c\x64isplay_data\x18\x03 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.DisplayData\"\xa8\x03\n\nPTransform\x12\x13\n\x0bunique_name\x18\x05
  \x01(\t\x12=\n\x04spec\x18\x01 
\x01(\x0b\x32/.org.apache.beam.runner_api.v1.UrnWithParameter\x12\x15\n\rsubtransforms\x18\x02
 \x03(\t\x12\x45\n\x06inputs\x18\x03 
\x03(\x0b\x32\x35.org.apache.beam.runner_api.v1.PTransform.InputsEntry\x12G\n\x07outputs\x18\x04
 
\x03(\x0b\x32\x36.org.apache.beam.runner_api.v1.PTransform.OutputsEntry\x12@\n\x0c\x64isplay_data\x18\x06
 
\x01(\x0b\x32*.org.apache.beam.runner_api.v1.DisplayData\x1a-\n\x0bInputsEntry\x12\x0b\n\x03key\x18\x01
 \x01(\t\x12\r\n\x05value\x18\x02 
\x01(\t:\x02\x38\x01\x1a.\n\x0cOutputsEntry\x12\x0b\n\x03key\x18\x01 
\x01(\t\x12\r\n\x05value\x18\x02 

[1/9] beam git commit: Runner API context helper classes.

[robertwb]

Repository: beam
Updated Branches:
  refs/heads/master f13a84d67 -> 2c2424cb4


Runner API context helper classes.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/bc76a186
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/bc76a186
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/bc76a186

Branch: refs/heads/master
Commit: bc76a186099568ef292ceb007388ae7174150bc2
Parents: 3bb125e
Author: Robert Bradshaw 
Authored: Tue Mar 7 12:04:27 2017 -0800
Committer: Robert Bradshaw 
Committed: Thu Mar 9 20:29:00 2017 -0800

--
 sdks/python/apache_beam/pipeline.py | 62 
 1 file changed, 62 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/bc76a186/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 7db39a9..4ec2e47 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -52,11 +52,14 @@ import os
 import shutil
 import tempfile
 
+from apache_beam import coders
 from apache_beam import pvalue
 from apache_beam import typehints
 from apache_beam.internal import pickler
 from apache_beam.runners import create_runner
 from apache_beam.runners import PipelineRunner
+from apache_beam.runners.api import beam_runner_api_pb2
+from apache_beam.transforms import core
 from apache_beam.transforms import ptransform
 from apache_beam.typehints import TypeCheckError
 from apache_beam.utils.pipeline_options import PipelineOptions
@@ -440,3 +443,62 @@ class AppliedPTransform(object):
 if v not in visited:
   visited.add(v)
   visitor.visit_value(v, self)
+
+
+class PipelineContextMap(object):
+  """This is a bi-directional map between objects and ids.
+
+  Under the hood it encodes and decodes these objects into runner API
+  representations.
+  """
+  def __init__(self, context, obj_type, proto_map=None):
+self._pipeline_context = context
+self._obj_type = obj_type
+self._obj_to_id = {}
+self._id_to_obj = {}
+self._id_to_proto = proto_map if proto_map else {}
+self._counter = 0
+
+  def _unique_ref(self):
+self._counter += 1
+return "ref_%s_%s" % (self._obj_type.__name__, self._counter)
+
+  def populate_map(self, proto_map):
+for id, obj in self._id_to_obj:
+  proto_map[id] = self._id_to_proto[id]
+
+  def get_id(self, obj):
+if obj not in self._obj_to_id:
+  id = self._unique_ref()
+  self._id_to_obj[id] = obj
+  self._obj_to_id[obj] = id
+  self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
+return self._obj_to_id[obj]
+
+  def get_by_id(self, id):
+if id not in self._id_to_obj:
+  self._id_to_obj[id] = self._obj_type.from_runner_api(
+self._id_to_proto[id], self._pipeline_context)
+return self._id_to_obj[id]
+
+
+class PipelineContext(object):
+
+  _COMPONENT_TYPES = {
+'transforms': AppliedPTransform,
+'pcollections': pvalue.PCollection,
+'coders': coders.Coder,
+'windowing_strategies': core.Windowing,
+# TODO: environment
+  }
+
+  def __init__(self, context_proto=None):
+for name, cls in self._COMPONENT_TYPES.items():
+  setattr(self, name,
+  PipelineContextMap(self, cls, getattr(context_proto, name, 
None)))
+
+  def to_runner_api(self):
+context_proto = beam_runner_api_pb2.Components()
+for name, cls in self._COMPONENT_TYEPS:
+  getattr(self, name).populate_map(getattr(context_proto, name))
+return context_proto

[6/9] beam git commit: Runner API translation of triggers and windowing strategies.

[robertwb]

Runner API translation of triggers and windowing strategies.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/5b86e1fc
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/5b86e1fc
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/5b86e1fc

Branch: refs/heads/master
Commit: 5b86e1fc22234a7a6dd00696326fa0fae8fe7a2d
Parents: aad32b7
Author: Robert Bradshaw 
Authored: Tue Mar 7 16:18:02 2017 -0800
Committer: Robert Bradshaw 
Committed: Thu Mar 9 20:29:01 2017 -0800

--
 sdks/python/apache_beam/coders/coders.py|  20 +++
 sdks/python/apache_beam/pipeline.py |   2 +-
 sdks/python/apache_beam/transforms/core.py  |  38 +
 sdks/python/apache_beam/transforms/trigger.py   | 143 ++-
 .../apache_beam/transforms/trigger_test.py  |  33 +
 sdks/python/apache_beam/transforms/window.py|  34 +++--
 .../apache_beam/transforms/window_test.py   |  23 ++-
 7 files changed, 272 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/5b86e1fc/sdks/python/apache_beam/coders/coders.py
--
diff --git a/sdks/python/apache_beam/coders/coders.py 
b/sdks/python/apache_beam/coders/coders.py
index 1d29f32..fd72af8 100644
--- a/sdks/python/apache_beam/coders/coders.py
+++ b/sdks/python/apache_beam/coders/coders.py
@@ -22,6 +22,8 @@ import cPickle as pickle
 import google.protobuf
 
 from apache_beam.coders import coder_impl
+from apache_beam.utils import urns
+from apache_beam.utils import proto_utils
 
 # pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports
 try:
@@ -182,6 +184,24 @@ class Coder(object):
 and self._dict_without_impl() == other._dict_without_impl())
 # pylint: enable=protected-access
 
+  def to_runner_api(self, context):
+# TODO(BEAM-115): Use specialized URNs and components.
+from apache_beam.runners.api import beam_runner_api_pb2
+return beam_runner_api_pb2.Coder(
+spec=beam_runner_api_pb2.FunctionSpec(
+spec=beam_runner_api_pb2.UrnWithParameter(
+urn=urns.PICKLED_CODER,
+parameter=proto_utils.pack_Any(
+google.protobuf.wrappers_pb2.BytesValue(
+value=serialize_coder(self))
+
+  @staticmethod
+  def from_runner_api(proto, context):
+any_proto = proto.spec.spec.parameter
+bytes_proto = google.protobuf.wrappers_pb2.BytesValue()
+any_proto.Unpack(bytes_proto)
+return deserialize_coder(bytes_proto.value)
+
 
 class StrUtf8Coder(Coder):
   """A coder used for reading and writing strings as UTF-8."""

http://git-wip-us.apache.org/repos/asf/beam/blob/5b86e1fc/sdks/python/apache_beam/pipeline.py
--
diff --git a/sdks/python/apache_beam/pipeline.py 
b/sdks/python/apache_beam/pipeline.py
index 4ec2e47..9edcf9b 100644
--- a/sdks/python/apache_beam/pipeline.py
+++ b/sdks/python/apache_beam/pipeline.py
@@ -499,6 +499,6 @@ class PipelineContext(object):
 
   def to_runner_api(self):
 context_proto = beam_runner_api_pb2.Components()
-for name, cls in self._COMPONENT_TYEPS:
+for name, cls in self._COMPONENT_TYPES:
   getattr(self, name).populate_map(getattr(context_proto, name))
 return context_proto

http://git-wip-us.apache.org/repos/asf/beam/blob/5b86e1fc/sdks/python/apache_beam/transforms/core.py
--
diff --git a/sdks/python/apache_beam/transforms/core.py 
b/sdks/python/apache_beam/transforms/core.py
index 7abd784..1fc63b2 100644
--- a/sdks/python/apache_beam/transforms/core.py
+++ b/sdks/python/apache_beam/transforms/core.py
@@ -27,6 +27,7 @@ from apache_beam import pvalue
 from apache_beam import typehints
 from apache_beam.coders import typecoders
 from apache_beam.internal import util
+from apache_beam.runners.api import beam_runner_api_pb2
 from apache_beam.transforms import ptransform
 from apache_beam.transforms.display import HasDisplayData, DisplayDataItem
 from apache_beam.transforms.ptransform import PTransform
@@ -49,6 +50,7 @@ from apache_beam.typehints import WithTypeHints
 from apache_beam.typehints.trivial_inference import element_type
 from apache_beam.utils.pipeline_options import TypeOptions
 
+
 # Type variables
 T = typehints.TypeVariable('T')
 K = typehints.TypeVariable('K')
@@ -1207,9 +1209,45 @@ class Windowing(object):
   self.accumulation_mode,
   self.output_time_fn)
 
+  def __eq__(self, other):
+if type(self) == type(other):
+  if self._is_default and other._is_default:
+return True
+  

[2/9] beam git commit: Auto-generated runner api proto bindings.

[robertwb]

http://git-wip-us.apache.org/repos/asf/beam/blob/3bb125e1/sdks/python/run_pylint.sh
--
diff --git a/sdks/python/run_pylint.sh b/sdks/python/run_pylint.sh
index afc5fb4..5e63856 100755
--- a/sdks/python/run_pylint.sh
+++ b/sdks/python/run_pylint.sh
@@ -34,7 +34,8 @@ EXCLUDED_GENERATED_FILES=(
 
"apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py"
 "apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py"
 "apache_beam/io/gcp/internal/clients/storage/storage_v1_messages.py"
-"apache_beam/coders/proto2_coder_test_messages_pb2.py")
+"apache_beam/coders/proto2_coder_test_messages_pb2.py"
+"apache_beam/runners/api/beam_runner_api_pb2.py")
 
 FILES_TO_IGNORE=""
 for file in "${EXCLUDED_GENERATED_FILES[@]}"; do

[9/9] beam git commit: Closes #2190

[robertwb]

Closes #2190


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2c2424cb
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2c2424cb
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2c2424cb

Branch: refs/heads/master
Commit: 2c2424cb44bb2976ea9099230106a639b5ee3993
Parents: f13a84d deff128
Author: Robert Bradshaw 
Authored: Thu Mar 9 20:29:03 2017 -0800
Committer: Robert Bradshaw 
Committed: Thu Mar 9 20:29:03 2017 -0800

--
 sdks/python/apache_beam/coders/coders.py|  113 +
 sdks/python/apache_beam/runners/api/__init__.py |   16 +
 .../runners/api/beam_runner_api_pb2.py  | 2772 ++
 .../apache_beam/runners/pipeline_context.py |   88 +
 .../runners/pipeline_context_test.py|   49 +
 sdks/python/apache_beam/transforms/core.py  |   39 +
 sdks/python/apache_beam/transforms/trigger.py   |  143 +-
 .../apache_beam/transforms/trigger_test.py  |   19 +
 sdks/python/apache_beam/transforms/window.py|  147 +-
 .../apache_beam/transforms/window_test.py   |   32 +
 sdks/python/apache_beam/utils/proto_utils.py|   54 +
 sdks/python/apache_beam/utils/urns.py   |   24 +
 sdks/python/run_pylint.sh   |3 +-
 13 files changed, 3481 insertions(+), 18 deletions(-)
--

[5/9] beam git commit: Add license to new files.

[robertwb]

Add license to new files.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/b2da21e2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/b2da21e2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/b2da21e2

Branch: refs/heads/master
Commit: b2da21e287660bb3077bf89e092f7aa3c385906b
Parents: 5b86e1f
Author: Robert Bradshaw 
Authored: Wed Mar 8 13:47:53 2017 -0800
Committer: Robert Bradshaw 
Committed: Thu Mar 9 20:29:01 2017 -0800

--
 sdks/python/apache_beam/runners/api/__init__.py| 16 
 .../apache_beam/runners/api/beam_runner_api_pb2.py | 17 +
 sdks/python/apache_beam/utils/proto_utils.py   | 17 +
 sdks/python/apache_beam/utils/urns.py  | 17 +
 4 files changed, 67 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/b2da21e2/sdks/python/apache_beam/runners/api/__init__.py
--
diff --git a/sdks/python/apache_beam/runners/api/__init__.py 
b/sdks/python/apache_beam/runners/api/__init__.py
index e69de29..cce3aca 100644
--- a/sdks/python/apache_beam/runners/api/__init__.py
+++ b/sdks/python/apache_beam/runners/api/__init__.py
@@ -0,0 +1,16 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#

http://git-wip-us.apache.org/repos/asf/beam/blob/b2da21e2/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
--
diff --git a/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py 
b/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
index 66c331b..f235ce8 100644
--- a/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
+++ b/sdks/python/apache_beam/runners/api/beam_runner_api_pb2.py
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
 # Generated by the protocol buffer compiler.  DO NOT EDIT!
 # source: beam_runner_api.proto
 

http://git-wip-us.apache.org/repos/asf/beam/blob/b2da21e2/sdks/python/apache_beam/utils/proto_utils.py
--
diff --git a/sdks/python/apache_beam/utils/proto_utils.py 
b/sdks/python/apache_beam/utils/proto_utils.py
index 0ece8f5..b4bfdca 100644
--- a/sdks/python/apache_beam/utils/proto_utils.py
+++ b/sdks/python/apache_beam/utils/proto_utils.py
@@ -1,3 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
 from google.protobuf import any_pb2
 from google.protobuf import struct_pb2
 

[GitHub] beam pull request #2215: [BEAM-1674] Exclude UsesTimersInParDo from Flink va...

[kennknowles]

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2215

[BEAM-1674] Exclude UsesTimersInParDo from Flink validation test suite

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aljoscha @JingsongLi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam flink-timers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2215.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2215


commit 43d91a05c4df395ff6146984cb5f0a8fa772f09b
Author: Kenneth Knowles 
Date:   2017-03-10T04:26:52Z

Exclude UsesTimersInParDo from Flink validation test suite




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (BEAM-1674) Flink user state GC depends on order of timer firing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904418#comment-15904418
 ] 

ASF GitHub Bot commented on BEAM-1674:
--

GitHub user kennknowles opened a pull request:

https://github.com/apache/beam/pull/2215

[BEAM-1674] Exclude UsesTimersInParDo from Flink validation test suite

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aljoscha @JingsongLi 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kennknowles/beam flink-timers

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2215.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2215


commit 43d91a05c4df395ff6146984cb5f0a8fa772f09b
Author: Kenneth Knowles 
Date:   2017-03-10T04:26:52Z

Exclude UsesTimersInParDo from Flink validation test suite




> Flink user state GC depends on order of timer firing
> 
>
> Key: BEAM-1674
> URL: https://issues.apache.org/jira/browse/BEAM-1674
> Project: Beam
>  Issue Type: Bug
>  Components: runner-flink
>Affects Versions: 0.5.0
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 0.6.0
>
>
> The newly added {{ParDoTest.testEventTimeTimerMultipleKeys()}} fails because 
> the {{DoFn}} sets a timer for {{window.maxTimestamp()}} which also happens to 
> be the GC timer for the user state. The Flink Runner uses timers to schedule 
> GC, the user-set timer and the GC timer have a different timer id, so they 
> don't clash. However, if the GC timer is being processed before the user 
> timer then the user doesn't have a chance to access the state anymore because 
> it will already be cleared out by the time the user timer is being processed. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1876

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904372#comment-15904372
 ] 

Kenneth Knowles commented on BEAM-1676:
---

If you can come up with an invocation that reproduces this, I am happy to 
debug. Based on the fact that there are different commands under the hood, 
perhaps the CLI will not be able to repro. It seems likely to have to do with 
classloaders.

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Issue Comment Deleted] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1676:
--
Comment: was deleted

(was: One thing I see in the logs is:

{code}
[beam] $ /home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn -f pom.xml 
-Dsha1=master "-Djdk=JDK 1.8 (latest)" -Dlabel=beam 
"-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_Version_Test/jdk/JDK
 1.8 (latest)/label/beam/.repository" -B -e -P release,dataflow-runner clean 
install coveralls:report -DrepoToken= -DskipITs=false 
'-DintegrationTestPipelineOptions=[ "--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]'
/home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn: 200: local: 1.8: bad 
variable name
{code}

I don't know if this causes a problem. I'm still scanning for anything else 
interesting.)

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.7 (latest),beam #16

[Apache Jenkins Server]

See 


Changes:

[klk] Bump Dataflow ROS timeout to 120 minutes

[davor] HadoopInputFormatIO with junits

--
[...truncated 473.62 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.295 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.058 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.062 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.088 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.004 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.095 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 7 (on Ubuntu only),beam #16

[Apache Jenkins Server]

See 


Changes:

[klk] Bump Dataflow ROS timeout to 120 minutes

[davor] HadoopInputFormatIO with junits

--
[...truncated 474.57 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.294 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.079 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.039 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.8 (latest),beam #16

[Apache Jenkins Server]

See 


Changes:

[klk] Bump Dataflow ROS timeout to 120 minutes

[davor] HadoopInputFormatIO with junits

--
[...truncated 471.83 KB...]
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.065 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.085 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.32 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.101 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.201 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running org.apache.beam.sdk.util.FluentBackoffTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.FluentBackoffTest
Running 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 8 (on Ubuntu only),beam #16

[Apache Jenkins Server]

See 


Changes:

[klk] Bump Dataflow ROS timeout to 120 minutes

[davor] HadoopInputFormatIO with junits

--
[...truncated 474.60 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.279 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.092 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.035 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.099 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904355#comment-15904355
 ] 

Kenneth Knowles commented on BEAM-1676:
---

One thing I see in the logs is:

{code}
[beam] $ /home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn -f pom.xml 
-Dsha1=master "-Djdk=JDK 1.8 (latest)" -Dlabel=beam 
"-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_Version_Test/jdk/JDK
 1.8 (latest)/label/beam/.repository" -B -e -P release,dataflow-runner clean 
install coveralls:report -DrepoToken= -DskipITs=false 
'-DintegrationTestPipelineOptions=[ "--project=apache-beam-testing", 
"--tempRoot=gs://temp-storage-for-end-to-end-tests", 
"--runner=org.apache.beam.runners.dataflow.testing.TestDataflowRunner" ]'
/home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn: 200: local: 1.8: bad 
variable name
{code}

I don't know if this causes a problem. I'm still scanning for anything else 
interesting.

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1647) Dataflow RunnableOnService tests timing out reliably

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904349#comment-15904349
 ] 

ASF GitHub Bot commented on BEAM-1647:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2197


> Dataflow RunnableOnService tests timing out reliably
> 
>
> Key: BEAM-1647
> URL: https://issues.apache.org/jira/browse/BEAM-1647
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow, testing
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Blocker
>
> Since [build 
> 2487|https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_RunnableOnService_Dataflow/2487/]
>  our build time seems to have pushed up past 1h40m and is timing out.
> It was just barely under that anyhow, so I don't expect this is a 
> catastrophic change but probably something small. Some builds (at HEAD) seem 
> to sneak under the limit.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] beam git commit: This closes #2197: Bump Dataflow ROS timeout to 120 minutes

[kenn]

This closes #2197: Bump Dataflow ROS timeout to 120 minutes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f13a84d6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f13a84d6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f13a84d6

Branch: refs/heads/master
Commit: f13a84d67e30dbf1a74df79ce964c18de850bf65
Parents: b79dd64 f117913
Author: Kenneth Knowles 
Authored: Thu Mar 9 19:31:58 2017 -0800
Committer: Kenneth Knowles 
Committed: Thu Mar 9 19:31:58 2017 -0800

--
 .jenkins/common_job_properties.groovy   | 5 +++--
 .../job_beam_PostCommit_Java_RunnableOnService_Dataflow.groovy  | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
--

[1/2] beam git commit: Bump Dataflow ROS timeout to 120 minutes

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master b79dd6428 -> f13a84d67


Bump Dataflow ROS timeout to 120 minutes


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/f1179137
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/f1179137
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/f1179137

Branch: refs/heads/master
Commit: f1179137bdb9e3df68ea9963b6d5b6759cb49f36
Parents: 37c17f2
Author: Kenneth Knowles 
Authored: Wed Mar 8 10:47:07 2017 -0800
Committer: Kenneth Knowles 
Committed: Wed Mar 8 14:24:13 2017 -0800

--
 .jenkins/common_job_properties.groovy   | 5 +++--
 .../job_beam_PostCommit_Java_RunnableOnService_Dataflow.groovy  | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/f1179137/.jenkins/common_job_properties.groovy
--
diff --git a/.jenkins/common_job_properties.groovy 
b/.jenkins/common_job_properties.groovy
index bcac19e..4506636 100644
--- a/.jenkins/common_job_properties.groovy
+++ b/.jenkins/common_job_properties.groovy
@@ -29,8 +29,9 @@ class common_job_properties {
 
   // Sets common top-level job properties for main repository jobs.
   static void setTopLevelMainJobProperties(context,
-   String defaultBranch = 'master') {
-setTopLevelJobProperties(context, 'beam', defaultBranch, 100)
+   String defaultBranch = 'master',
+   int defaultTimeout = 100) {
+setTopLevelJobProperties(context, 'beam', defaultBranch, defaultTimeout)
   }
 
   // Sets common top-level job properties. Accessed through one of the above

http://git-wip-us.apache.org/repos/asf/beam/blob/f1179137/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Dataflow.groovy
--
diff --git 
a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Dataflow.groovy 
b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Dataflow.groovy
index a3b8150..77c4864 100644
--- a/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Dataflow.groovy
+++ b/.jenkins/job_beam_PostCommit_Java_RunnableOnService_Dataflow.groovy
@@ -26,7 +26,7 @@ mavenJob('beam_PostCommit_Java_RunnableOnService_Dataflow') {
   previousNames('beam_PostCommit_RunnableOnService_GoogleCloudDataflow')
 
   // Set common parameters.
-  common_job_properties.setTopLevelMainJobProperties(delegate)
+  common_job_properties.setTopLevelMainJobProperties(delegate, 'master', 120)
 
   // Set maven parameters.
   common_job_properties.setMavenConfig(delegate)

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2859

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1875

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1040) Hadoop InputFormat - IO Transform for reads

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904158#comment-15904158
 ] 

ASF GitHub Bot commented on BEAM-1040:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1994


> Hadoop InputFormat - IO Transform for reads
> ---
>
> Key: BEAM-1040
> URL: https://issues.apache.org/jira/browse/BEAM-1040
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Stephen Sisk
>Assignee: Dipti Kulkarni
>
> We should build a IO read transform that will read data from services 
> supporting the Hadoop InputFormat  interface [1]
> This will make it easy to connect to a variety of databases while still 
> providing some aspects of scalability since the InputFormat interface 
> provides for parallel reading. 
> [1] 
> https://hadoop.apache.org/docs/r2.7.3/api/org/apache/hadoop/mapred/InputFormat.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #1994: BEAM-1040 Hadoop InputFormat - IO Transform for rea...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/1994


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[2/3] beam git commit: HadoopInputFormatIO with junits

[davor]

HadoopInputFormatIO with junits


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/d01620c0
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/d01620c0
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/d01620c0

Branch: refs/heads/master
Commit: d01620c05edd947bd1c55570f683101323cdec5d
Parents: ef256df
Author: Radhika S Kulkarni 
Authored: Mon Mar 6 19:41:45 2017 +0530
Committer: Davor Bonaci 
Committed: Thu Mar 9 15:20:31 2017 -0800

--
 sdks/java/io/hadoop-input-format/README.md  | 167 
 sdks/java/io/hadoop-input-format/pom.xml| 136 +++
 .../hadoop/inputformat/HadoopInputFormatIO.java | 941 +++
 .../sdk/io/hadoop/inputformat/package-info.java |  23 +
 .../ConfigurableEmployeeInputFormat.java| 131 +++
 .../sdk/io/hadoop/inputformat/Employee.java |  85 ++
 .../hadoop/inputformat/EmployeeInputFormat.java | 172 
 .../inputformat/HadoopInputFormatIOTest.java| 844 +
 .../ReuseObjectsEmployeeInputFormat.java| 176 
 .../hadoop/inputformat/TestEmployeeDataSet.java |  76 ++
 10 files changed, 2751 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/d01620c0/sdks/java/io/hadoop-input-format/README.md
--
diff --git a/sdks/java/io/hadoop-input-format/README.md 
b/sdks/java/io/hadoop-input-format/README.md
new file mode 100644
index 000..d91f019
--- /dev/null
+++ b/sdks/java/io/hadoop-input-format/README.md
@@ -0,0 +1,167 @@
+
+
+# Hadoop InputFormat IO
+
+A HadoopInputFormatIO is a Transform for reading data from any source which
+implements Hadoop InputFormat. For example- Cassandra, Elasticsearch, HBase, 
Redis, Postgres, etc.
+
+HadoopInputFormatIO has to make several performance trade-offs in connecting 
to InputFormat, so if there is another Beam IO Transform specifically for 
connecting to your data source of choice, we would recommend using that one, 
but this IO Transform allows you to connect to many data sources that do not 
yet have a Beam IO Transform.
+
+You will need to pass a Hadoop Configuration with parameters specifying how 
the read will occur. Many properties of the Configuration are optional, and 
some are required for certain InputFormat classes, but the following properties 
must be set for all InputFormats:
+
+mapreduce.job.inputformat.class: The InputFormat class used to connect to your 
data source of choice.
+key.class: The key class returned by the InputFormat in 
'mapreduce.job.inputformat.class'.
+value.class: The value class returned by the InputFormat in 
'mapreduce.job.inputformat.class'.
+
+For example:
+```java
+Configuration myHadoopConfiguration = new Configuration(false);
+// Set Hadoop InputFormat, key and value class in configuration
+myHadoopConfiguration.setClass("mapreduce.job.inputformat.class", 
InputFormatClass,
+  InputFormat.class);
+myHadoopConfiguration.setClass("key.class", InputFormatKeyClass, Object.class);
+myHadoopConfiguration.setClass("value.class", InputFormatValueClass, 
Object.class);
+```
+
+You will need to check to see if the key and value classes output by the 
InputFormat have a Beam Coder available. If not, You can use 
withKeyTranslation/withValueTranslation to specify a method transforming 
instances of those classes into another class that is supported by a Beam 
Coder. These settings are optional and you don't need to specify translation 
for both key and value.
+
+For example:
+```java
+SimpleFunction myOutputKeyType =
+new SimpleFunction() {
+  public MyKeyClass apply(InputFormatKeyClass input) {
+  // ...logic to transform InputFormatKeyClass to MyKeyClass
+  }
+};
+SimpleFunction myOutputValueType =
+new SimpleFunction() {
+  public MyValueClass apply(InputFormatValueClass input) {
+  // ...logic to transform InputFormatValueClass to MyValueClass
+  }
+};
+```
+
+### Reading using Hadoop InputFormat IO
+Pipeline p = ...; // Create pipeline.
+// Read data only with Hadoop configuration.
+
+```java
+p.apply("read",
+  HadoopInputFormatIO.read()
+  .withConfiguration(myHadoopConfiguration);
+```
+
+// Read data with configuration and key translation (Example scenario: Beam 
Coder is not
+available for key class hence key translation is required.).
+
+```java
+p.apply("read",
+  HadoopInputFormatIO.read()
+  .withConfiguration(myHadoopConfiguration)
+  .withKeyTranslation(myOutputKeyType);
+```
+
+// Read data with configuration and value translation (Example scenario: Beam 
Coder is not
+available for value 

[1/3] beam git commit: HadoopInputFormatIO with junits

[davor]

Repository: beam
Updated Branches:
  refs/heads/master ef256df18 -> b79dd6428


http://git-wip-us.apache.org/repos/asf/beam/blob/d01620c0/sdks/java/io/hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
--
diff --git 
a/sdks/java/io/hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
 
b/sdks/java/io/hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
new file mode 100644
index 000..c25cf51
--- /dev/null
+++ 
b/sdks/java/io/hadoop-input-format/src/test/java/org/apache/beam/sdk/io/hadoop/inputformat/HadoopInputFormatIOTest.java
@@ -0,0 +1,844 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more 
contributor license
+ * agreements. See the NOTICE file distributed with this work for additional 
information regarding
+ * copyright ownership. The ASF licenses this file to you under the Apache 
License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the 
License. You may obtain a
+ * copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software 
distributed under the License
+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY 
KIND, either express
+ * or implied. See the License for the specific language governing permissions 
and limitations under
+ * the License.
+ */
+package org.apache.beam.sdk.io.hadoop.inputformat;
+
+import static 
org.apache.beam.sdk.transforms.display.DisplayDataMatchers.hasDisplayItem;
+import static org.hamcrest.Matchers.containsInAnyOrder;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertThat;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map.Entry;
+
+import org.apache.beam.sdk.Pipeline.PipelineExecutionException;
+import org.apache.beam.sdk.coders.AvroCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.io.BoundedSource;
+import org.apache.beam.sdk.io.BoundedSource.BoundedReader;
+import org.apache.beam.sdk.io.hadoop.WritableCoder;
+import 
org.apache.beam.sdk.io.hadoop.inputformat.EmployeeInputFormat.EmployeeRecordReader;
+import 
org.apache.beam.sdk.io.hadoop.inputformat.EmployeeInputFormat.NewObjectsEmployeeInputSplit;
+import 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIO.HadoopInputFormatBoundedSource;
+import 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIO.SerializableConfiguration;
+import 
org.apache.beam.sdk.io.hadoop.inputformat.HadoopInputFormatIO.SerializableSplit;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.SourceTestUtils;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.LongWritable;
+import org.apache.hadoop.io.MapWritable;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapreduce.InputFormat;
+import org.apache.hadoop.mapreduce.InputSplit;
+import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.ExpectedException;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.mockito.Mockito;
+
+/**
+ * Unit tests for {@link HadoopInputFormatIO}.
+ */
+@RunWith(JUnit4.class)
+public class HadoopInputFormatIOTest {
+  static SerializableConfiguration serConf;
+  static SimpleFunction myKeyTranslate;
+  static SimpleFunction myValueTranslate;
+
+  @Rule public final transient TestPipeline p = TestPipeline.create();
+  @Rule public ExpectedException thrown = ExpectedException.none();
+
+  private PBegin input = PBegin.in(p);
+
+  @BeforeClass
+  public static void setUp() throws IOException, InterruptedException {
+serConf = loadTestConfiguration(
+  EmployeeInputFormat.class,
+  Text.class,
+  Employee.class);
+myKeyTranslate = new SimpleFunction() {
+  @Override
+  public String apply(Text input) {
+return input.toString();
+  }
+};
+myValueTranslate = new SimpleFunction() {
+  @Override
+  public String apply(Employee input) {
+return input.getEmpName() + "_" + input.getEmpAddress();
+  }
+};
+  }
+
+  @Test
+  public void testReadBuildsCorrectly() {
+

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 7 (on Ubuntu only),beam #15

[Apache Jenkins Server]

See 


--
[...truncated 474.48 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.413 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.074 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.096 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.212 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.112 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.162 sec - in 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.7 (latest),beam #15

[Apache Jenkins Server]

See 


--
[...truncated 473.60 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.301 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.068 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.102 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.252 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.186 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.8 (latest),beam #15

[Apache Jenkins Server]

See 


--
[...truncated 472.21 KB...]
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.156 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.122 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.169 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running org.apache.beam.sdk.util.FluentBackoffTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.FluentBackoffTest
Running org.apache.beam.sdk.util.IntervalBoundedExponentialBackOffTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 8 (on Ubuntu only),beam #15

[Apache Jenkins Server]

See 


--
[...truncated 474.56 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.323 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.051 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.035 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.119 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.101 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.168 sec - in 

[GitHub] beam-site pull request #170: Fix possible erranous naming conventions refere...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam-site/pull/170


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[3/3] beam-site git commit: This closes #170

[davor]

This closes #170


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/8b046d48
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/8b046d48
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/8b046d48

Branch: refs/heads/asf-site
Commit: 8b046d48ac663abb2699b57ffe783042ab3bea2d
Parents: 7b78cc3 07748b5
Author: Davor Bonaci 
Authored: Thu Mar 9 16:29:15 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Mar 9 16:29:15 2017 -0800

--
 content/contribute/ptransform-style-guide/index.html | 2 +-
 src/contribute/ptransform-style-guide.md | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--

[1/3] beam-site git commit: Fix possible erranous naming conventions reference

[davor]

Repository: beam-site
Updated Branches:
  refs/heads/asf-site 7b78cc342 -> 8b046d48a


Fix possible erranous naming conventions reference


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/9cdf959c
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/9cdf959c
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/9cdf959c

Branch: refs/heads/asf-site
Commit: 9cdf959c9a6fa18d1401b03fdd6815986e3b6030
Parents: 7b78cc3
Author: Zur, Aviem 
Authored: Sat Mar 4 14:01:45 2017 +0200
Committer: Davor Bonaci 
Committed: Thu Mar 9 16:26:27 2017 -0800

--
 src/contribute/ptransform-style-guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/9cdf959c/src/contribute/ptransform-style-guide.md
--
diff --git a/src/contribute/ptransform-style-guide.md 
b/src/contribute/ptransform-style-guide.md
index 583f264..8bcb2d9 100644
--- a/src/contribute/ptransform-style-guide.md
+++ b/src/contribute/ptransform-style-guide.md
@@ -41,7 +41,7 @@ Do not:
 
 Do:
 
-* Respect language-specific naming conventions, e.g. name classes in 
`CamelCase` in Java and Python, functions in `snakeCase` in Java but 
`with_underscores` in Python, etc.
+* Respect language-specific naming conventions, e.g. name classes in 
`PascalCase` in Java and Python, functions in `camelCase` in Java but 
`snake_case` in Python, etc.
 * Name factory functions so that either the function name is a verb, or 
referring to the transform reads like a verb: e.g. `MongoDbIO.read()`, 
`Flatten.iterables()`.
 * In typed languages, name `PTransform` classes also like verbs (e.g.: 
`MongoDbIO.Read`, `Flatten.Iterables`).
 * Name families of transforms for interacting with a storage system using the 
word "IO": `MongoDbIO`, `JdbcIO`.

[2/3] beam-site git commit: Regenerate website

[davor]

Regenerate website


Project: http://git-wip-us.apache.org/repos/asf/beam-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam-site/commit/07748b57
Tree: http://git-wip-us.apache.org/repos/asf/beam-site/tree/07748b57
Diff: http://git-wip-us.apache.org/repos/asf/beam-site/diff/07748b57

Branch: refs/heads/asf-site
Commit: 07748b578f352ed0a84f244b8f37f7928df27d1f
Parents: 9cdf959
Author: Davor Bonaci 
Authored: Thu Mar 9 16:29:14 2017 -0800
Committer: Davor Bonaci 
Committed: Thu Mar 9 16:29:14 2017 -0800

--
 content/contribute/ptransform-style-guide/index.html | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam-site/blob/07748b57/content/contribute/ptransform-style-guide/index.html
--
diff --git a/content/contribute/ptransform-style-guide/index.html 
b/content/contribute/ptransform-style-guide/index.html
index 3006e9e..9e61954 100644
--- a/content/contribute/ptransform-style-guide/index.html
+++ b/content/contribute/ptransform-style-guide/index.html
@@ -240,7 +240,7 @@ As a rule of thumb: expose these if you anticipate that the 
full packaged Do:
 
 
-  Respect language-specific naming conventions, e.g. name classes in CamelCase in Java and Python, functions in 
snakeCase in Java but with_underscores in Python, etc.
+  Respect language-specific naming conventions, e.g. name classes in PascalCase in Java and Python, functions in 
camelCase in Java but snake_case in Python, etc.
   Name factory functions so that either the function name is a verb, or 
referring to the transform reads like a verb: e.g. MongoDbIO.read(), Flatten.iterables().
   In typed languages, name PTransform classes also like verbs (e.g.: 
MongoDbIO.Read, Flatten.Iterables).
   Name families of transforms for interacting with a storage system using 
the word “IO”: MongoDbIO, JdbcIO.

[GitHub] beam pull request #2213: Increase timeout for site tests

[sb2nov]

Github user sb2nov closed the pull request at:

https://github.com/apache/beam/pull/2213


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] beam pull request #2214: Ensure that assert_that takes a PCollection as its ...

[robertwb]

GitHub user robertwb opened a pull request:

https://github.com/apache/beam/pull/2214

Ensure that assert_that takes a PCollection as its first argument.

This can avoid (silent) error such as

with test_pipeline.TestPipeline() as p:
  bad_value = "Create0" >> beam.Create([1])  # Note missing "p | "
  beam_util.assert_that(bad_value, beam_util.equal_to([0]))

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/robertwb/incubator-beam assert-that

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2214.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2214


commit 479cc0181daa3b98ef6ccdc8c3c1512bc2e3efae
Author: Robert Bradshaw 
Date:   2017-03-10T00:18:46Z

Ensure that assert_that takes a PCollection as its first argument.

This can avoid (silent) error such as

with test_pipeline.TestPipeline() as p:
  bad_value = "Create0" >> beam.Create([1])  # Note missing "p | "
  beam_util.assert_that(bad_value, beam_util.equal_to([0]))




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Issue Comment Deleted] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Comment: was deleted

(was: [~kenn] any thoughts?)

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904113#comment-15904113
 ] 

Mark Liu commented on BEAM-1676:


[~kenn] any thoughts?

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Davor Bonaci (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904103#comment-15904103
 ] 

Davor Bonaci commented on BEAM-1676:


[~kenn], any ideas perhaps?

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904078#comment-15904078
 ] 

Mark Liu commented on BEAM-1676:


two discovers from job logs:
1. failed version test uses matrixJob.steps.maven() to run maven command, 
instead of mavenJob() directly. After compared job logs, it looks like Jenkins 
uses different build mechanism to start maven job.
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/jdk=JDK%201.8%20(latest),label=beam/consoleFull
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/2847/consoleFull

2. Found this error right after maven command got executed:
/home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn: 200: local: 1.8: bad 
variable name
JDK 1.8 (latest) is used in "-Djdk=JDK 1.8 (latest)" and 
"-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_Version_Test/jdk/JDK
 1.8 (latest)/label/beam/.repository" in execution command, which might be the 
reason causing test failure?

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15904078#comment-15904078
 ] 

Mark Liu edited comment on BEAM-1676 at 3/9/17 11:49 PM:
-

two discovers from job logs:
1. failed version test uses matrixJob.steps.maven() to run maven command, 
instead of mavenJob() directly. After compared job logs, it looks like Jenkins 
uses different build mechanism to start maven job.

2. Found this error right after maven command got executed:
/home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn: 200: local: 1.8: bad 
variable name
JDK 1.8 (latest) is used in "-Djdk=JDK 1.8 (latest)" and 
"-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_Version_Test/jdk/JDK
 1.8 (latest)/label/beam/.repository" in execution command, which might be the 
reason causing test failure?

https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/jdk=JDK%201.8%20(latest),label=beam/consoleFull
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/2847/consoleFull



was (Author: markflyhigh):
two discovers from job logs:
1. failed version test uses matrixJob.steps.maven() to run maven command, 
instead of mavenJob() directly. After compared job logs, it looks like Jenkins 
uses different build mechanism to start maven job.
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/jdk=JDK%201.8%20(latest),label=beam/consoleFull
https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/2847/consoleFull

2. Found this error right after maven command got executed:
/home/jenkins/tools/maven/apache-maven-3.3.3/bin/mvn: 200: local: 1.8: bad 
variable name
JDK 1.8 (latest) is used in "-Djdk=JDK 1.8 (latest)" and 
"-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_Version_Test/jdk/JDK
 1.8 (latest)/label/beam/.repository" in execution command, which might be the 
reason causing test failure?

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in 
> good health. And the maven command in version test is the same as 
> beam_PostCommit_Java_MavenInstall.
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2213: Increase timeout for site tests

[sb2nov]

GitHub user sb2nov opened a pull request:

https://github.com/apache/beam/pull/2213

Increase timeout for site tests

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

 - [ ] Make sure the PR title is formatted like:
   `[BEAM-] Description of pull request`
 - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
   Travis-CI on your fork and ensure the whole test matrix passes).
 - [ ] Replace `` in the title with the actual Jira issue
   number, if there is one.
 - [ ] If this contribution is large, please file an Apache
   [Individual Contributor License 
Agreement](https://www.apache.org/licenses/icla.txt).

---

R: @aaltay PTAL

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sb2nov/beam Bump-site-test-timeout

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/2213.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2213


commit 71aa70eb7669fa708d8f373654a348b41fa2e788
Author: Sourabh Bajaj 
Date:   2017-03-09T23:45:05Z

Increase timeout for site tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Summary: SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins  
(was: SdkCoreApiSurfaceTest Failed in JDK7 and OpenJDK7&8 on Jenkins)

> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: https://issues.apache.org/jira/browse/BEAM-1676
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Mark Liu
>Assignee: Davor Bonaci
>
> After running beam_PostCommit_Java_MavenInstall in different JDK versions 
> (JDK7, OpenJDK7&8) on Jenkins, all failed due to following error:
> {code}
> Failed tests: 
>   SdkCoreApiSurfaceTest.testSdkApiSurface:59 
> Expected: API surface to include only:
>   Classes in package "org.apache.beam"
>   Classes in package "com.google.api.client"
>   Classes in package "com.google.api.services.bigquery"
>   Classes in package "com.google.api.services.cloudresourcemanager"
>   Classes in package "com.google.api.services.pubsub"
>   Classes in package "com.google.api.services.storage"
>   Classes in package "com.google.auth"
>   Classes in package "com.google.protobuf"
>   Classes in package "com.fasterxml.jackson.annotation"
>   Classes in package "com.fasterxml.jackson.core"
>   Classes in package "com.fasterxml.jackson.databind"
>   Classes in package "org.apache.avro"
>   Classes in package "org.hamcrest"
>   Classes in package "org.codehaus.jackson"
>   Classes in package "org.joda.time"
>   Classes in package "org.junit"
>   
>  but: The following white-listed scopes did not have matching classes on 
> the API surface:
>   No Classes in package "com.fasterxml.jackson.annotation"
>   No Classes in package "com.fasterxml.jackson.core"
>   No Classes in package "com.fasterxml.jackson.databind"
>   No Classes in package "com.google.api.client"
>   No Classes in package "com.google.api.services.bigquery"
>   No Classes in package "com.google.api.services.cloudresourcemanager"
>   No Classes in package "com.google.api.services.pubsub"
>   No Classes in package "com.google.api.services.storage"
>   No Classes in package "com.google.auth"
>   No Classes in package "com.google.protobuf"
>   No Classes in package "org.apache.avro"
>   No Classes in package "org.apache.beam"
>   No Classes in package "org.codehaus.jackson"
>   No Classes in package "org.hamcrest"
>   No Classes in package "org.joda.time"
>   No Classes in package "org.junit"
> {code}
> Job link:
> https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/11/
> Multi-JDK version test is based on this PR:
> https://github.com/apache/beam/pull/2204/files
> Any ideas?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Description: 
After running beam_PostCommit_Java_MavenInstall in different JDK versions 
(JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:

{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/14/

Multi-JDK version test is based on this PR:
https://github.com/apache/beam/pull/2204/files

Our beam_PostCommit_Java_MavenInstall is using JDK 1.8 (latest), which in good 
health. And the maven command in version test is the same as 
beam_PostCommit_Java_MavenInstall.
Any ideas?

  was:
After running beam_PostCommit_Java_MavenInstall in different JDK versions 
(JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:

{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/11/

Multi-JDK version test is based on this PR:
https://github.com/apache/beam/pull/2204/files

Any ideas?


> SdkCoreApiSurfaceTest Failed in JDK7&8 and 

[jira] [Updated] (BEAM-1676) SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins

[Mark Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Liu updated BEAM-1676:
---
Description: 
After running beam_PostCommit_Java_MavenInstall in different JDK versions 
(JDK7&8, OpenJDK7&8) on Jenkins, all failed due to following error:

{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/11/

Multi-JDK version test is based on this PR:
https://github.com/apache/beam/pull/2204/files

Any ideas?

  was:
After running beam_PostCommit_Java_MavenInstall in different JDK versions 
(JDK7, OpenJDK7&8) on Jenkins, all failed due to following error:

{code}
Failed tests: 
  SdkCoreApiSurfaceTest.testSdkApiSurface:59 
Expected: API surface to include only:
Classes in package "org.apache.beam"
Classes in package "com.google.api.client"
Classes in package "com.google.api.services.bigquery"
Classes in package "com.google.api.services.cloudresourcemanager"
Classes in package "com.google.api.services.pubsub"
Classes in package "com.google.api.services.storage"
Classes in package "com.google.auth"
Classes in package "com.google.protobuf"
Classes in package "com.fasterxml.jackson.annotation"
Classes in package "com.fasterxml.jackson.core"
Classes in package "com.fasterxml.jackson.databind"
Classes in package "org.apache.avro"
Classes in package "org.hamcrest"
Classes in package "org.codehaus.jackson"
Classes in package "org.joda.time"
Classes in package "org.junit"

 but: The following white-listed scopes did not have matching classes on 
the API surface:
No Classes in package "com.fasterxml.jackson.annotation"
No Classes in package "com.fasterxml.jackson.core"
No Classes in package "com.fasterxml.jackson.databind"
No Classes in package "com.google.api.client"
No Classes in package "com.google.api.services.bigquery"
No Classes in package "com.google.api.services.cloudresourcemanager"
No Classes in package "com.google.api.services.pubsub"
No Classes in package "com.google.api.services.storage"
No Classes in package "com.google.auth"
No Classes in package "com.google.protobuf"
No Classes in package "org.apache.avro"
No Classes in package "org.apache.beam"
No Classes in package "org.codehaus.jackson"
No Classes in package "org.hamcrest"
No Classes in package "org.joda.time"
No Classes in package "org.junit"
{code}

Job link:
https://builds.apache.org/job/beam_PostCommit_Java_Version_Test/11/

Multi-JDK version test is based on this PR:
https://github.com/apache/beam/pull/2204/files

Any ideas?


> SdkCoreApiSurfaceTest Failed in JDK7&8 and OpenJDK7&8 on Jenkins
> 
>
> Key: BEAM-1676
> URL: 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.7 (latest),beam #14

[Apache Jenkins Server]

See 


--
[...truncated 475.83 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.349 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.083 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.044 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.051 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.101 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.082 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.15 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.22 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 7 (on Ubuntu only),beam #14

[Apache Jenkins Server]

See 


--
[...truncated 476.77 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.32 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.075 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.048 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.024 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.157 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.267 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.141 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.232 sec - in 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 8 (on Ubuntu only),beam #14

[Apache Jenkins Server]

See 


--
[...truncated 476.83 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.281 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.053 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.043 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.094 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.016 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.229 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.093 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.151 sec - in 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.8 (latest),beam #14

[Apache Jenkins Server]

See 


--
[...truncated 473.52 KB...]
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.05 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.088 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.021 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.122 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.129 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.24 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running org.apache.beam.sdk.util.FluentBackoffTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.FluentBackoffTest
Running org.apache.beam.sdk.util.IntervalBoundedExponentialBackOffTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1874

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Java_RunnableOnService_Dataflow #2510

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #2857

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_RunnableOnService_Flink #1873

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-1675) deprecate withCoder() in JdbcIO

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903786#comment-15903786
 ] 

Jean-Baptiste Onofré commented on BEAM-1675:


Yes, definitely a good idea (we discussed about this with Ken and it's on my 
TODO).

> deprecate withCoder() in JdbcIO
> ---
>
> Key: BEAM-1675
> URL: https://issues.apache.org/jira/browse/BEAM-1675
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Jean-Baptiste Onofré
>
> In JdbcIO.read(), a Coder is required to handle the serialize/deserialize of 
> input record similar as below example. It's better to leverage Coders defined 
> in CoderRegistry.  
> {code}
> .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of()))
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (BEAM-1675) deprecate withCoder() in JdbcIO

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-1675:
--

Assignee: Jean-Baptiste Onofré  (was: Davor Bonaci)

> deprecate withCoder() in JdbcIO
> ---
>
> Key: BEAM-1675
> URL: https://issues.apache.org/jira/browse/BEAM-1675
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-extensions
>Reporter: Xu Mingmin
>Assignee: Jean-Baptiste Onofré
>
> In JdbcIO.read(), a Coder is required to handle the serialize/deserialize of 
> input record similar as below example. It's better to leverage Coders defined 
> in CoderRegistry.  
> {code}
> .withCoder(KvCoder.of(BigEndianIntegerCoder.of(), StringUtf8Coder.of()))
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: beam_PostCommit_Java_RunnableOnService_Dataflow #2509

[Apache Jenkins Server]

See 


--
[...truncated 993.91 KB...]
2017-03-09T20:12:06.350 [INFO] --- maven-site-plugin:3.5.1:attach-descriptor 
(attach-descriptor) @ beam-runners-core-construction-java ---
2017-03-09T20:12:06.794 [INFO] 
2017-03-09T20:12:06.794 [INFO] --- maven-shade-plugin:3.0.0:shade 
(bundle-and-repackage) @ beam-runners-core-construction-java ---
2017-03-09T20:12:06.798 [INFO] Excluding 
org.apache.beam:beam-sdks-java-core:jar:0.7.0-SNAPSHOT from the shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding 
org.apache.beam:beam-sdks-common-runner-api:jar:0.7.0-SNAPSHOT from the shaded 
jar.
2017-03-09T20:12:06.798 [INFO] Excluding io.grpc:grpc-auth:jar:1.0.1 from the 
shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding io.grpc:grpc-core:jar:1.0.1 from the 
shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding io.grpc:grpc-context:jar:1.0.1 from 
the shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding io.grpc:grpc-netty:jar:1.0.1 from the 
shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding 
io.netty:netty-codec-http2:jar:4.1.3.Final from the shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding 
io.netty:netty-codec-http:jar:4.1.3.Final from the shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding io.grpc:grpc-stub:jar:1.0.1 from the 
shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding io.grpc:grpc-all:jar:1.0.1 from the 
shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding io.grpc:grpc-protobuf:jar:1.0.1 from 
the shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding 
com.google.protobuf:protobuf-java-util:jar:3.0.0 from the shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding com.google.code.gson:gson:jar:2.3 from 
the shaded jar.
2017-03-09T20:12:06.798 [INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.0.1 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 from the shaded 
jar.
2017-03-09T20:12:06.799 [INFO] Excluding io.grpc:grpc-okhttp:jar:1.0.1 from the 
shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from 
the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.0.1 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.protobuf:protobuf-lite:jar:3.0.1 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.auth:google-auth-library-credentials:jar:0.6.0 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.auth:google-auth-library-oauth2-http:jar:0.6.0 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding io.netty:netty-handler:jar:4.1.3.Final 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding io.netty:netty-buffer:jar:4.1.3.Final 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding io.netty:netty-common:jar:4.1.3.Final 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
io.netty:netty-transport:jar:4.1.3.Final from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
io.netty:netty-resolver:jar:4.1.3.Final from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding io.netty:netty-codec:jar:4.1.3.Final 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.api.grpc:grpc-google-common-protos:jar:0.1.0 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.api-client:google-api-client:jar:1.22.0 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev295-1.22.0 from the 
shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev6-1.22.0 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 from the shaded 
jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.http-client:google-http-client:jar:1.22.0 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
commons-logging:commons-logging:jar:1.1.1 from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding commons-codec:commons-codec:jar:1.3 
from the shaded jar.
2017-03-09T20:12:06.799 [INFO] Excluding 
com.google.http-client:google-http-client-jackson:jar:1.22.0 from 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 7 (on Ubuntu only),beam #13

[Apache Jenkins Server]

See 


Changes:

[tgroh] Deprecate Pipeline.getOptions

[iemejia] Make all uses of CountingOutputStream close their resources

--
[...truncated 475.81 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.292 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.023 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.088 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.042 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.012 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.064 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.089 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.367 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.101 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.7 (latest),beam #13

[Apache Jenkins Server]

See 


Changes:

[tgroh] Deprecate Pipeline.getOptions

[iemejia] Make all uses of CountingOutputStream close their resources

--
[...truncated 473.63 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.353 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.076 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.061 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.098 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.622 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.143 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » OpenJDK 8 (on Ubuntu only),beam #13

[Apache Jenkins Server]

See 


Changes:

[tgroh] Deprecate Pipeline.getOptions

[iemejia] Make all uses of CountingOutputStream close their resources

--
[...truncated 474.98 KB...]
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.284 sec - in 
org.apache.beam.sdk.transforms.CreateTest
Running org.apache.beam.sdk.transforms.CombineTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.transforms.CombineTest
Running org.apache.beam.sdk.util.VarIntTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.058 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.05 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.024 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.325 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.095 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests 

Build failed in Jenkins: beam_PostCommit_Java_Version_Test » JDK 1.8 (latest),beam #13

[Apache Jenkins Server]

See 


--
[...truncated 472.59 KB...]
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.VarIntTest
Running org.apache.beam.sdk.util.StructsTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.StructsTest
Running org.apache.beam.sdk.util.StreamUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.StreamUtilsTest
Running org.apache.beam.sdk.util.PubsubTestClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.PubsubTestClientTest
Running org.apache.beam.sdk.util.StringUtilsTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.StringUtilsTest
Running org.apache.beam.sdk.util.MutationDetectorsTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.MutationDetectorsTest
Running org.apache.beam.sdk.util.ValueWithRecordIdTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec - in 
org.apache.beam.sdk.util.ValueWithRecordIdTest
Running org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayOutputStreamTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryRegistrarTest
Running org.apache.beam.sdk.util.GcpProjectUtilTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec - in 
org.apache.beam.sdk.util.GcpProjectUtilTest
Running org.apache.beam.sdk.util.ZipFilesTest
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 sec - in 
org.apache.beam.sdk.util.ZipFilesTest
Running org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.GcsIOChannelFactoryTest
Running org.apache.beam.sdk.util.ApiSurfaceTest
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 sec - in 
org.apache.beam.sdk.util.ApiSurfaceTest
Running org.apache.beam.sdk.util.PubsubClientTest
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.02 sec - in 
org.apache.beam.sdk.util.PubsubClientTest
Running org.apache.beam.sdk.util.UnownedInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in 
org.apache.beam.sdk.util.UnownedInputStreamTest
Running org.apache.beam.sdk.util.FileIOChannelFactoryTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 sec - in 
org.apache.beam.sdk.util.FileIOChannelFactoryTest
Running org.apache.beam.sdk.util.ReleaseInfoTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ReleaseInfoTest
Running org.apache.beam.sdk.util.WindowedValueTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01 sec - in 
org.apache.beam.sdk.util.WindowedValueTest
Running org.apache.beam.sdk.util.common.ReflectHelpersTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec - in 
org.apache.beam.sdk.util.common.ReflectHelpersTest
Running org.apache.beam.sdk.util.PubsubJsonClientTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.062 sec - in 
org.apache.beam.sdk.util.PubsubJsonClientTest
Running org.apache.beam.sdk.util.DefaultBucketTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 sec - in 
org.apache.beam.sdk.util.DefaultBucketTest
Running org.apache.beam.sdk.util.GcsUtilTest
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.051 sec - in 
org.apache.beam.sdk.util.GcsUtilTest
Running org.apache.beam.sdk.util.WindowingStrategiesTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.088 sec - in 
org.apache.beam.sdk.util.WindowingStrategiesTest
Running org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 
org.apache.beam.sdk.util.ExposedByteArrayInputStreamTest
Running org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.173 sec - in 
org.apache.beam.sdk.util.BufferedElementCountingOutputStreamTest
Running org.apache.beam.sdk.util.FluentBackoffTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec - in 
org.apache.beam.sdk.util.FluentBackoffTest
Running org.apache.beam.sdk.util.IntervalBoundedExponentialBackOffTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in 

[jira] [Commented] (BEAM-818) Remove Pipeline.getPipelineOptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903748#comment-15903748
 ] 

ASF GitHub Bot commented on BEAM-818:
-

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2142


> Remove Pipeline.getPipelineOptions
> --
>
> Key: BEAM-818
> URL: https://issues.apache.org/jira/browse/BEAM-818
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This stops transforms from changing their operation based on 
> construction-time options, and instead requires that configuration to be 
> explicit, or to obtain the configuration at runtime.
> https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk--XmGMGgDwwNwWZaFbxLKvPqEQ/edit#



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] beam pull request #2181: Make all uses of CountingOutputStream close their r...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/2181


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[1/2] beam git commit: Deprecate Pipeline.getOptions

[kenn]

Repository: beam
Updated Branches:
  refs/heads/master 013365f73 -> ef256df18


Deprecate Pipeline.getOptions

Configuration should be explicit pre-execution, and after the call to
Run a PipelineRunner controls the PipelineOptions of the executions.
Runtime access to options is unaffected by this change.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/4f3c5bd2
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/4f3c5bd2
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/4f3c5bd2

Branch: refs/heads/master
Commit: 4f3c5bd24eb0d34b1b176c7248973400c3e8dfea
Parents: b322a5d
Author: Thomas Groh 
Authored: Thu Mar 2 09:41:37 2017 -0800
Committer: Thomas Groh 
Committed: Thu Mar 2 09:41:37 2017 -0800

--
 sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java | 4 
 1 file changed, 4 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/4f3c5bd2/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
--
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
index fe1d526..f09f2b4 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/Pipeline.java
@@ -464,7 +464,11 @@ public class Pipeline {
 
   /**
* Returns the configured {@link PipelineOptions}.
+   *
+   * @deprecated see BEAM-818 Remove Pipeline.getPipelineOptions. 
Configuration should be explicitly
+   * provided to a transform if it is required.
*/
+  @Deprecated
   public PipelineOptions getOptions() {
 return options;
   }

[jira] [Commented] (BEAM-1418) MapElements and FlatMapElements should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903697#comment-15903697
 ] 

Kenneth Knowles commented on BEAM-1418:
---

I don't know that it allows getting rid of the intermediate class, just 
renaming it to {{MissingFunction}}. FWIW I don't find it awkward :-)

But if the style guide is to just use one class with nullables and do 
validation at {{expand}} time you can make that change any time and allow any 
order of arguments, yes?

> MapElements and FlatMapElements should comply with PTransform style guide
> -
>
> Key: BEAM-1418
> URL: https://issues.apache.org/jira/browse/BEAM-1418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Type structure of these classes can be improved by slightly changing the API:
> FlatMapElements.into(TypeDescriptor).via(as usual).
> Likewise for MapElements. This allows getting rid of the awkward 
> MissingOutputTypeDescriptor intermediate class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1422) ParDo should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1422:
--
Fix Version/s: First stable release

> ParDo should comply with PTransform style guide
> ---
>
> Key: BEAM-1422
> URL: https://issues.apache.org/jira/browse/BEAM-1422
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Suggested changes:
> - Get rid of ParDo.Unbound and UnboundMulti classes completely
> - Get rid of static methods such as withSideInputs/Outputs() - the only entry 
> point should be via ParDo.of(). Respectively, get rid of non-static .of().
> - Rename ParDo.Bound and ParDo.BoundMulti respectively to simply ParDo and 
> ParDoWithSideOutputs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1425) Window should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903698#comment-15903698
 ] 

Kenneth Knowles commented on BEAM-1425:
---

This is WAI, in fact. You can window without touching the {{WindowFn}}.

> Window should comply with PTransform style guide
> 
>
> Key: BEAM-1425
> URL: https://issues.apache.org/jira/browse/BEAM-1425
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
>
> Suggested changes:
> - Remove static builder-like methods such as triggering(), 
> discardingFiredPanes() - the only static entry point should be .into().
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1425) Window should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1425:
--
Fix Version/s: First stable release

> Window should comply with PTransform style guide
> 
>
> Key: BEAM-1425
> URL: https://issues.apache.org/jira/browse/BEAM-1425
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Suggested changes:
> - Remove static builder-like methods such as triggering(), 
> discardingFiredPanes() - the only static entry point should be .into().
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-828) Remove PipelineOptions from BigQueryIO

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-828:
-
Fix Version/s: First stable release

> Remove PipelineOptions from BigQueryIO
> --
>
> Key: BEAM-828
> URL: https://issues.apache.org/jira/browse/BEAM-828
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-gcp
>Reporter: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> BigQueryIO uses PipelineOptions to configure itself at construction time.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1418) MapElements and FlatMapElements should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1418:
--
Labels: backward-incompatible starter  (was: backward-incompatible)

> MapElements and FlatMapElements should comply with PTransform style guide
> -
>
> Key: BEAM-1418
> URL: https://issues.apache.org/jira/browse/BEAM-1418
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Type structure of these classes can be improved by slightly changing the API:
> FlatMapElements.into(TypeDescriptor).via(as usual).
> Likewise for MapElements. This allows getting rid of the awkward 
> MissingOutputTypeDescriptor intermediate class.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (BEAM-1425) Window should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15903701#comment-15903701
 ] 

Kenneth Knowles commented on BEAM-1425:
---

But we should still get rid of {{Bound}} and likely use autovalue.

> Window should comply with PTransform style guide
> 
>
> Key: BEAM-1425
> URL: https://issues.apache.org/jira/browse/BEAM-1425
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
>
> Suggested changes:
> - Remove static builder-like methods such as triggering(), 
> discardingFiredPanes() - the only static entry point should be .into().
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1521) Pass element (and/or generic context) for choosing a table in BigQueryIO

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1521:
--
Fix Version/s: First stable release

> Pass element (and/or generic context) for choosing a table in BigQueryIO
> 
>
> Key: BEAM-1521
> URL: https://issues.apache.org/jira/browse/BEAM-1521
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> It would be easy and helpful to also pass the element when choosing a table 
> for a BigQuery write. And perhaps future-proof this by a Context parameter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1283) DoFn.Context.output spec for startBundle/finishBundle is a mess

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1283:
--
Fix Version/s: First stable release

> DoFn.Context.output spec for startBundle/finishBundle is a mess
> ---
>
> Key: BEAM-1283
> URL: https://issues.apache.org/jira/browse/BEAM-1283
> Project: Beam
>  Issue Type: Bug
>  Components: beam-model, sdk-java-core
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> The spec is here in Javadoc: 
> https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L128
> "If invoked from {{@StartBundle}} or {{@FinishBundle}}, this will attempt to 
> use the {{WindowFn}} of the input {{PCollection}} to determine what windows 
> the element should be in, throwing an exception if the {{WindowFn}} attempts 
> to access any information about the input element. The output element will 
> have a timestamp of negative infinity."
> This is a collection of caveats that make this method not always technically 
> wrong, but quite a mess. Ideas that reasonable folks have suggested lately:
>  - The {{WindowFn}} cannot actually be applied because {{WindowFn}} is 
> allowed to see the element type. The spec just avoids this by limiting which 
> {{WindowFn}} can be used.
>  - There is no natural output timestamp, so it should always be provided. The 
> spec avoids this by specifying an arbitrary and fairly useless timestamp.
>  - If it is a merging {{WindowFn}} like sessions that has already been merged 
> then you'll just have a bogus proto window regardless of explicit timestamp 
> or not.
> The use cases for these methods are best addressed by state plus window 
> expiry callback, so we should revisit this spec and probably just wipe it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1402) Make TextIO and AvroIO use best-practice types.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1402:
--
Fix Version/s: First stable release

> Make TextIO and AvroIO use best-practice types.
> ---
>
> Key: BEAM-1402
> URL: https://issues.apache.org/jira/browse/BEAM-1402
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Replace static Read/Write classes with type-instantiated classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-217) BoundedSource.splitAtFraction should be splitAfterFraction

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-217:
-
Fix Version/s: First stable release

> BoundedSource.splitAtFraction should be splitAfterFraction
> --
>
> Key: BEAM-217
> URL: https://issues.apache.org/jira/browse/BEAM-217
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>Priority: Minor
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Dynamic work rebalancing works by 1) determining how long the bundle should 
> take in order to not be a straggler - the "deadline", 2) predicting where the 
> bundle will be (position or fraction) by that deadline, and 3) requesting an 
> atomic split (splitAtFraction).
> Currently all BoundedSource's and (in Dataflow runner) NativeReaderIterator's 
> refuse splits if they have already consumed the requested split position.
> Splitting a task [A, C) at position B generates [A, B) and [B, C), so if we 
> predict that by deadline the task will have last consumed position X, we 
> should split not "at" X, but "after" X (i.e. at next(X)) - i.e. into [A, X] 
> (because X is already consumed) and (X, C) equivalently [A, next(X)) and 
> [next(X), C).
> One way to fit this into the current BoundedSource API is to rename 
> splitAtFraction to splitAfterFraction and adjust the documentation. 
> Documentation of getFractionConsumed also needs to be clarified to emphasize 
> that it should return what fraction of all positions in the source have 
> already been consumed, including the position of the last consumed record. 
> For example, for an index-range task with range [0, 5), after it has read the 
> first record at position 0, it has consumed 20%, rather than 0% (and of 
> course not 40% even if an internal "next index" variable is now 1 - this 
> mistake is especially easy to make in a file-based source if you base the 
> calculations on the file's offset *after* consuming the record - the correct 
> way is to calculate based on offsets of beginning of records).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-818) Remove Pipeline.getPipelineOptions

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-818:
-
Fix Version/s: First stable release

> Remove Pipeline.getPipelineOptions
> --
>
> Key: BEAM-818
> URL: https://issues.apache.org/jira/browse/BEAM-818
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This stops transforms from changing their operation based on 
> construction-time options, and instead requires that configuration to be 
> explicit, or to obtain the configuration at runtime.
> https://docs.google.com/document/d/1Wr05cYdqnCfrLLqSk--XmGMGgDwwNwWZaFbxLKvPqEQ/edit#



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1355) HDFS IO should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1355:
--
Labels: backward-incompatible starter  (was: )

> HDFS IO should comply with PTransform style guide
> -
>
> Key: BEAM-1355
> URL: https://issues.apache.org/jira/browse/BEAM-1355
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Eugene Kirpichov
>Assignee: Stephen Sisk
>  Labels: backward-incompatible, starter
>
> https://github.com/apache/beam/tree/master/sdks/java/io/hdfs/src/main/java/org/apache/beam/sdk/io/hdfs
>  does not comply with 
> https://beam.apache.org/contribute/ptransform-style-guide/ in a number of 
> ways:
> - It is not packaged as a PTransform (should be: HDFSIO.Read,Write or 
> something like that)
> - Should probably use AutoValue for specifying parameters
> Stephen knows about the current state of HDFS IO.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1414) CountingInput should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1414:
--
Fix Version/s: First stable release

> CountingInput should comply with PTransform style guide
> ---
>
> Key: BEAM-1414
> URL: https://issues.apache.org/jira/browse/BEAM-1414
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Eugene Kirpichov
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Suggested changes:
> - Rename the whole class and its inner transforms to sound more verb-like, 
> e.g.: GenerateRange.Bounded/Unbounded (as opposed to current 
> CountingInput.BoundedCountingInput)
> - Provide a more unified API between bounded and unbounded cases: 
> GenerateRange.from(100) should return a GenerateRange.Unbounded; 
> GenerateRange.from(100).to(200) should return a GenerateRange.Bounded. They 
> both should accept a timestampFn. The unbounded one _should not_ have a 
> withMaxNumRecords builder - that's redundant with specifying the range.
> - (optional) Use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-827) Remove PipelineOptions from Write

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-827:
-
Fix Version/s: First stable release

> Remove PipelineOptions from Write
> -
>
> Key: BEAM-827
> URL: https://issues.apache.org/jira/browse/BEAM-827
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Thomas Groh
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> This may require validating the sink is properly configured at Runtime rather 
> than at apply-time.
> Requires making the writer-result coder part of the sink interface, rather 
> than the WriteOperation interface, as otherwise the coder is not available at 
> runtime.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-447) Stop referring to types with Bound/Unbound

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-447:
-
Fix Version/s: First stable release

> Stop referring to types with Bound/Unbound
> --
>
> Key: BEAM-447
> URL: https://issues.apache.org/jira/browse/BEAM-447
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-core
>Reporter: Thomas Groh
>Assignee: Davor Bonaci
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Bounded and Unbounded are used to refer to PCollections, and the overlap is 
> confusing.  These classes should be renamed to be both more specific (e.g. 
> ParDo.LackingDoFnSingleOutput, ParDo.SingleOutput, Window.AssignWindows) 
> which remove the overlap.
> examples:
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java#L658
> https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java#L868



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1402) Make TextIO and AvroIO use best-practice types.

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1402:
--
Labels: backward-incompatible starter  (was: )

> Make TextIO and AvroIO use best-practice types.
> ---
>
> Key: BEAM-1402
> URL: https://issues.apache.org/jira/browse/BEAM-1402
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Reuven Lax
>  Labels: backward-incompatible, starter
> Fix For: First stable release
>
>
> Replace static Read/Write classes with type-instantiated classes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-40) Replace rawtype lambda-incompatible uses of SerializableFunction with SimpleFunction (as appropriate)

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-40?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-40:

Fix Version/s: First stable release

> Replace rawtype lambda-incompatible uses of SerializableFunction with 
> SimpleFunction (as appropriate)
> -
>
> Key: BEAM-40
> URL: https://issues.apache.org/jira/browse/BEAM-40
> Project: Beam
>  Issue Type: Task
>  Components: sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Minor
>  Labels: Java8, backward-incompatible
> Fix For: First stable release
>
>
> When a lambda or method reference is used in Java 8 to provide a 
> SerializableFunction, it is instantiated at the raw type 
> SerializableFunction. We occasionally require reflective access to the actual 
> parameter for OutputT, but it will be unavailable.
> MapElements and FlatMapElements thus use the analogous abstract class 
> SimpleFunction in such situations to prevent use of a lambda or method 
> reference. They then support lambda via separate constructors that require 
> user help to determine the concrete output type.
> This ticket calls for an audit of such situations.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (BEAM-1427) BigQueryIO should comply with PTransform style guide

[Kenneth Knowles (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kenneth Knowles updated BEAM-1427:
--
Fix Version/s: First stable release

> BigQueryIO should comply with PTransform style guide
> 
>
> Key: BEAM-1427
> URL: https://issues.apache.org/jira/browse/BEAM-1427
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Eugene Kirpichov
>Assignee: Eugene Kirpichov
>  Labels: backward-incompatible
> Fix For: First stable release
>
>
> Suggested changes:
> - Remove Read/Write.Bound classes - Read and Write themselves should be the 
> transform classes
> - Remove static builder-like .withBlah() methods
> - (optional) use AutoValue



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)