Build failed in Jenkins: beam_Release_NightlySnapshot #321

[Apache Jenkins Server]

See 

Changes:

[kirpichov] [BEAM-65] ProcessFn: support setup/teardown

[klk] Remove extraneous chunking from GroupAlsoByWindowsViaOutputBufferDoFn

[klk] Autoformat GroupAlsoByWindowsViaOutputBufferDoFn

[klk] Remove inaccurate pluralization from GABWViaOutputBufferDoFn

[klk] Eliminate use of InMemoryStateInternals in the Java SDK

[klk] Move InMemoryStateInternals to runners/core-java

[klk] Delete long-deprecated StateContexts methods

[klk] Move WindowingInternals to runners/core-java

[klk] Remove comment-only uses that block runners/core-java migrations

[klk] Move StateInternalsFactory to runners/core-java

[klk] Move TimerInternalsFactory to runners/core-java

[klk] Move TimerInternals to runners/core-java

[klk] Move StateTable to runners/core-java

[klk] Move StateInternals to runners/core-java

[klk] Move StateNamespace and related to runners/core-java

[klk] Upgrade Dataflow container version to beam-master-20170203

[altay] Reduce test times for two retry based tests

[altay] Updates default values used by retry decorator.

[ansela] [BEAM-1395] Remove chunking.

[klk] Allow absolute timers

[klk] Add some more RunnableOnService tests for stateful ParDo

[robertwb] Add snippet for reading from compressed text sources

[younghee.kwon] Improve DirectRunner performance by tuning BoundedReadEvaluator.

[altay] Removes deprecated fileio.TextFileSink.

--
[...truncated 7603 lines...]
2017-02-07T07:25:29.407 [INFO] 2017-02-07\T\07:25:29.407 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-jdk5/13.0/guava-jdk5-13.0.pom
2017-02-07T07:25:29.436 [INFO] 2017-02-07\T\07:25:29.435 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-jdk5/13.0/guava-jdk5-13.0.pom
 (6 KB at 194.0 KB/sec)
2017-02-07T07:25:29.438 [INFO] 2017-02-07\T\07:25:29.438 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent-jdk5/13.0/guava-parent-jdk5-13.0.pom
2017-02-07T07:25:29.467 [INFO] 2017-02-07\T\07:25:29.466 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/guava/guava-parent-jdk5/13.0/guava-parent-jdk5-13.0.pom
 (3 KB at 89.4 KB/sec)
2017-02-07T07:25:29.471 [INFO] 2017-02-07\T\07:25:29.471 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/apis/google-api-services-storage/v1-rev35-1.20.0/google-api-services-storage-v1-rev35-1.20.0.pom
2017-02-07T07:25:29.500 [INFO] 2017-02-07\T\07:25:29.499 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/apis/google-api-services-storage/v1-rev35-1.20.0/google-api-services-storage-v1-rev35-1.20.0.pom
 (3 KB at 91.6 KB/sec)
2017-02-07T07:25:29.503 [INFO] 2017-02-07\T\07:25:29.503 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/2.0.3/jsr305-2.0.3.pom
2017-02-07T07:25:29.533 [INFO] 2017-02-07\T\07:25:29.533 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/code/findbugs/jsr305/2.0.3/jsr305-2.0.3.pom
 (2 KB at 46.7 KB/sec)
2017-02-07T07:25:29.556 [INFO] 2017-02-07\T\07:25:29.555 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.0/jackson-annotations-2.7.0.pom
2017-02-07T07:25:29.584 [INFO] 2017-02-07\T\07:25:29.584 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.7.0/jackson-annotations-2.7.0.pom
 (2 KB at 49.0 KB/sec)
2017-02-07T07:25:29.687 [INFO] 2017-02-07\T\07:25:29.686 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.20.0/google-api-client-java6-1.20.0.jar
2017-02-07T07:25:29.688 [INFO] 2017-02-07\T\07:25:29.687 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-jackson2/1.20.0/google-api-client-jackson2-1.20.0.jar
2017-02-07T07:25:29.689 [INFO] 2017-02-07\T\07:25:29.689 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.20.0/google-oauth-client-java6-1.20.0.jar
2017-02-07T07:25:29.716 [INFO] 2017-02-07\T\07:25:29.716 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/api-client/google-api-client-java6/1.20.0/google-api-client-java6-1.20.0.jar
 (3 KB at 70.7 KB/sec)
2017-02-07T07:25:29.752 [INFO] 2017-02-07\T\07:25:29.752 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/google/oauth-client/google-oauth-client-java6/1.20.0/google-oauth-client-java6-1.20.0.jar
 (11 KB at 164.0 KB/sec)
2017-02-07T07:25:29.756 [INFO] 2017-02-07\T\07:25:29.756 [INFO] 

2017-02-07T07:25:29.756 [INFO] 2017-02-07\T\07:25:29.756 [INFO] BUILD FAILURE
2017-02-07T07:25:29.756 [INFO] 2017-02-07\T\07:25:29.756 [INFO] 

2017-02-07T07:25:29.757 [INFO] 

Re: Beam connector development for Hive as a data source

[Davor Bonaci]

Hi Madhu,
Welcome! I suggest subscribing to the dev@ mailing list and using the same
email address when sending to the list, to avoid your email being caught in
moderation.

It would be great to have a connector for Apache Hive. Keep in mind that
several folks have expressed interest in using and contributing this
connector. As far as I know, nobody is *actively* working on it, so you
should be good to go. Please use BEAM-1158 [1] to coordinate this work with
any other interested contributor.

Note that there are several different ways of connecting Beam and Hive. The
simplest one is to write HiveIO that which would run a Hive query and
process Hive's results in Beam. Another would be to use Beam within Hive to
compute the results of a Hive query. Finally, one could possibly write a
Hive-based DSL on top of a Beam SDK.

All of these approaches are valid and somewhat orthogonal one to another.
I'm assuming you are after the first one. If so, and if you plan to follow
already established patterns in other connectors, you don't necessarily
need a design document. Otherwise, please start with a design document. We
have linked a template in the Contribution Guide [2, 3].

Once again, welcome and let us know if we can help in any way!

Davor

[1] https://issues.apache.org/jira/browse/BEAM-1158
[2] https://beam.apache.org/contribute/contribution-guide/
[3]
https://docs.google.com/document/d/1qYQPGtabN5-E4MjHsecqqC7PXvJtXvZukPfLXQ8rHJs

On Mon, Feb 6, 2017 at 4:27 PM, Madhusudan Borkar 
wrote:

> Hello,
>
> I am Big Data Architect working at eTouch Systems. We are GCP partners. We
> are planning to contribute to Beam by developing a connector for Apache
> Hive as a data source.
> I understand that before any development work begins, we need to submit our
> design to Beam community.  I would like to request you to please share a
> "design template" document for the same.  We will submit our design
> document, using the template.
>
>
> Thank you.
>
> best regards
> Madhu Borkar
>

Beam connector development for Hive as a data source

[Madhusudan Borkar]

Hello,

I am Big Data Architect working at eTouch Systems. We are GCP partners. We
are planning to contribute to Beam by developing a connector for Apache
Hive as a data source.
I understand that before any development work begins, we need to submit our
design to Beam community.  I would like to request you to please share a
"design template" document for the same.  We will submit our design
document, using the template.


Thank you.

best regards
Madhu Borkar

Report to the Board, February 2017 edition

[Davor Bonaci]

We are expected to submit a project report to the ASF Board of Directors
ahead of its next meeting. The report is due on Wednesday, 2/8.

This is the second is the series of three monthly reports required for new
projects. We'll need to report next month as well.

If interested, please take a look at the draft [1], and comment as
appropriate. I'll submit the report sometime on Wednesday.

Thanks!

Davor

[1] https://docs.google.com/document/d/1QXc6lH8Zi6qqp_
tmVkh4lvJRcW18UO3F9EopjPhcJfQ/

Re: BEAM-307(KafkaIO on Kafka 0.10)

[Davor Bonaci]

This would be a great contribution, Mingmin!

As a general rule, we'd like the connector to work with as many versions as
possible, with as little code duplication as possible.

Slightly orthogonal -- BigtableIO [1] is an example of a connector that
separates API portions from the underlying service. With a similar
layering, perhaps KafkaIO can support multiple versions of Kafka, while
maintaining as much of the common code as possible. KafkaIO case is a bit
more complicated, given that it needs changes on the API side as well.

[1]
https://github.com/apache/beam/tree/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable

On Mon, Feb 6, 2017 at 12:21 PM, Raghu Angadi 
wrote:

> I see. kafka-clients dependency could also be in 'provided' scope so that
> is simpler to use different versions at runtime.
>
> On Mon, Feb 6, 2017 at 12:05 PM, Xu Mingmin  wrote:
>
> > The one I meet is external authentication added in 0.10, we use a
> > standalone token-based security service. In 0.9 the SASL-based
> > implementation is fixed with Kerberos.
> > Kafka client 0.10 cannot connect to Kafka server 0.9, that's why I
> mention
> > a separated project.
> >
> > Mingmin
> >
> > On Mon, Feb 6, 2017 at 11:45 AM, Raghu Angadi  >
> > wrote:
> >
> > > Current KafkaIO works just fine with Kafka 0.10. I don't know of any
> > > incompatibilities or regressions.
> > >
> > > It does not take advantage  of message timestamps, of course. It would
> be
> > > good to take handle tme in in a backward compatible way.. it might be
> > > required anyway if they are optional in 0.10.
> > >
> > > Not sure of scope of (1) below. I don't think it needs to be a new
> > > implementation.
> > >
> > > On Mon, Feb 6, 2017 at 11:35 AM, Xu Mingmin 
> wrote:
> > >
> > > > Hello,
> > > >
> > > > Is there anybody working on https://issues.apache.org/
> > > jira/browse/BEAM-307
> > > > ?
> > > > The existing KafkaIO is implemented with Kafka 0.9, and not
> compatible
> > > well
> > > > with Kafka 0.10.
> > > >
> > > > I'd like to take this task if not duplicated:
> > > > 1). a new KafkaIO based on Kafka 0.10, suggest a separated project
> for
> > > > easy-to-build;
> > > > 2). use timestamp of Kafka message as default event-timestamp
> > > >
> > > > Thanks!
> > > > Mingmin
> > > >
> > >
> >
>

Re: BEAM-307(KafkaIO on Kafka 0.10)

[Raghu Angadi]

I see. kafka-clients dependency could also be in 'provided' scope so that
is simpler to use different versions at runtime.

On Mon, Feb 6, 2017 at 12:05 PM, Xu Mingmin  wrote:

> The one I meet is external authentication added in 0.10, we use a
> standalone token-based security service. In 0.9 the SASL-based
> implementation is fixed with Kerberos.
> Kafka client 0.10 cannot connect to Kafka server 0.9, that's why I mention
> a separated project.
>
> Mingmin
>
> On Mon, Feb 6, 2017 at 11:45 AM, Raghu Angadi 
> wrote:
>
> > Current KafkaIO works just fine with Kafka 0.10. I don't know of any
> > incompatibilities or regressions.
> >
> > It does not take advantage  of message timestamps, of course. It would be
> > good to take handle tme in in a backward compatible way.. it might be
> > required anyway if they are optional in 0.10.
> >
> > Not sure of scope of (1) below. I don't think it needs to be a new
> > implementation.
> >
> > On Mon, Feb 6, 2017 at 11:35 AM, Xu Mingmin  wrote:
> >
> > > Hello,
> > >
> > > Is there anybody working on https://issues.apache.org/
> > jira/browse/BEAM-307
> > > ?
> > > The existing KafkaIO is implemented with Kafka 0.9, and not compatible
> > well
> > > with Kafka 0.10.
> > >
> > > I'd like to take this task if not duplicated:
> > > 1). a new KafkaIO based on Kafka 0.10, suggest a separated project for
> > > easy-to-build;
> > > 2). use timestamp of Kafka message as default event-timestamp
> > >
> > > Thanks!
> > > Mingmin
> > >
> >
>

Re: BEAM-307(KafkaIO on Kafka 0.10)

[Raghu Angadi]

Current KafkaIO works just fine with Kafka 0.10. I don't know of any
incompatibilities or regressions.

It does not take advantage  of message timestamps, of course. It would be
good to take handle tme in in a backward compatible way.. it might be
required anyway if they are optional in 0.10.

Not sure of scope of (1) below. I don't think it needs to be a new
implementation.

On Mon, Feb 6, 2017 at 11:35 AM, Xu Mingmin  wrote:

> Hello,
>
> Is there anybody working on https://issues.apache.org/jira/browse/BEAM-307
> ?
> The existing KafkaIO is implemented with Kafka 0.9, and not compatible well
> with Kafka 0.10.
>
> I'd like to take this task if not duplicated:
> 1). a new KafkaIO based on Kafka 0.10, suggest a separated project for
> easy-to-build;
> 2). use timestamp of Kafka message as default event-timestamp
>
> Thanks!
> Mingmin
>

Re: [VOTE] Apache Beam, version 0.5.0, release candidate #2

[Davor Bonaci]

This vote is now complete. I'll summarize the results and next steps in a
separate thread.

On Mon, Feb 6, 2017 at 2:51 AM, Sergio Fernández  wrote:

> +1 (non-binding)
>
> So far I've successfully checked:
> * signatures and digests
> * source releases file layouts
> * no binaries included in the source release
> * matched git tag
> * NOTICE and LICENSE files
> * license headers
> * clean build (Java 1.8.0_91, Maven 3.3.9, Debian amd64)
>
> As I already commented on RC1, formally it's better to include commit id
> referencing the release, since git tags can be changed. Just take that into
> account for upcoming releases.
>
>
> On Fri, Feb 3, 2017 at 1:27 AM, Davor Bonaci  wrote:
>
> > Hi everyone,
> > With JB leaving for his vacation, I'll try to push the 0.5.0 release
> across
> > the finish line. Please review and vote on the release candidate #2 for
> the
> > version 0.5.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > which is signed with the key with fingerprint 8F0D334F [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v0.5.0-RC2" [5],
> > * website pull request listing the release and publishing the API
> reference
> > manual [6].
> >
> > Compared to release candidate #1, this candidate contains pull requests
> > #1903 [7] and #1908 [8]; see the discussion for reasoning.
> >
> > A passing suite of Jenkins jobs:
> > * PreCommit_Java_MavenInstall [9],
> > * PostCommit_Java_MavenInstall [10],
> > * PostCommit_Java_RunnableOnService_Apex [11],
> > * PostCommit_Java_RunnableOnService_Flink [12],
> > * PostCommit_Java_RunnableOnService_Spark [13],
> > * PostCommit_Java_RunnableOnService_Dataflow [14].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Davor
> >
> > [1]
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> > ctId=12319527=12338859
> > [2] https://dist.apache.org/repos/dist/dev/beam/0.5.0/RC2/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4] https://repository.apache.org/content/repositories/
> orgapachebeam-1011/
> > [5] https://github.com/apache/beam/tree/v0.5.0-RC2
> > [6] https://github.com/apache/beam-site/pull/132
> > [7] https://github.com/apache/beam/pull/1903
> > [8] https://github.com/apache/beam/pull/1908
> > [9] https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7028/
> > [10] https://builds.apache.org/job/beam_PostCommit_Java_
> MavenInstall/2514/
> > [11]
> > https://builds.apache.org/job/beam_PostCommit_Java_RunnableO
> > nService_Apex/386/
> > [12]
> > https://builds.apache.org/job/beam_PostCommit_Java_RunnableO
> > nService_Flink/1521/
> > [13]
> > https://builds.apache.org/job/beam_PostCommit_Java_RunnableO
> > nService_Spark/830/
> > [14]
> > https://builds.apache.org/job/beam_PostCommit_Java_RunnableO
> > nService_Dataflow/2180/
> >
>
>
>
> --
> Sergio Fernández
> Partner Technology Manager
> Redlink GmbH
> m: +43 6602747925 <+43%20660%202747925>
> e: sergio.fernan...@redlink.co
> w: http://redlink.co
>