[Discussion] Clarify the support story for released Beam versions

[Ahmet Altay]

Hi all,

I would like us to clarify the life cycle of Beam releases a little bit
more for our users. I think we increased the predictability significantly
by agreeing to a release cadence and kudos to everyone on that. As a follow
up to that I would like to address the following problem:

It is unclear for a user of Beam how long an existing version will be
supported. And if it will be supported at all, what does that support mean.
(This is especially an important problem for users who would like to use
stable versions and care less about being on the cutting edge.)

Our current state is:

- With our agreed release cadence Beam makes 8 releases a year.
- We have precedence for patching released versions for major issues.
- Patching all existing releases at any point (even patching a year full of
8 releases) will be significant work.

With the problem and the information, I have the following proposal to
define the life cycle of existing releases.

- Define what is a major issue with Beam. (For example this could be high
severity security issues and high risk data integrity issues.)
- Have a concept of long term support (LTS) releases. Designate every 4th
release as an LTS release (~6 months).
- Deprecate non-LTS releases the moment any new Beam release is out. Never
patch non-LTS releases.
- Deprecate LTS releases after a new LTS release comes out. Patch any LTS
release within 1 year of its initial release date.
- Add the above information to our website.

I think this proposal would give clear information to our users about what
they can expect from us, and reduce our burden to maintain existing
releases.

I also would like to state my assumptions:

- Releases will happen not because of a policy but because there are
volunteers willing to do it. This proposal is only a framework for those
volunteers to take action. If Beam does not support its releases, with or
without a policy, we will reduce the trust of our users.
- After we agreed to have a regular release cadence we started to see
improvements towards having regular releases even though we did not
perfectly hit 6 weeks mark each time. I do expect the same here: an
improvement in the direction of user happiness even if we cannot be perfect.

What do you think?

Ahmet

Jenkins build is back to normal : beam_SeedJob #2404

[Apache Jenkins Server]

See 

[HELP] Blog post for 2.6.0 Release

[Pablo Estrada]

Hello all,
During my work on the release, I missed that we have started doing blog
posts for every release. I decided to announce the release, and start the
blog post afterwards - the blog post will only be late for a few days.

Please add all your release notes and comments in this doc:
https://docs.google.com/document/d/1Jwz5AxInSm9C6z0TZqer6JYE2gpbJ_dZn88X37VvQEY/edit?usp=sharing

Thanks everyone for your help and work for this release!

Best
-P.
-- 
Got feedback? go/pabloem-feedback

Build failed in Jenkins: beam_SeedJob #2403

[Apache Jenkins Server]

See 

--
GitHub pull request #6188 of commit 3f3e784b136f12ae6ffca29ff4766522ae409b92, 
no merge conflicts.
Setting status of 3f3e784b136f12ae6ffca29ff4766522ae409b92 to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2403/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/6188/*:refs/remotes/origin/pr/6188/*
 > git rev-parse refs/remotes/origin/pr/6188/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/6188/merge^{commit} # timeout=10
Checking out Revision 5676f4a64ed8f228ee4d82fe945ca66b5e4de813 
(refs/remotes/origin/pr/6188/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5676f4a64ed8f228ee4d82fe945ca66b5e4de813
Commit message: "Merge 3f3e784b136f12ae6ffca29ff4766522ae409b92 into 
1348cae2daaa403867fba192e4e06ff4addbf5ae"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_HadoopInputFormat.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Dataflow.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
ERROR: (PrecommitJobBuilder.groovy, line 79) No such property: steps for class: 
javaposse.jobdsl.dsl.jobs.FreeStyleJob

Build failed in Jenkins: beam_SeedJob #2402

[Apache Jenkins Server]

See 

--
GitHub pull request #6188 of commit 4353d9295466500fb2b069265e045ff298dd28de, 
no merge conflicts.
Setting status of 4353d9295466500fb2b069265e045ff298dd28de to PENDING with url 
https://builds.apache.org/job/beam_SeedJob/2402/ and message: 'Build started 
for merge commit.'
Using context: Jenkins: Seed Job
[EnvInject] - Loading node environment variables.
Building remotely on beam12 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/6188/*:refs/remotes/origin/pr/6188/*
 > git rev-parse refs/remotes/origin/pr/6188/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/6188/merge^{commit} # timeout=10
Checking out Revision 6d2f8d8179cb5605420b75eb2ed95785f0eba078 
(refs/remotes/origin/pr/6188/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6d2f8d8179cb5605420b75eb2ed95785f0eba078
Commit message: "Merge 4353d9295466500fb2b069265e045ff298dd28de into 
1348cae2daaa403867fba192e4e06ff4addbf5ae"
First time build. Skipping changelog.
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
Processing DSL script job_00_seed.groovy
Processing DSL script job_Dependency_Check.groovy
Processing DSL script job_Inventory.groovy
Processing DSL script job_PerformanceTests_Dataflow.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT.groovy
Processing DSL script job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
Processing DSL script job_PerformanceTests_HadoopInputFormat.groovy
Processing DSL script job_PerformanceTests_JDBC.groovy
Processing DSL script job_PerformanceTests_MongoDBIO_IT.groovy
Processing DSL script job_PerformanceTests_Python.groovy
Processing DSL script job_PerformanceTests_Spark.groovy
Processing DSL script job_PostCommit_Go_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_GradleBuild.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Dataflow.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Direct.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Flink.groovy
Processing DSL script job_PostCommit_Java_Nexmark_Spark.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Apex.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Flink.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Gearpump.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Samza.groovy
Processing DSL script job_PostCommit_Java_ValidatesRunner_Spark.groovy
Processing DSL script job_PostCommit_Python_ValidatesContainer_Dataflow.groovy
Processing DSL script job_PostCommit_Python_ValidatesRunner_Dataflow.groovy
Processing DSL script job_PostCommit_Python_Verify.groovy
Processing DSL script job_PostRelease_NightlySnapshot.groovy
Processing DSL script job_PreCommit_Go.groovy
ERROR: (PrecommitJobBuilder.groovy, line 79) No signature of method: 
javaposse.jobdsl.dsl.jobs.FreeStyleJob.switches() is applicable for argument 
types: (java.lang.String) values: [-PbuildTimer=30]
Possible solutions: with(groovy.lang.Closure)

[ANNOUNCE] Apache Beam 2.6.0 released!

[Pablo Estrada]

The Apache Beam team is pleased to announce the release of 2.6.0 version!

Apache Beam is an open source unified programming model to define and
execute data processing pipelines, including ETL, batch and stream
(continuous) processing. See https://beam.apache.org

You can download the release here:

https://beam.apache.org/get-started/downloads/

This release includes the following major new features & improvements,
among others:
- Improvements for internal Context Management in Python SDK
- A number of improvements to the Portability Framework
- A Universal Local Runner has been added to Beam. This runner runs in a
single machine using portability, and containerized SDK harnesses.
- Increased the coverage of ErrorProne analysis of the codebase.
- Updates to various dependency versions
- Updates to stability, performance, and documentation.
- SQL - improvements: support exists operator, implemented sum()
aggregations, fixes to CASE expression, support for date comparison,
support LIMIT on Unbounded Data
- Provide automatic schema registration for POJOs

You can take a look at the Release Notes for more details:

*https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343392=12319527
*

Thanks to everyone who participated in this release, and we hope you'll
have a good time using Beam 2.6.0.
--
Pablo Estrada, on behalf of The Apache Beam team
-- 
Got feedback? go/pabloem-feedback

Re: Renewing the Slack channel join link?

[Pablo Estrada]

Thanks Luke!

On Wed, Aug 8, 2018 at 3:04 PM Lukasz Cwik  wrote:

> Ask on the #general channel for a slack admin to regenerate a slack self
> invite link.
> Go to http://s.apache.org and paste the URL in the URL field and set the
> ID field to "slack-invite" and click "Get short-URL"
>
> On Wed, Aug 8, 2018 at 2:17 PM Pablo Estrada  wrote:
>
>> Hello all,
>> Can we renew the link please? Also, if y{all teach me how, I'll gladly
>> set reminders to do it every month : )
>> Best
>> -P.
>> --
>> Got feedback? go/pabloem-feedback
>> 
>>
> --
Got feedback? go/pabloem-feedback

Re: Renewing the Slack channel join link?

[Lukasz Cwik]

Ask on the #general channel for a slack admin to regenerate a slack self
invite link.
Go to http://s.apache.org and paste the URL in the URL field and set the ID
field to "slack-invite" and click "Get short-URL"

On Wed, Aug 8, 2018 at 2:17 PM Pablo Estrada  wrote:

> Hello all,
> Can we renew the link please? Also, if y{all teach me how, I'll gladly set
> reminders to do it every month : )
> Best
> -P.
> --
> Got feedback? go/pabloem-feedback
> 
>

Re: [VOTE] Community Examples Repository

[Ahmet Altay]

Charles, I agree with your comments and questions. I want to add one more
benefit that was mentioned earlier:

A place for people to contribute example that is not tied to the beam
release cycle. Such a repository could be a place for casual contributors
to add examples over time.

On Wed, Aug 8, 2018 at 2:25 PM, Charles Chen  wrote:

> It looks like the main claim is that 1 and 2 have the benefit of
> increasing visibility for examples on the Beam site.  I agree with Robert's
> comments on the doc which claim that this is orthogonal to whether a
> separate repository is created (the comments are unresolved:
> https://docs.google.com/a/google.com/document/d/
> 1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=BzifZxY).
>
> I would add that the maintenance and testing burden has not been
> adequately addressed in the proposal (i.e. are we creating new Jenkins
> jobs?; will postcommits on the main Beam repo run examples tests?; are we
> releasing artifacts--if so, is this together with the main package or
> separately in new packages?).  If we go with the half-way solution in (2),
> there is also the issue of where the threshold is--for example, if a
> user-contributed example is particularly useful, do we move it to the main
> repo?
>
> On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas  wrote:
>
>> I'd vote for 2.
>>
>> Giving independence to an example repository and creating the right
>> infrastructure to maintain them will give visibility to the efforts our
>> users are creating to solve their uses cases with Beam. I also want to make
>> the process of sharing common work more easily.
>>
>> Re:The examples that will remain in core, I agree that it's crucial to
>> keep some examples for testing.
>>
>>
>> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik  wrote:
>>
>>> I would vote for 3.
>>>
>>> My reasoning is that Java has a good mechanism to get a starter/example
>>> project going by using the the maven archetypes already. Our quickstart
>>> guide for Apache Beam for the Java SDK already covers generating the
>>> examples archetype.
>>> We could point users to the starter project at the end of the java
>>> quickstart.
>>>
>>> If python/go have a similar mechanism that is commonly used, I would go
>>> with those over creating a separate repo for examples and adding the
>>> maintenance burden involved.
>>>
>>>
>>>
>>> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang  wrote:
>>>
 2 - examples that rely on experimental API can still stay in where they
 are because such examples could be changed.

 -Rui

 On Wed, Aug 8, 2018 at 10:52 AM Charles Chen  wrote:

> 3 - We benefit from increased test coverage by having examples
> together with the rest of the code.  As Robert mentions in the doc, 
> hosting
> the Beam examples in the main repository is the best way to keep the
> examples visible, tested and maintained.  Given that we recently moved to 
> a
> single repository for the website since that previously caused a lot of
> pain, it makes sense to be consistent here.
>
> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay  wrote:
>
>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>> tested and maintained against head. At the same time I understand the 
>> value
>> of a growing set of community grown examples that are targeted against a
>> pre-defined versions of Beam and not necessarily updated at every 
>> release.
>>
>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan > > wrote:
>>
>>> 2 - I like the idea of having a separate repo where we can have more
>>> freedom to check in examples. However, we benefit from having immediate
>>> core examples in Beam for testing purposes.
>>>
>>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos 
>>> wrote:
>>>
 Hi everyone!

 We discussed several options as well as some of the implications of
 each option. Please vote for your favorite option, feel free to back 
 it up
 with any reasons that make you feel that way.

 1) Move *all* samples to a *new *examples* repository*
 2) Move *some* samples to a *new *examples* repository*
 3) Leave samples where they are

 Some implications to creating a new repository:
 - Every example would be independent from every other example, so
 tests can be run in parallel
 - Examples would now show how to use Beam *externally*
 - The examples repository would need a testing infrastructure
 - Decoupling makes examples easier to test on different versions
 - Easier to copy-paste an existing example and start from there,
 almost like a template
 - Smaller size for the core Beam library
 - Two different repositories to maintain
 - Versioning could mirror Beam's current version

 Link to proposal
 

Re: [VOTE] Community Examples Repository

[Charles Chen]

It looks like the main claim is that 1 and 2 have the benefit of increasing
visibility for examples on the Beam site.  I agree with Robert's comments
on the doc which claim that this is orthogonal to whether a separate
repository is created (the comments are unresolved:
https://docs.google.com/a/google.com/document/d/1vhcKJlP0qH1C7NZPDjohT2PUbOD-k71avv1CjEYapdw/edit?disco=BzifZxY
).

I would add that the maintenance and testing burden has not been adequately
addressed in the proposal (i.e. are we creating new Jenkins jobs?; will
postcommits on the main Beam repo run examples tests?; are we releasing
artifacts--if so, is this together with the main package or separately in
new packages?).  If we go with the half-way solution in (2), there is also
the issue of where the threshold is--for example, if a user-contributed
example is particularly useful, do we move it to the main repo?

On Wed, Aug 8, 2018 at 1:35 PM Griselda Cuevas  wrote:

> I'd vote for 2.
>
> Giving independence to an example repository and creating the right
> infrastructure to maintain them will give visibility to the efforts our
> users are creating to solve their uses cases with Beam. I also want to make
> the process of sharing common work more easily.
>
> Re:The examples that will remain in core, I agree that it's crucial to
> keep some examples for testing.
>
>
> On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik  wrote:
>
>> I would vote for 3.
>>
>> My reasoning is that Java has a good mechanism to get a starter/example
>> project going by using the the maven archetypes already. Our quickstart
>> guide for Apache Beam for the Java SDK already covers generating the
>> examples archetype.
>> We could point users to the starter project at the end of the java
>> quickstart.
>>
>> If python/go have a similar mechanism that is commonly used, I would go
>> with those over creating a separate repo for examples and adding the
>> maintenance burden involved.
>>
>>
>>
>> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang  wrote:
>>
>>> 2 - examples that rely on experimental API can still stay in where they
>>> are because such examples could be changed.
>>>
>>> -Rui
>>>
>>> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen  wrote:
>>>
 3 - We benefit from increased test coverage by having examples together
 with the rest of the code.  As Robert mentions in the doc, hosting the Beam
 examples in the main repository is the best way to keep the examples
 visible, tested and maintained.  Given that we recently moved to a single
 repository for the website since that previously caused a lot of pain, it
 makes sense to be consistent here.

 On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay  wrote:

> 2 - Similar to Huygaa, I see value in keeping a core set of examples
> tested and maintained against head. At the same time I understand the 
> value
> of a growing set of community grown examples that are targeted against a
> pre-defined versions of Beam and not necessarily updated at every release.
>
> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan 
> wrote:
>
>> 2 - I like the idea of having a separate repo where we can have more
>> freedom to check in examples. However, we benefit from having immediate
>> core examples in Beam for testing purposes.
>>
>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos 
>> wrote:
>>
>>> Hi everyone!
>>>
>>> We discussed several options as well as some of the implications of
>>> each option. Please vote for your favorite option, feel free to back it 
>>> up
>>> with any reasons that make you feel that way.
>>>
>>> 1) Move *all* samples to a *new *examples* repository*
>>> 2) Move *some* samples to a *new *examples* repository*
>>> 3) Leave samples where they are
>>>
>>> Some implications to creating a new repository:
>>> - Every example would be independent from every other example, so
>>> tests can be run in parallel
>>> - Examples would now show how to use Beam *externally*
>>> - The examples repository would need a testing infrastructure
>>> - Decoupling makes examples easier to test on different versions
>>> - Easier to copy-paste an existing example and start from there,
>>> almost like a template
>>> - Smaller size for the core Beam library
>>> - Two different repositories to maintain
>>> - Versioning could mirror Beam's current version
>>>
>>> Link to proposal
>>> 
>>>
>>
>

Renewing the Slack channel join link?

[Pablo Estrada]

Hello all,
Can we renew the link please? Also, if y{all teach me how, I'll gladly set
reminders to do it every month : )
Best
-P.
-- 
Got feedback? go/pabloem-feedback

Re: [VOTE] Community Examples Repository

[Griselda Cuevas]

I'd vote for 2.

Giving independence to an example repository and creating the right
infrastructure to maintain them will give visibility to the efforts our
users are creating to solve their uses cases with Beam. I also want to make
the process of sharing common work more easily.

Re:The examples that will remain in core, I agree that it's crucial to keep
some examples for testing.


On Wed, 8 Aug 2018 at 11:44, Lukasz Cwik  wrote:

> I would vote for 3.
>
> My reasoning is that Java has a good mechanism to get a starter/example
> project going by using the the maven archetypes already. Our quickstart
> guide for Apache Beam for the Java SDK already covers generating the
> examples archetype.
> We could point users to the starter project at the end of the java
> quickstart.
>
> If python/go have a similar mechanism that is commonly used, I would go
> with those over creating a separate repo for examples and adding the
> maintenance burden involved.
>
>
>
> On Wed, Aug 8, 2018 at 11:01 AM Rui Wang  wrote:
>
>> 2 - examples that rely on experimental API can still stay in where they
>> are because such examples could be changed.
>>
>> -Rui
>>
>> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen  wrote:
>>
>>> 3 - We benefit from increased test coverage by having examples together
>>> with the rest of the code.  As Robert mentions in the doc, hosting the Beam
>>> examples in the main repository is the best way to keep the examples
>>> visible, tested and maintained.  Given that we recently moved to a single
>>> repository for the website since that previously caused a lot of pain, it
>>> makes sense to be consistent here.
>>>
>>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay  wrote:
>>>
 2 - Similar to Huygaa, I see value in keeping a core set of examples
 tested and maintained against head. At the same time I understand the value
 of a growing set of community grown examples that are targeted against a
 pre-defined versions of Beam and not necessarily updated at every release.

 On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan 
 wrote:

> 2 - I like the idea of having a separate repo where we can have more
> freedom to check in examples. However, we benefit from having immediate
> core examples in Beam for testing purposes.
>
> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos 
> wrote:
>
>> Hi everyone!
>>
>> We discussed several options as well as some of the implications of
>> each option. Please vote for your favorite option, feel free to back it 
>> up
>> with any reasons that make you feel that way.
>>
>> 1) Move *all* samples to a *new *examples* repository*
>> 2) Move *some* samples to a *new *examples* repository*
>> 3) Leave samples where they are
>>
>> Some implications to creating a new repository:
>> - Every example would be independent from every other example, so
>> tests can be run in parallel
>> - Examples would now show how to use Beam *externally*
>> - The examples repository would need a testing infrastructure
>> - Decoupling makes examples easier to test on different versions
>> - Easier to copy-paste an existing example and start from there,
>> almost like a template
>> - Smaller size for the core Beam library
>> - Two different repositories to maintain
>> - Versioning could mirror Beam's current version
>>
>> Link to proposal
>> 
>>
>

Re: [VOTE] Community Examples Repository

[Jean-Baptiste Onofré]

I would go for 1 or 3 to be consistent.

Regards
JB

On 08/08/2018 18:38, David Cavazos wrote:
> Hi everyone!
> 
> We discussed several options as well as some of the implications of each
> option. Please vote for your favorite option, feel free to back it up
> with any reasons that make you feel that way.
> 
> 1) Move *all* samples to a *new *examples*repository*
> 2) Move *some* samples to a *new *examples*repository*
> 3) Leave samples where they are
> 
> Some implications to creating a new repository:
> - Every example would be independent from every other example, so tests
> can be run in parallel
> - Examples would now show how to use Beam /externally/
> - The examples repository would need a testing infrastructure
> - Decoupling makes examples easier to test on different versions
> - Easier to copy-paste an existing example and start from there, almost
> like a template
> - Smaller size for the core Beam library
> - Two different repositories to maintain
> - Versioning could mirror Beam's current version
> 
> Link to proposal
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Schema Aware PCollections

[Anton Kedin]

Yes, this should be possible eventually. In fact, limited version of this
functionality is already supported for Beans (e.g. see this test
),
but it's still experimental and there are no good end-to-end examples yet.

Regards,
Anton

On Wed, Aug 8, 2018 at 5:45 AM Akanksha Sharma B <
akanksha.b.sha...@ericsson.com> wrote:

> Hi,
>
>
> (changed the email-subject to make it generic)
>
>
> It is mentioned in Schema-Aware PCollections design doc (
> https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc
> )
>
>
> "There are a number of existing data types from which schemas can be
> inferred. Protocol buffers, Avro objects, Json objects, POJOs, primitive
> Java types - all of these have schemas that can be inferred from the type
> itself at pipeline-construction time. We should be able to automatically
> infer these schemas with a minimum of involvement from the programmer. "
>
> Can I assume that the following usecase will be possible sometime in
> future :-
> "read parquet (along with inferred schema) into something like dataframe
> or Beam Rows. And vice versa for write i.e. get rows and write parquet
> based on Row's schema.""
>
> Regards,
> Akanksha
>
> --
> *From:* Chamikara Jayalath 
> *Sent:* Wednesday, August 1, 2018 3:57 PM
> *To:* u...@beam.apache.org
> *Cc:* dev@beam.apache.org
> *Subject:* Re: pipeline with parquet and sql
>
>
>
> On Wed, Aug 1, 2018 at 1:12 AM Akanksha Sharma B <
> akanksha.b.sha...@ericsson.com> wrote:
>
> Hi,
>
>
> Thanks. I understood the Parquet point. I will wait for couple of days on
> this topic. Even if this scenario cannot be achieved now, any design
> document or future plans towards this direction will also be helpful to me.
>
>
> To summarize, I do not understand beam well enough, can someone please
> help me and comment whether the following fits with beam's model and
> future direction :-
>
> "read parquet (along with inferred schema) into something like dataframe
> or Beam Rows. And vice versa for write i.e. get rows and write parquet
> based on Row's schema."
>
>
> Beam currently does not have a standard message format. A Beam pipeline
> consists of PCollections and transforms (that converts PCollections to
> other PCollections). You can transform the PCollection read from Parquet
> using a ParDo and writing the resulting transform back to Parquet format. I
> think Schema aware PCollections [1] might be close to what you need but not
> sure if it fulfills your exact requirement.
>
> Thanks,
> Cham
>
> [1]
> https://lists.apache.org/thread.html/fe327866c6c81b7e55af28f81cedd9b2e588279def330940e8b8ebd7@%3Cdev.beam.apache.org%3E
>
>
>
>
>
> Regards,
>
> Akanksha
>
>
> --
> *From:* Łukasz Gajowy 
> *Sent:* Tuesday, July 31, 2018 12:43:32 PM
> *To:* u...@beam.apache.org
> *Cc:* dev@beam.apache.org
> *Subject:* Re: pipeline with parquet and sql
>
> In terms of schema and ParquetIO source/sink, there was an answer in some
> previous thread:
>
> Currently (without introducing any change in ParquetIO) there is no way to
> not pass the avro schema. It will probably be replaced with Beam's schema
> in the future ()
>
> [1]
> https://lists.apache.org/thread.html/a466ddeb55e47fd780be3bcd8eec9d6b6eaf1dfd566ae5278b5fb9e8@%3Cuser.beam.apache.org%3E
>
>
> wt., 31 lip 2018 o 10:19 Akanksha Sharma B 
> napisał(a):
>
> Hi,
>
>
> I am hoping to get some hints/pointers from the experts here.
>
> I hope the scenario described below was understandable. I hope it is a
> valid use-case. Please let me know if I need to explain the scenario
> better.
>
>
> Regards,
>
> Akanksha
>
> --
> *From:* Akanksha Sharma B
> *Sent:* Friday, July 27, 2018 9:44 AM
> *To:* dev@beam.apache.org
> *Subject:* Re: pipeline with parquet and sql
>
>
> Hi,
>
>
> Please consider following pipeline:-
>
>
> Source is Parquet file, having hundreds of columns.
>
> Sink is Parquet. Multiple output parquet files are generated after
> applying some sql joins. Sql joins to be applied differ for each output
> parquet file. Lets assume we have a sql queries generator or some
> configuration file with the needed info.
>
>
> Can this be implemented generically, such that there is no need of the
> schema of the parquet files involved or any intermediate POJO or beam
> schema.
>
> i.e. the way spark can handle it - read parquet into dataframe, create
> temp view and apply sql queries to it, and write it back to parquet.
>
> As I understand, beam SQL needs (Beam Schema or POJOs) and parquetIO needs
> avro schemas. Ideally we dont want to see POJOs or schemas.
> If there is a way we can achieve this with beam, please do help.
>
> Regards,
> Akanksha
>
> --
> *From:* Akanksha Sharma B
> *Sent:* Tuesday, July 24, 2018 4:47:25 PM
> *To:* 

Re: [VOTE] Community Examples Repository

[Lukasz Cwik]

I would vote for 3.

My reasoning is that Java has a good mechanism to get a starter/example
project going by using the the maven archetypes already. Our quickstart
guide for Apache Beam for the Java SDK already covers generating the
examples archetype.
We could point users to the starter project at the end of the java
quickstart.

If python/go have a similar mechanism that is commonly used, I would go
with those over creating a separate repo for examples and adding the
maintenance burden involved.



On Wed, Aug 8, 2018 at 11:01 AM Rui Wang  wrote:

> 2 - examples that rely on experimental API can still stay in where they
> are because such examples could be changed.
>
> -Rui
>
> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen  wrote:
>
>> 3 - We benefit from increased test coverage by having examples together
>> with the rest of the code.  As Robert mentions in the doc, hosting the Beam
>> examples in the main repository is the best way to keep the examples
>> visible, tested and maintained.  Given that we recently moved to a single
>> repository for the website since that previously caused a lot of pain, it
>> makes sense to be consistent here.
>>
>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay  wrote:
>>
>>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>>> tested and maintained against head. At the same time I understand the value
>>> of a growing set of community grown examples that are targeted against a
>>> pre-defined versions of Beam and not necessarily updated at every release.
>>>
>>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan 
>>> wrote:
>>>
 2 - I like the idea of having a separate repo where we can have more
 freedom to check in examples. However, we benefit from having immediate
 core examples in Beam for testing purposes.

 On Wed, Aug 8, 2018 at 9:38 AM David Cavazos 
 wrote:

> Hi everyone!
>
> We discussed several options as well as some of the implications of
> each option. Please vote for your favorite option, feel free to back it up
> with any reasons that make you feel that way.
>
> 1) Move *all* samples to a *new *examples* repository*
> 2) Move *some* samples to a *new *examples* repository*
> 3) Leave samples where they are
>
> Some implications to creating a new repository:
> - Every example would be independent from every other example, so
> tests can be run in parallel
> - Examples would now show how to use Beam *externally*
> - The examples repository would need a testing infrastructure
> - Decoupling makes examples easier to test on different versions
> - Easier to copy-paste an existing example and start from there,
> almost like a template
> - Smaller size for the core Beam library
> - Two different repositories to maintain
> - Versioning could mirror Beam's current version
>
> Link to proposal
> 
>

>>>

Re: [VOTE] Community Examples Repository

[Andrea Foegler]

I guess I'm voting for 2.  Tests obviously belong in the code repo, both as
a sample usage (not "how-to", more like "man") and, well, for testing.
These pipelines might not be annotated as completely and include hitting
edge cases and other non-standard situations.

Any example where the primary purpose is external consumption seems totally
reasonable to move out.

Regardless of where they are stored, trying to share these artifacts
explicitly seems less than ideal.  The cost of duplication here seems low.

On Wed, Aug 8, 2018 at 11:01 AM Rui Wang  wrote:

> 2 - examples that rely on experimental API can still stay in where they
> are because such examples could be changed.
>
> -Rui
>
> On Wed, Aug 8, 2018 at 10:52 AM Charles Chen  wrote:
>
>> 3 - We benefit from increased test coverage by having examples together
>> with the rest of the code.  As Robert mentions in the doc, hosting the Beam
>> examples in the main repository is the best way to keep the examples
>> visible, tested and maintained.  Given that we recently moved to a single
>> repository for the website since that previously caused a lot of pain, it
>> makes sense to be consistent here.
>>
>> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay  wrote:
>>
>>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>>> tested and maintained against head. At the same time I understand the value
>>> of a growing set of community grown examples that are targeted against a
>>> pre-defined versions of Beam and not necessarily updated at every release.
>>>
>>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan 
>>> wrote:
>>>
 2 - I like the idea of having a separate repo where we can have more
 freedom to check in examples. However, we benefit from having immediate
 core examples in Beam for testing purposes.

 On Wed, Aug 8, 2018 at 9:38 AM David Cavazos 
 wrote:

> Hi everyone!
>
> We discussed several options as well as some of the implications of
> each option. Please vote for your favorite option, feel free to back it up
> with any reasons that make you feel that way.
>
> 1) Move *all* samples to a *new *examples* repository*
> 2) Move *some* samples to a *new *examples* repository*
> 3) Leave samples where they are
>
> Some implications to creating a new repository:
> - Every example would be independent from every other example, so
> tests can be run in parallel
> - Examples would now show how to use Beam *externally*
> - The examples repository would need a testing infrastructure
> - Decoupling makes examples easier to test on different versions
> - Easier to copy-paste an existing example and start from there,
> almost like a template
> - Smaller size for the core Beam library
> - Two different repositories to maintain
> - Versioning could mirror Beam's current version
>
> Link to proposal
> 
>

>>>

Re: [VOTE] Community Examples Repository

[Rui Wang]

2 - examples that rely on experimental API can still stay in where they are
because such examples could be changed.

-Rui

On Wed, Aug 8, 2018 at 10:52 AM Charles Chen  wrote:

> 3 - We benefit from increased test coverage by having examples together
> with the rest of the code.  As Robert mentions in the doc, hosting the Beam
> examples in the main repository is the best way to keep the examples
> visible, tested and maintained.  Given that we recently moved to a single
> repository for the website since that previously caused a lot of pain, it
> makes sense to be consistent here.
>
> On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay  wrote:
>
>> 2 - Similar to Huygaa, I see value in keeping a core set of examples
>> tested and maintained against head. At the same time I understand the value
>> of a growing set of community grown examples that are targeted against a
>> pre-defined versions of Beam and not necessarily updated at every release.
>>
>> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan 
>> wrote:
>>
>>> 2 - I like the idea of having a separate repo where we can have more
>>> freedom to check in examples. However, we benefit from having immediate
>>> core examples in Beam for testing purposes.
>>>
>>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos 
>>> wrote:
>>>
 Hi everyone!

 We discussed several options as well as some of the implications of
 each option. Please vote for your favorite option, feel free to back it up
 with any reasons that make you feel that way.

 1) Move *all* samples to a *new *examples* repository*
 2) Move *some* samples to a *new *examples* repository*
 3) Leave samples where they are

 Some implications to creating a new repository:
 - Every example would be independent from every other example, so tests
 can be run in parallel
 - Examples would now show how to use Beam *externally*
 - The examples repository would need a testing infrastructure
 - Decoupling makes examples easier to test on different versions
 - Easier to copy-paste an existing example and start from there, almost
 like a template
 - Smaller size for the core Beam library
 - Two different repositories to maintain
 - Versioning could mirror Beam's current version

 Link to proposal
 

>>>
>>

Re: [VOTE] Community Examples Repository

[Charles Chen]

3 - We benefit from increased test coverage by having examples together
with the rest of the code.  As Robert mentions in the doc, hosting the Beam
examples in the main repository is the best way to keep the examples
visible, tested and maintained.  Given that we recently moved to a single
repository for the website since that previously caused a lot of pain, it
makes sense to be consistent here.

On Wed, Aug 8, 2018 at 10:27 AM Ahmet Altay  wrote:

> 2 - Similar to Huygaa, I see value in keeping a core set of examples
> tested and maintained against head. At the same time I understand the value
> of a growing set of community grown examples that are targeted against a
> pre-defined versions of Beam and not necessarily updated at every release.
>
> On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan 
> wrote:
>
>> 2 - I like the idea of having a separate repo where we can have more
>> freedom to check in examples. However, we benefit from having immediate
>> core examples in Beam for testing purposes.
>>
>> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos  wrote:
>>
>>> Hi everyone!
>>>
>>> We discussed several options as well as some of the implications of each
>>> option. Please vote for your favorite option, feel free to back it up with
>>> any reasons that make you feel that way.
>>>
>>> 1) Move *all* samples to a *new *examples* repository*
>>> 2) Move *some* samples to a *new *examples* repository*
>>> 3) Leave samples where they are
>>>
>>> Some implications to creating a new repository:
>>> - Every example would be independent from every other example, so tests
>>> can be run in parallel
>>> - Examples would now show how to use Beam *externally*
>>> - The examples repository would need a testing infrastructure
>>> - Decoupling makes examples easier to test on different versions
>>> - Easier to copy-paste an existing example and start from there, almost
>>> like a template
>>> - Smaller size for the core Beam library
>>> - Two different repositories to maintain
>>> - Versioning could mirror Beam's current version
>>>
>>> Link to proposal
>>> 
>>>
>>
>

Re: Looking for a re-review of 6101

[Jean-Baptiste Onofré]

Hi,

I gonna take a look.

Regards
JB

On 08/08/2018 19:38, John Rudolf Lewis wrote:
> I'm looking for a re-review of my SqsIO contribution.
> 
> https://github.com/apache/beam/pull/6101
> 
> I would like to see this accepted into master so that we can start to
> use the daily builds in one of our projects.

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Looking for a re-review of 6101

[John Rudolf Lewis]

I'm looking for a re-review of my SqsIO contribution.

https://github.com/apache/beam/pull/6101

I would like to see this accepted into master so that we can start to use
the daily builds in one of our projects.

Re: Portable streaming side inputs

[Lukasz Cwik]

I have been using this inside Dataflow with the PushbackSideInputDoFnRunner
as the window mapping fn:
https://github.com/lukecwik/incubator-beam/commit/4324185192b27026270ee342b01720ff40e71df8

The input is a KV with the key being a nonce and the value being a window,
the output must be a KV with the key being the same nonce as the input and
the value being the mapped window.
All window mapping fns are deterministic and all window encodings are
deterministic which means that you can cache the encoded output window
based upon the encoded input window after it has been mapped once
indefinitely.


On Wed, Aug 8, 2018 at 9:08 AM Thomas Weise  wrote:

> I'm working on support for side inputs in streaming mode in the portable
> Flink runner [1].
>
> The runner would be responsible for holding main inputs (durably) when
> side inputs are not available.
>
> To check if side inputs are available, the window mapping function is
> required (see SimplePushbackSideInputDoFnRunner.isReady).
>
> The window mapping function (like viewFn) is SDK specific serialized in
> the proto (see PCollectionViewTranslation).
>
> With Python SDK, the Flink runner cannot rehydrate these objects.
>
> Is my understanding correct and what assumptions can be made about the
> window mapping in the runner?
>
> Thanks,
> Thomas
>
> [1] https://issues.apache.org/jira/browse/BEAM-2930
>

Re: [VOTE] Community Examples Repository

[Ahmet Altay]

2 - Similar to Huygaa, I see value in keeping a core set of examples tested
and maintained against head. At the same time I understand the value of a
growing set of community grown examples that are targeted against a
pre-defined versions of Beam and not necessarily updated at every release.

On Wed, Aug 8, 2018 at 10:22 AM, Huygaa Batsaikhan 
wrote:

> 2 - I like the idea of having a separate repo where we can have more
> freedom to check in examples. However, we benefit from having immediate
> core examples in Beam for testing purposes.
>
> On Wed, Aug 8, 2018 at 9:38 AM David Cavazos  wrote:
>
>> Hi everyone!
>>
>> We discussed several options as well as some of the implications of each
>> option. Please vote for your favorite option, feel free to back it up with
>> any reasons that make you feel that way.
>>
>> 1) Move *all* samples to a *new *examples* repository*
>> 2) Move *some* samples to a *new *examples* repository*
>> 3) Leave samples where they are
>>
>> Some implications to creating a new repository:
>> - Every example would be independent from every other example, so tests
>> can be run in parallel
>> - Examples would now show how to use Beam *externally*
>> - The examples repository would need a testing infrastructure
>> - Decoupling makes examples easier to test on different versions
>> - Easier to copy-paste an existing example and start from there, almost
>> like a template
>> - Smaller size for the core Beam library
>> - Two different repositories to maintain
>> - Versioning could mirror Beam's current version
>>
>> Link to proposal
>> 
>>
>

Re: [VOTE] Community Examples Repository

[Huygaa Batsaikhan]

2 - I like the idea of having a separate repo where we can have more
freedom to check in examples. However, we benefit from having immediate
core examples in Beam for testing purposes.

On Wed, Aug 8, 2018 at 9:38 AM David Cavazos  wrote:

> Hi everyone!
>
> We discussed several options as well as some of the implications of each
> option. Please vote for your favorite option, feel free to back it up with
> any reasons that make you feel that way.
>
> 1) Move *all* samples to a *new *examples* repository*
> 2) Move *some* samples to a *new *examples* repository*
> 3) Leave samples where they are
>
> Some implications to creating a new repository:
> - Every example would be independent from every other example, so tests
> can be run in parallel
> - Examples would now show how to use Beam *externally*
> - The examples repository would need a testing infrastructure
> - Decoupling makes examples easier to test on different versions
> - Easier to copy-paste an existing example and start from there, almost
> like a template
> - Smaller size for the core Beam library
> - Two different repositories to maintain
> - Versioning could mirror Beam's current version
>
> Link to proposal
> 
>

[VOTE] Community Examples Repository

[David Cavazos]

Hi everyone!

We discussed several options as well as some of the implications of each
option. Please vote for your favorite option, feel free to back it up with
any reasons that make you feel that way.

1) Move *all* samples to a *new *examples* repository*
2) Move *some* samples to a *new *examples* repository*
3) Leave samples where they are

Some implications to creating a new repository:
- Every example would be independent from every other example, so tests can
be run in parallel
- Examples would now show how to use Beam *externally*
- The examples repository would need a testing infrastructure
- Decoupling makes examples easier to test on different versions
- Easier to copy-paste an existing example and start from there, almost
like a template
- Smaller size for the core Beam library
- Two different repositories to maintain
- Versioning could mirror Beam's current version

Link to proposal

Portable streaming side inputs

[Thomas Weise]

I'm working on support for side inputs in streaming mode in the portable
Flink runner [1].

The runner would be responsible for holding main inputs (durably) when side
inputs are not available.

To check if side inputs are available, the window mapping function is
required (see SimplePushbackSideInputDoFnRunner.isReady).

The window mapping function (like viewFn) is SDK specific serialized in the
proto (see PCollectionViewTranslation).

With Python SDK, the Flink runner cannot rehydrate these objects.

Is my understanding correct and what assumptions can be made about the
window mapping in the runner?

Thanks,
Thomas

[1] https://issues.apache.org/jira/browse/BEAM-2930

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #135

[Apache Jenkins Server]

See 


Changes:

[apilloud] Pin Dataflow Nexmark Autoscale to 4

[relax] Make sure that SchemaCoder overrides consistentWithEquals.

[relax] Address code-review comments.

[apilloud] Make event counts match documentation

[amaliujia] improve BeamSqlRowComparator

[apilloud] [BEAM-4807] [SQL] Upgrade Calcite and Avatica

[apilloud] Drop extra projection on uncollect

[pablo] Interactive Beam - enable caching in GCS

--
[...truncated 17.94 MB...]
Task ':beam-sdks-python-container:prepare' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
Use project GOPATH: 

:beam-sdks-python-container:prepare (Thread[Task worker for ':' Thread 
4,5,main]) completed. Took 0.001 secs.
:beam-sdks-python-container:resolveBuildDependencies (Thread[Task worker for 
':' Thread 4,5,main]) started.

> Task :beam-sdks-python-container:resolveBuildDependencies
Build cache key for task ':beam-sdks-python-container:resolveBuildDependencies' 
is cc0a0752f032ffadab710cf660052203
Caching disabled for task 
':beam-sdks-python-container:resolveBuildDependencies': Caching has not been 
enabled for the task
Task ':beam-sdks-python-container:resolveBuildDependencies' is not up-to-date 
because:
  No history is available.
Cache 

 not found, skip.
Cache 

 not found, skip.
Resolving 
./github.com/apache/beam/sdks/go@
:beam-sdks-python-container:resolveBuildDependencies (Thread[Task worker for 
':' Thread 4,5,main]) completed. Took 1.551 secs.
:beam-sdks-python-container:installDependencies (Thread[Task worker for ':' 
Thread 4,5,main]) started.

> Task :beam-sdks-python-container:installDependencies
Caching disabled for task ':beam-sdks-python-container:installDependencies': 
Caching has not been enabled for the task
Task ':beam-sdks-python-container:installDependencies' is not up-to-date 
because:
  Task has not declared any outputs despite executing actions.
Cache 

 not found, skip.
:beam-sdks-python-container:installDependencies (Thread[Task worker for ':' 
Thread 4,5,main]) completed. Took 0.613 secs.
:beam-sdks-python-container:buildLinuxAmd64 (Thread[Task worker for ':' Thread 
4,5,main]) started.

> Task :beam-sdks-python-container:buildLinuxAmd64
Build cache key for task ':beam-sdks-python-container:buildLinuxAmd64' is 
a418a33950c19ee57dbfc55fc73a1bb6
Caching disabled for task ':beam-sdks-python-container:buildLinuxAmd64': 
Caching has not been enabled for the task
Task ':beam-sdks-python-container:buildLinuxAmd64' is not up-to-date because:
  No history is available.
:beam-sdks-python-container:buildLinuxAmd64 (Thread[Task worker for ':' Thread 
4,5,main]) completed. Took 3.126 secs.
:beam-sdks-python-container:build (Thread[Task worker for ':' Thread 4,5,main]) 
started.

> Task :beam-sdks-python-container:build
Caching disabled for task ':beam-sdks-python-container:build': Caching has not 
been enabled for the task
Task ':beam-sdks-python-container:build' is not up-to-date because:
  Task has not declared any outputs despite executing actions.
:beam-sdks-python-container:build (Thread[Task worker for ':' Thread 4,5,main]) 
completed. Took 0.002 secs.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 4,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:jar
Build cache key for task ':beam-vendor-sdks-java-extensions-protobuf:jar' is 
318184f3acfe150a3a38d605826c992a
Caching disabled for task ':beam-vendor-sdks-java-extensions-protobuf:jar': 
Caching has not been enabled for the task
Task ':beam-vendor-sdks-java-extensions-protobuf:jar' is not up-to-date because:
  No history is available.
:beam-vendor-sdks-java-extensions-protobuf:jar (Thread[Task worker for ':' 
Thread 4,5,main]) completed. Took 0.008 secs.
:beam-vendor-sdks-java-extensions-protobuf:compileTestJava (Thread[Task worker 
for ':' Thread 4,5,main]) started.

> Task :beam-vendor-sdks-java-extensions-protobuf:compileTestJava NO-SOURCE
file or directory 
'
 not found
Skipping task ':beam-vendor-sdks-java-extensions-protobuf:compileTestJava' as 
it has no source files and no