Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Alex Van Boxel]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ X] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ X] Angler fish

 _/
_/ Alex Van Boxel


On Wed, Nov 20, 2019 at 3:57 AM Reza Rokni  wrote:

> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [X] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [X] Dumbo Octopus
> [X] Angler fish
>
> On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles  wrote:
>
>> Please cast your votes of approval [1] for animals you would support as
>> Beam mascot. The animal with the most approval will be identified as the
>> favorite.
>>
>> *** Vote for as many as you like, using this checklist as a template 
>>
>> [ ] Beaver
>> [ ] Hedgehog
>> [ ] Lemur
>> [ ] Owl
>> [ ] Salmon
>> [ ] Trout
>> [ ] Robot dinosaur
>> [ ] Firefly
>> [ ] Cuttlefish
>> [ ] Dumbo Octopus
>> [ ] Angler fish
>>
>> This vote will remain open for at least 72 hours.
>>
>> Kenn
>>
>> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
>> https://www.electionscience.org/library/approval-voting/
>>
>
>
> --
>
> This email may be confidential and privileged. If you received this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person.
>
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and do
> not constitute a legally binding obligation. No legally binding obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>

Re: [ANNOUNCE] New committer: Brian Hulette

[Tanay Tummalapalli]

Congratulations!

On Wed, Nov 20, 2019 at 6:15 AM Aizhamal Nurmamat kyzy 
wrote:

> Congratulations, Brian!
>
> On Mon, Nov 18, 2019 at 10:29 AM Łukasz Gajowy 
> wrote:
>
>> Congratulations! :)
>>
>> pt., 15 lis 2019 o 15:52 Chamikara Jayalath 
>> napisał(a):
>>
>>> Congrats, Brian!
>>>
>>> On Fri, Nov 15, 2019 at 11:39 AM Yichi Zhang  wrote:
>>>
 Congratulations, Brian!

 On Fri, Nov 15, 2019 at 11:12 AM Alan Myrvold 
 wrote:

> Congratulations, Brian!
>
> On Fri, Nov 15, 2019 at 11:11 AM Yifan Zou 
> wrote:
>
>> Well deserved. Congratulations!
>>
>> On Fri, Nov 15, 2019 at 11:06 AM Kirill Kozlov <
>> kirillkoz...@google.com> wrote:
>>
>>> Congratulations, Brian!
>>>
>>> On Fri, Nov 15, 2019 at 10:56 AM Udi Meiri  wrote:
>>>
 Congrats Brian!


 On Fri, Nov 15, 2019 at 10:47 AM Ruoyun Huang 
 wrote:

> Congrats Brian!
>
> On Fri, Nov 15, 2019 at 10:41 AM Robin Qiu 
> wrote:
>
>> Congrats, Brian!
>>
>> On Fri, Nov 15, 2019 at 10:02 AM Daniel Oliveira <
>> danolive...@google.com> wrote:
>>
>>> Congratulations Brian! It's well deserved.
>>>
>>> On Fri, Nov 15, 2019, 9:37 AM Alexey Romanenko <
>>> aromanenko@gmail.com> wrote:
>>>
 Congratulations, Brian!

 On 15 Nov 2019, at 18:27, Rui Wang  wrote:

 Congrats!


 -Rui

 On Fri, Nov 15, 2019 at 8:16 AM Thomas Weise 
 wrote:

> Congratulations!
>
>
> On Fri, Nov 15, 2019 at 6:34 AM Connell O'Callaghan <
> conne...@google.com> wrote:
>
>> Well done Brian!!!
>>
>> Kenn thank you for sharing
>>
>> On Fri, Nov 15, 2019 at 6:31 AM Cyrus Maden <
>> cma...@google.com> wrote:
>>
>>> Congrats Brian!
>>>
>>> On Fri, Nov 15, 2019 at 5:25 AM Ismaël Mejía <
>>> ieme...@gmail.com> wrote:
>>>
 Congratulations Brian!
 Happy to see this happening and eager to see more of your
 work!

 On Fri, Nov 15, 2019 at 11:02 AM Ankur Goenka <
 goe...@google.com> wrote:
 >
 > Congrats Brian!
 >
 > On Fri, Nov 15, 2019, 2:42 PM Jan Lukavský <
 je...@seznam.cz> wrote:
 >>
 >> Congrats Brian!
 >>
 >> On 11/15/19 9:58 AM, Reza Rokni wrote:
 >>
 >> Great news!
 >>
 >> On Fri, 15 Nov 2019 at 15:09, Gleb Kanterov <
 g...@spotify.com> wrote:
 >>>
 >>> Congratulations!
 >>>
 >>> On Fri, Nov 15, 2019 at 5:44 AM Valentyn Tymofieiev <
 valen...@google.com> wrote:
 
  Congratulations, Brian!
 
  On Thu, Nov 14, 2019 at 6:25 PM jincheng sun <
 sunjincheng...@gmail.com> wrote:
 >
 > Congratulation Brian!
 >
 > Best,
 > Jincheng
 >
 > Kyle Weaver  于2019年11月15日周五
 上午7:19写道：
 >>
 >> Thanks for your contributions and congrats Brian!
 >>
 >> On Thu, Nov 14, 2019 at 3:14 PM Kenneth Knowles <
 k...@apache.org> wrote:
 >>>
 >>> Hi all,
 >>>
 >>> Please join me and the rest of the Beam PMC in
 welcoming a new committer: Brian Hulette
 >>>
 >>> Brian introduced himself to dev@ earlier this year
 and has been contributing since then. His contributions to 
 Beam include
 explorations of integration with Arrow, standardizing coders, 
 portability
 for schemas, and presentations at Beam events.
 >>>
 >>> In consideration of Brian's contributions, the Beam
 PMC trusts him with the responsibilities of a Beam committer 
 [1].
 >>>
 >>> Thank you, Brian, for your contributions and
 looking forward to many more!
 >>>
 >>> Kenn, on behalf of the Apache Beam PMC
 >>>
 >>> [1]

Re: [UPDATE] Preparing for Beam 2.17.0 release

[Kenneth Knowles]

I've poked through the bugs and there do seem to be a few that are finished
and a few that may not be started that should probably be deferred if they
can be triaged to not be blockers.

Kenn

On Fri, Nov 15, 2019 at 2:13 PM Mikhail Gryzykhin  wrote:

> Hi everyone,
>
> There's still an outstanding cherry-pick PR that I can't merge due to
> tests failing on it and release branch validation PR
> . Once I get tests green, I'll
> send another update and review outstanding open issues.
>
> --Mikhail
>
> On Fri, Nov 15, 2019 at 10:40 AM Thomas Weise  wrote:
>
>> Any update regarding the release?
>>
>> The list still shows 10 open issues:
>>
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20fixVersion%20%3D%202.17.0%20and%20resolution%20is%20EMPTY
>>
>> Is the RC blocked on those?
>>
>>
>>
>>
>>
>>
>> On Mon, Oct 28, 2019 at 12:46 PM Ahmet Altay  wrote:
>>
>>>
>>>
>>> On Mon, Oct 28, 2019 at 12:44 PM Gleb Kanterov  wrote:
>>>
 It looks like BigQueryIO DIRECT_READ is broken since 2.16.0, I've added
 a ticket describing the problem and possible fix, see BEAM-8504
  [1].

>>>
>>> Should this be added to 2.16 blog post as a known issue?
>>>
>>>

 [1]: https://issues.apache.org/jira/browse/BEAM-8504

 On Wed, Oct 23, 2019 at 9:19 PM Kenneth Knowles 
 wrote:

> I opened https://github.com/apache/beam/pull/9862 to raise the
> documentation of Fix Version to the top level. It also includes the write
> up of Jira priorities, to make clear that "Blocker" priority does not 
> refer
> to release blocking.
>
> On Wed, Oct 23, 2019 at 11:16 AM Kenneth Knowles 
> wrote:
>
>> I've gone over the tickets and removed Fix Version from many of them
>> that do not seem to be critical defects. If I removed Fix Version from a
>> ticket you care about, please feel free to add it back. I am not trying 
>> to
>> decide what is in/out of the release, just trying to triage the Jira data
>> to match expected practices.
>>
>> It should probably be documented somewhere outside of the release
>> guide. As far as I can tell, the fact that we triage them down to zero is
>> the only place we mention that it is used to indicate release blockers 
>> and
>> not used for feature targets.
>>
>> Kenn
>>
>> On Wed, Oct 23, 2019 at 10:40 AM Kenneth Knowles 
>> wrote:
>>
>>>  Wow, 28 release blocking tickets! That is the most I've ever seen,
>>> by far. Many appear to be feature requests, not release-blocking 
>>> defects. I
>>> believe this is not according to our normal best practice. The release
>>> cadence should not wait for features in progress, with exceptions 
>>> discussed
>>> on dev@. As a matter of best practice, I think we should triage
>>> feature requests to not have Fix Version set until it has been 
>>> discussed on
>>> dev@.
>>>
>>> Kenn
>>>
>>> On Wed, Oct 23, 2019 at 9:55 AM Mikhail Gryzykhin 
>>> wrote:
>>>
 Hi all,

 Beam 2.17 release branch cut is scheduled today (2019/10/23)
 according to the release calendar [1].  I'll start working on the
 branch cutoff and later work on cherry picking blocker fixes.

 If you have release blocking issues for 2.17 please mark their
 "Fix Version" as 2.17.0 [2]. This tag is already created in JIRA in 
 case
 you would like to move any non-blocking issues to that version.

 There is a decent amount of open bugs to be resolved in 2.17.0 [2]
 and only 4 [3] are marked as blockers. Please, review those if these 
 bugs
 are actually to be resolved in 2.17.0 and prioritize fixes if possible.

 Any thoughts, comments, objections?

 Regards.
 Mikhail.


 [1]
 https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
 [2]
 https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.17.0
 [3]
 https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%202.17.0

>>>

Re: GCP libraries up-to-date versions in Java

[Kenneth Knowles]

Hi David,

This requires some thought. Avoiding breaking changes is more than a
preference.

As I understand it, the problem is not dependency incompatibilities in the
Beam Java SDK, but self-incompatibility in Google's libraries across
releases. It makes sense - inside Google's "monorepo" one does not need to
be as careful, so these things are bound to happen.

Nonetheless, in practice, breaking changes == forks. The term "upgrade" and
"downgrade" are misleading in this case. The situation is more like a
milder version of Python 2 vs 3.

So taking "Pubsub 1.28" and "Pubsub 1.43" which are simply different*,
which one(s) should Beam support? Should we choose to support only the
versions of the libraries that Google suggests via its BOM? Can we manage
to cleverly support them with a single PubsubIO like we do w/ Flink and
ElasticSearch? Given Google's track record, there will surely be more
mutually incompatible versions to come. Any popular storage system that
wants to make breaking changes will need a connector at least for the prior
dominant version and the new ascending version, or else harm users.

If we can pin to a version of the BOM with no breaking changes in Beam or
its dependencies, then this is all a non-issue.

If adopting the bom is a breaking change, then it would be a radical new
policy. It makes some sense, since Beam users who care about GCP connectors
are probably willing and interested in adhering to Google's recommendations
even if they have to adjust their code one time. But notably, there are
plenty of Beam users who upgrade their Beam jars without recompilation, so
the breakage is more severe. And presumably any change to the BOM version
would include breaking changes - would we establish a policy of changing
it? Or basically never change it once we do the first breaking change to
pin?

I think the scope of this proposal must be severely limited. It should have
zero impact on users outside of the GCP connectors and Dataflow runner -
any other Beam use of Google's OSS utility libraries is out of scope. Even
so, I am not sure this is best for users overall.

There's a lot to consider and flesh out. I'm not even sure how / if we can
get the data needed to guide these decisions.

Kenn

*I just made up the version numbers

On Tue, Nov 19, 2019 at 5:23 PM David Cavazos  wrote:

> Hi Beamers,
>
> I recently was a part of a discussion about some dependency
> incompatibilities in the Java SDK. Specifically on the GRPC versions when
> trying to use one of the Google Cloud client libraries as part of a Beam
> pipeline. Their workaround was downgrading to an older version of the
> client library to match Beam's version of the GRPC library. However, this
> could not have been possible if they *needed* the newer version for any
> reason.
>
> I'm aware that Java development environments usually prefer to hardcode
> versions to avoid breaking changes, but it would be great to have the
> latest versions of dependencies that could be *shared* with other
> libraries, like the GRPC libraries.
>
> It looks like the Google Cloud client library team has been aware of this
> problem, as well as the tricky interactions between the hundreds of
> libraries they offer. They mentioned that they are starting to roll out a GCP
> Libraries BOM
> 
>  to
> help everyone have up-to-date versions of their libraries, including
> *guava*, *protobuf*, *grpc-java*, *google-http-java-client*, and
> *google-cloud-java*.
>
> Would everyone feel comfortable on using the BOM to manage the Google
> Cloud dependency versions? If so, is there anyone comfortable in Gradle
> willing to do these changes?
>
> Cheers!
> David
>

Re: [VOTE] Beam Mascot animal choice: vote for as many as you want

[Reza Rokni]

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[X] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[X] Dumbo Octopus
[X] Angler fish

On Wed, 20 Nov 2019 at 10:43, Kenneth Knowles  wrote:

> Please cast your votes of approval [1] for animals you would support as
> Beam mascot. The animal with the most approval will be identified as the
> favorite.
>
> *** Vote for as many as you like, using this checklist as a template 
>
> [ ] Beaver
> [ ] Hedgehog
> [ ] Lemur
> [ ] Owl
> [ ] Salmon
> [ ] Trout
> [ ] Robot dinosaur
> [ ] Firefly
> [ ] Cuttlefish
> [ ] Dumbo Octopus
> [ ] Angler fish
>
> This vote will remain open for at least 72 hours.
>
> Kenn
>
> [1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
> https://www.electionscience.org/library/approval-voting/
>


-- 

This email may be confidential and privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person.

The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.

[VOTE] Beam Mascot animal choice: vote for as many as you want

[Kenneth Knowles]

Please cast your votes of approval [1] for animals you would support as
Beam mascot. The animal with the most approval will be identified as the
favorite.

*** Vote for as many as you like, using this checklist as a template 

[ ] Beaver
[ ] Hedgehog
[ ] Lemur
[ ] Owl
[ ] Salmon
[ ] Trout
[ ] Robot dinosaur
[ ] Firefly
[ ] Cuttlefish
[ ] Dumbo Octopus
[ ] Angler fish

This vote will remain open for at least 72 hours.

Kenn

[1] See https://en.wikipedia.org/wiki/Approval_voting#Description and
https://www.electionscience.org/library/approval-voting/

Re: [Discuss] Beam mascot

[Kenneth Knowles]

Well then...

I think this thread has gone on for some time and seems to have settled
down. We probably have enough candidates. Gris summarized the status
earlier but I think it will help to do something slightly more formal. I
think I'll try moving on with what I suggested. I will start a slightly
more formal thread to identify the favorite. This will be more clear than a
mixed discussion with preferences. If people do not like the style of the
thread I will start, we can figure out another way.

Kenn

On Fri, Nov 15, 2019 at 3:06 PM Alex Van Boxel  wrote:

> I like the "Angler fish" as well... it certainly faster then a
> cuttlefish
>
>  _/
> _/ Alex Van Boxel
>
>
> On Fri, Nov 15, 2019 at 9:57 PM Kirill Kozlov 
> wrote:
>
>> Angler fish? Found a few animated examples that may look interesting [1,
>> 2, 3, 4].
>> [1] https://www.pinterest.com/pin/121175046208927340/
>> [2] https://www.pinterest.com/pin/353180795779533157/
>> [3] https://www.pinterest.com/pin/121175046208927334/
>> [4]
>> https://graphicriver.net/item/cartoon-angler-fish/16828573?ref=gvector_id=1413785108_back=true
>>
>> On Fri, Nov 15, 2019 at 11:39 AM Aizhamal Nurmamat kyzy <
>> aizha...@apache.org> wrote:
>>
>>> That last sketch of a cuttlefish with a hat is really really good. I
>>> vote for that.
>>>
>>> On Fri, Nov 15, 2019 at 10:33 AM Kenneth Knowles 
>>> wrote:
>>>
 PSA: cuttlefish tentacles look more like their face, not their legs.
 Please find attached illustrations to that effect. And also evidence that
 they are occasionally fancy. Definitely *not* from a respectable designer
 or anyone who could execution a professional logo.

 I do think the PMC needs to shepherd this. I would suggest starting
 with an approval vote [1] for animal (no plant ideas?) and then an
 ASF-style vote to record the validated result. From there, we can go
 through more standard design process.

 Kenn

 [1] https://www.electionscience.org/library/approval-voting/

 On Fri, Nov 15, 2019 at 6:54 AM Maximilian Michels 
 wrote:

> It's great we're having this discussion and we came up with a lot of
> great ideas through it. However, it is unclear how we proceed from
> here.
> Certainly, we can't let designers work with an open-ended discussion
> on
> the type of mascot.
>
> Personally, I'm fine with _any_ kind of mascot, as long as it is made
> by
> a decent designer. The respectable designer I'm talking to was very
> generous to donate these sketches for free. I don't think that this is
> to be expected at all. Coming back to the original idea of hiring
> multiple designers, I don't see how we will pay those designers to all
> come up with a logo, unless we all donate money.
>
> The sketches I've sent might not look like much because there is still
> a
> decent process involved in coming up with the final logo which looks
> good in different sizes, possibly with many iterations. So far I've
> tried to incorporate as many of the suggestions here, but for the sake
> of protecting the designer, I think I'll have to stop doing that. It
> simply won't work.
>
> How to proceed from here? I think _any_ professionally executed logo
> will do, this is really more about the community agreeing for the
> greater good. Ultimately, I think the PMC might have to decide on how
> to
> proceed here.
>
> Cheers,
> Max
>
> On 15.11.19 13:59, Hannah Jiang wrote:
> > I also vote for firefly.
> >
> > On Wed, Nov 13, 2019 at 1:38 PM Valentyn Tymofieiev <
> valen...@google.com
> > > wrote:
> >
> > I like the firefly sketch a lot, it's my favorite so far.
> >
> > On Wed, Nov 13, 2019 at 12:58 PM Robert Bradshaw
> > mailto:rober...@google.com>> wrote:
> >
> > #37 from the sketches was the cuttlefish, which would put it
> at
> > (with
> > 4 votes) the most popular so far. I do like the firefly too.
> >
> > On Wed, Nov 13, 2019 at 12:03 PM Gris Cuevas <
> g...@apache.org
> > > wrote:
> >  >
> >  > Hi everyone, so exciting to see this convo taking off!
> >  >
> >  > I loved Alex's firefly! -- it can have so many cool
> > variations, and as a stuffed animal is very original.
> >  >
> >  > Other ideas I had are a caterpillar because it looks like
> a
> > data pipeline, lol or the beaver!
> >  >
> >  > Feedback on the current sketches.
> >  > - They resemble a lot either the Octocat or Totoro [1]
> >  > - I'd like to see something that is completely new and
> > original, pancakes from gRPC is an example[2]
> >  > - Something more 

GCP libraries up-to-date versions in Java

[David Cavazos]

Hi Beamers,

I recently was a part of a discussion about some dependency
incompatibilities in the Java SDK. Specifically on the GRPC versions when
trying to use one of the Google Cloud client libraries as part of a Beam
pipeline. Their workaround was downgrading to an older version of the
client library to match Beam's version of the GRPC library. However, this
could not have been possible if they *needed* the newer version for any
reason.

I'm aware that Java development environments usually prefer to hardcode
versions to avoid breaking changes, but it would be great to have the
latest versions of dependencies that could be *shared* with other
libraries, like the GRPC libraries.

It looks like the Google Cloud client library team has been aware of this
problem, as well as the tricky interactions between the hundreds of
libraries they offer. They mentioned that they are starting to roll out a GCP
Libraries BOM

to
help everyone have up-to-date versions of their libraries, including *guava*,
*protobuf*, *grpc-java*, *google-http-java-client*, and *google-cloud-java*.

Would everyone feel comfortable on using the BOM to manage the Google Cloud
dependency versions? If so, is there anyone comfortable in Gradle willing
to do these changes?

Cheers!
David

Beam Dependency Check Report (2019-11-19)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
google-cloud-datastore
1.7.4
1.10.0
2019-05-27
2019-10-21BEAM-8443
mock
2.0.0
3.0.5
2019-05-20
2019-05-20BEAM-7369
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
pytest
4.6.6
5.2.4
2019-11-11
2019-11-18BEAM-8606
Sphinx
1.8.5
2.2.1
2019-05-20
2019-10-28BEAM-7370
tenacity
5.1.5
6.0.0
2019-11-11
2019-11-11BEAM-8607
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.alibaba:fastjson
1.2.49
1.2.62
2018-08-04
2019-10-07BEAM-8632
com.datastax.cassandra:cassandra-driver-core
3.6.0
4.0.0
2018-08-29
2019-03-18BEAM-8674
com.datastax.cassandra:cassandra-driver-mapping
3.6.0
3.8.0
2018-08-29
2019-10-29BEAM-8749
com.esotericsoftware:kryo
4.0.2
5.0.0-RC4
2018-03-20
2019-04-14BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.20.0
0.27.0
2019-02-11
2019-10-21BEAM-6645
com.github.spotbugs:spotbugs
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-7792
com.github.spotbugs:spotbugs-annotations
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-6951
com.google.api:gax-grpc
1.38.0
1.50.1
2019-02-05
2019-11-07BEAM-8676
com.google.api.grpc:grpc-google-cloud-pubsub-v1
1.43.0
1.83.0
2019-01-23
2019-11-14BEAM-8677
com.google.api.grpc:grpc-google-common-protos
1.12.0
1.17.0
2018-06-29
2019-10-04BEAM-8633
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1
0.44.0
0.84.0
2019-01-23
2019-11-14BEAM-8678
com.google.api.grpc:proto-google-cloud-bigtable-v2
0.44.0
1.7.1
2019-01-23
2019-11-13BEAM-8679
com.google.api.grpc:proto-google-cloud-datastore-v1
0.44.0
0.84.0
2019-01-23
2019-11-14BEAM-8680
com.google.api.grpc:proto-google-cloud-pubsub-v1
1.43.0
1.83.0
2019-01-23
2019-11-14BEAM-8681
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1
1.6.0
1.46.0
2019-01-23
2019-11-14BEAM-8682
com.google.api.grpc:proto-google-common-protos
1.12.0
1.17.0
2018-06-29
2019-10-04BEAM-6899
com.google.apis:google-api-services-bigquery
v2-rev20181221-1.28.0
v2-rev20190917-1.30.3
2019-01-17
2019-10-09BEAM-8684
com.google.apis:google-api-services-clouddebugger
v2-rev20181114-1.28.0
v2-rev20191003-1.30.3
2019-01-17
2019-10-19BEAM-8750
com.google.apis:google-api-services-cloudresourcemanager
v1-rev20181015-1.28.0
v2-rev20191018-1.30.3
2019-01-17
2019-10-30BEAM-8751
com.google.apis:google-api-services-dataflow
v1b3-rev20190927-1.28.0
v1beta3-rev12-1.20.0
2019-10-11
2015-04-29BEAM-8752
com.google.apis:google-api-services-pubsub
v1-rev20181213-1.28.0
v1-rev20191001-1.30.3
2019-01-18
2019-10-19BEAM-8753
com.google.apis:google-api-services-storage
v1-rev20181109-1.28.0
v1-rev20191011-1.30.3
2019-01-18
2019-10-30BEAM-8754
com.google.auth:google-auth-library-credentials
0.13.0
0.18.0
2019-01-17
2019-10-09BEAM-6478
com.google.auth:google-auth-library-oauth2-http
0.12.0
0.18.0
2018-11-14
2019-10-09BEAM-8685
com.google.cloud:google-cloud-bigquery
1.28.0
1.101.0
2018-04-27
2019-11-14BEAM-8687
com.google.cloud:google-cloud-bigquerystorage
0.79.0-alpha
0.119.0-beta
2019-01-23
2019-11-14BEAM-8755
com.google.cloud:google-cloud-core
1.61.0
1.91.3
2019-01-23
2019-10-23BEAM-8756
com.google.cloud:google-cloud-core-grpc
1.61.0
1.91.3
2019-01-23
2019-10-23BEAM-8757

Beam Dependency Check Report (2019-11-19)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
mock
2.0.0
3.0.5
2019-05-20
2019-05-20BEAM-7369
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
Sphinx
1.8.5
2.2.1
2019-05-20
2019-10-28BEAM-7370
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.esotericsoftware:kryo
4.0.2
5.0.0-RC4
2018-03-20
2019-04-14BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.20.0
0.27.0
2019-02-11
2019-10-21BEAM-6645
com.github.spotbugs:spotbugs
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-7792
com.github.spotbugs:spotbugs-annotations
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-6951
com.google.api.grpc:proto-google-common-protos
1.12.0
1.17.0
2018-06-29
2019-10-04BEAM-6899
com.google.auth:google-auth-library-credentials
0.13.0
0.18.0
2019-01-17
2019-10-09BEAM-6478
com.google.code.gson:gson
2.7
2.8.6
2016-06-14
2019-10-04BEAM-5558
io.grpc:grpc-auth
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5896
io.grpc:grpc-context
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5897
io.grpc:grpc-core
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5898
io.grpc:grpc-netty
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5899
io.grpc:grpc-protobuf
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5900
io.grpc:grpc-stub
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5901
io.grpc:grpc-testing
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5902
io.grpc:protoc-gen-grpc-java
1.21.0
1.25.0
2019-05-22
2019-11-05BEAM-5903
io.opencensus:opencensus-api
0.21.0
0.24.0
2019-05-01
2019-08-27BEAM-5580
io.opencensus:opencensus-contrib-grpc-metrics
0.21.0
0.24.0
2019-05-01
2019-08-27BEAM-5581
javax.servlet:javax.servlet-api
3.1.0
4.0.1
2013-04-25
2018-04-20BEAM-5750
net.java.dev.javacc:javacc
4.0
7.0.5
2006-03-17
2019-10-20BEAM-5570
net.java.dev.jna:jna
4.1.0
5.5.0
2014-03-06
2019-10-30BEAM-5573
org.apache.hbase:hbase-common
1.2.6
2.2.2
2017-05-29
2019-10-20BEAM-5560
org.apache.hbase:hbase-hadoop-compat
1.2.6
2.2.2
2017-05-29
2019-10-20BEAM-5561
org.apache.hbase:hbase-hadoop2-compat
1.2.6
2.2.2
2017-05-29
2019-10-20BEAM-5562
org.apache.hbase:hbase-server
1.2.6
2.2.2
2017-05-29
2019-10-20BEAM-5563
org.apache.hbase:hbase-shaded-client
1.2.6
2.2.1
2017-05-29
2019-09-04BEAM-5564
org.apache.hive:hive-cli
2.1.0
3.1.2
2016-06-17
2019-08-22BEAM-5566
org.apache.hive:hive-common
2.1.0
3.1.2
2016-06-17
2019-08-22BEAM-5567
org.apache.hive:hive-exec
2.1.0
3.1.2
2016-06-17
2019-08-22BEAM-5568
org.apache.hive.hcatalog:hive-hcatalog-core
2.1.0
3.1.2
2016-06-17
2019-08-22BEAM-5569
org.apache.qpid:proton-j
0.13.1
0.33.2
2016-07-02
2019-08-09BEAM-5582
org.apache.solr:solr-core
5.5.4
8.3.0
2017-02-14
2019-10-31BEAM-5589
org.apache.solr:solr-solrj
5.5.4
8.3.0
2017-02-14
2019-10-31BEAM-5590
org.apache.solr:solr-test-framework
5.5.4
8.3.0
2017-02-14
2019-10-31BEAM-5591
org.conscrypt:conscrypt-openjdk
1.1.3
2.2.1
2018-06-04
2019-08-08BEAM-5748
org.eclipse.jetty:jetty-server
9.2.10.v20150310
10.0.0-alpha0
2015-03-10
2019-07-11BEAM-5752
org.eclipse.jetty:jetty-servlet
9.2.10.v20150310
10.0.0-alpha0

Re: Improve container support

[Robert Bradshaw]

Good to know. I added an icon and link. Good pointer about trying to
see what we can do to make these official.

On Tue, Nov 19, 2019 at 11:16 AM Kyle Weaver  wrote:
>
> We do link from https://beam.apache.org/documentation/runtime/environments/.
>
> On Tue, Nov 19, 2019 at 10:59 AM Robert Bradshaw  wrote:
>>
>> We should probably add a link to these from our site as well, for visibility.
>>
>> On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver  wrote:
>> >
>> > +1 Thanks for bringing that up Chad, I had the same problem locating the 
>> > docker images on Docker hub (searching "apachebeam" is the only way that 
>> > seems to work).
>> >
>> > Another little thing I noticed is that https://hub.docker.com/u/apachebeam 
>> > is mostly empty. It'd be good to at least add the Beam logo as a profile 
>> > picture and a link to the website in the description.
>> >
>> > On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova  wrote:
>> >>
>> >> Hi all,
>> >> I think it might be good to update the description of the beam docker 
>> >> images and add some descriptive tags, because searching for "apache beam" 
>> >> in docker hub does not turn up anything: 
>> >> https://hub.docker.com/search?q=apache%20beam=image.
>> >>
>> >> I clicked through 10 pages worth and couldn't find it.  Maybe I missed 
>> >> something, but it clearly shouldn't be this hard.  I did eventually 
>> >> manage to find it through the docs.
>> >>
>> >> Also, googling "apache beam python docker" also does not yield anything 
>> >> useful.  In fact, it turns up an unofficial apache beam docker hub image.
>> >>
>> >> One thing I noticed is that images designated as "Official Images" get 
>> >> listed first, so it would be good to get that done as well.
>> >>
>> >> thanks!
>> >> chad
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang  
>> >> wrote:
>> >>>
>> >>> Hi team
>> >>>
>> >>> I haven't received any objections, so will proceed with settings 
>> >>> mentioned in a previous email.
>> >>>
>> >>> A reminder to PMC members, please let me know your docker hub id if you 
>> >>> want to be an admin.
>> >>>
>> >>> Thanks,
>> >>> Hannah
>> >>>
>> >>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:
>> 
>>  Please ignore the previous email. I was looking at the older document 
>>  in the mail thread.
>> 
>>  On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>> >
>> > I think sdk in the name is obsolete as they are all under sdks name 
>> > space.
>> >
>> > On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang  
>> > wrote:
>> >>
>> >> Hi Team
>> >>
>> >> Thanks for all the comments about beam containers.
>> >> After considering various opinions and investigating gcr and docker 
>> >> hub, we decided to push images to docker hub.
>> >>
>> >> Each image will have two tags, {version}_rc and {version}. {version} 
>> >> tag will be added after the release candidate image is verified.
>> >> Meanwhile, we will have latest tag for each repository, which always 
>> >> points to the most recent verified release image, so users can pull 
>> >> it by default.
>> >>
>> >> Docker hub doesn't support leveled repository, which means we should 
>> >> follow repository:tag format.
>> >> it's too general if we use {language_version} as repository for SDK 
>> >> images. (version is added when we support multiple versions.)
>> >> So I would like to include sdk to repository. Images generated at 
>> >> local will also have the same name.
>> >> Here are some examples:
>> >>
>> >> python2.7_sdk:2.15.0
>> >> java_sdk:2.15.0_rc
>> >> go_sdk:latest
>> >>
>> >> I will proceed with this format if there is no strong opposition by 
>> >> tomorrow noon(PST).
>> >>
>> >> To PMC members:
>> >> Permission control will follow the pypi model. All interested PMC 
>> >> members will be added as admins and release managers will be granted 
>> >> push permission.
>> >> Please let me know your docker id if you want to be added as an admin.
>> >>
>> >> Thanks,
>> >> Hannah
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>> >>>
>> >>> This will greatly simplify trying out portable runners: 
>> >>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>> >>>
>> >>> Can't wait for following to disappear from the instructions page: 
>> >>> ./gradlew :sdks:python:container:docker
>> >>>
>> >>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>> 
>>  Awesome, thank you!
>> 
>> 
>>  On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>>   wrote:
>> >
>> > Hi Thomas
>> >
>> > I created snapshot images from head as of around 2PM today.
>> > You can pull images from 

Re: [discuss] Using a logger hierarchy in Python

[Chad Dombrova]

Pablo, it might be necessary to setup a root logging handler if one does
not exist already.  I noticed that a LocalJobServicer that I was testing
against stopped emitting tracebacks when I rebased onto the latest from
master.  Setting up the root handler fixed it.  I'm still testing this, and
I might be misinterpreting what I saw, but I wanted to get eyes on it in
case I don't have time to get a definitive answer.

-chad



On Fri, Nov 15, 2019 at 4:30 PM Pablo Estrada  wrote:

> Thanks all,
> 2/3 of PRs are merged (using _LOGGER). It should be pretty easy to
> switch the variable name to _log via sed.
> Best
> -P.
>
> On Fri, Nov 15, 2019 at 2:08 PM Kyle Weaver  wrote:
>
>> +1 for per-module loggers (what Robert said).
>>
>> On Fri, Nov 15, 2019 at 1:48 PM Udi Meiri  wrote:
>>
>>> +1, but can we use something less verbose and shift key heavy than
>>> _LOGGER like log or _log?
>>>
>>> Also please dedupe with these existing bugs:
>>> https://issues.apache.org/jira/browse/BEAM-3523
>>> https://issues.apache.org/jira/browse/BEAM-1825
>>>
>>> On Thu, Nov 14, 2019 at 8:02 AM Thomas Weise  wrote:
>>>
 Awesome, thanks Chad!

 On Wed, Nov 13, 2019 at 10:26 PM Chad Dombrova 
 wrote:

> Hi Thomas,
>
>
>> Will this include the ability for users to configure logging via
>> pipeline options?
>>
>
> We're working on a proposal to allow pluggable logging handlers that
> can be configured via pipeline options.  For example, it would allow you 
> to
> add a new logging handler for StackDriver or Elasticsearch.  Will 
> hopefully
> have a document to share soon.
>
> -chad
>
>

Re: Improve container support

[Kyle Weaver]

We do link from https://beam.apache.org/documentation/runtime/environments/.

On Tue, Nov 19, 2019 at 10:59 AM Robert Bradshaw 
wrote:

> We should probably add a link to these from our site as well, for
> visibility.
>
> On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver  wrote:
> >
> > +1 Thanks for bringing that up Chad, I had the same problem locating the
> docker images on Docker hub (searching "apachebeam" is the only way that
> seems to work).
> >
> > Another little thing I noticed is that
> https://hub.docker.com/u/apachebeam is mostly empty. It'd be good to at
> least add the Beam logo as a profile picture and a link to the website in
> the description.
> >
> > On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova 
> wrote:
> >>
> >> Hi all,
> >> I think it might be good to update the description of the beam docker
> images and add some descriptive tags, because searching for "apache beam"
> in docker hub does not turn up anything:
> https://hub.docker.com/search?q=apache%20beam=image.
> >>
> >> I clicked through 10 pages worth and couldn't find it.  Maybe I missed
> something, but it clearly shouldn't be this hard.  I did eventually manage
> to find it through the docs.
> >>
> >> Also, googling "apache beam python docker" also does not yield anything
> useful.  In fact, it turns up an unofficial apache beam docker hub image.
> >>
> >> One thing I noticed is that images designated as "Official Images" get
> listed first, so it would be good to get that done as well.
> >>
> >> thanks!
> >> chad
> >>
> >>
> >>
> >>
> >> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang 
> wrote:
> >>>
> >>> Hi team
> >>>
> >>> I haven't received any objections, so will proceed with settings
> mentioned in a previous email.
> >>>
> >>> A reminder to PMC members, please let me know your docker hub id if
> you want to be an admin.
> >>>
> >>> Thanks,
> >>> Hannah
> >>>
> >>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:
> 
>  Please ignore the previous email. I was looking at the older document
> in the mail thread.
> 
>  On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka 
> wrote:
> >
> > I think sdk in the name is obsolete as they are all under sdks name
> space.
> >
> > On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
> wrote:
> >>
> >> Hi Team
> >>
> >> Thanks for all the comments about beam containers.
> >> After considering various opinions and investigating gcr and docker
> hub, we decided to push images to docker hub.
> >>
> >> Each image will have two tags, {version}_rc and {version}.
> {version} tag will be added after the release candidate image is verified.
> >> Meanwhile, we will have latest tag for each repository, which
> always points to the most recent verified release image, so users can pull
> it by default.
> >>
> >> Docker hub doesn't support leveled repository, which means we
> should follow repository:tag format.
> >> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> >> So I would like to include sdk to repository. Images generated at
> local will also have the same name.
> >> Here are some examples:
> >>
> >> python2.7_sdk:2.15.0
> >> java_sdk:2.15.0_rc
> >> go_sdk:latest
> >>
> >> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
> >>
> >> To PMC members:
> >> Permission control will follow the pypi model. All interested PMC
> members will be added as admins and release managers will be granted push
> permission.
> >> Please let me know your docker id if you want to be added as an
> admin.
> >>
> >> Thanks,
> >> Hannah
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
> >>>
> >>> This will greatly simplify trying out portable runners:
> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
> >>>
> >>> Can't wait for following to disappear from the instructions page:
> ./gradlew :sdks:python:container:docker
> >>>
> >>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise 
> wrote:
> 
>  Awesome, thank you!
> 
> 
>  On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang <
> hannahji...@google.com> wrote:
> >
> > Hi Thomas
> >
> > I created snapshot images from head as of around 2PM today.
> > You can pull images from
> gcr.io/apache-beam-testing/beam/sdks/snapshot.
> >
> > Thanks,
> > Hannah
> >
> > On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise 
> wrote:
> >>
> >> Hi Hannah,
> >>
> >> Thank you, I know how to build the containers locally, but not
> how to publish them!
> >>
> >> The cwiki says "Publishing images to gcr.io/beam requires
> permissions in apache-beam-testing 

Re: Improve container support

[Robert Bradshaw]

We should probably add a link to these from our site as well, for visibility.

On Tue, Nov 19, 2019 at 10:56 AM Kyle Weaver  wrote:
>
> +1 Thanks for bringing that up Chad, I had the same problem locating the 
> docker images on Docker hub (searching "apachebeam" is the only way that 
> seems to work).
>
> Another little thing I noticed is that https://hub.docker.com/u/apachebeam is 
> mostly empty. It'd be good to at least add the Beam logo as a profile picture 
> and a link to the website in the description.
>
> On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova  wrote:
>>
>> Hi all,
>> I think it might be good to update the description of the beam docker images 
>> and add some descriptive tags, because searching for "apache beam" in docker 
>> hub does not turn up anything: 
>> https://hub.docker.com/search?q=apache%20beam=image.
>>
>> I clicked through 10 pages worth and couldn't find it.  Maybe I missed 
>> something, but it clearly shouldn't be this hard.  I did eventually manage 
>> to find it through the docs.
>>
>> Also, googling "apache beam python docker" also does not yield anything 
>> useful.  In fact, it turns up an unofficial apache beam docker hub image.
>>
>> One thing I noticed is that images designated as "Official Images" get 
>> listed first, so it would be good to get that done as well.
>>
>> thanks!
>> chad
>>
>>
>>
>>
>> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang  wrote:
>>>
>>> Hi team
>>>
>>> I haven't received any objections, so will proceed with settings mentioned 
>>> in a previous email.
>>>
>>> A reminder to PMC members, please let me know your docker hub id if you 
>>> want to be an admin.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:

 Please ignore the previous email. I was looking at the older document in 
 the mail thread.

 On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>
> I think sdk in the name is obsolete as they are all under sdks name space.
>
> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang  
> wrote:
>>
>> Hi Team
>>
>> Thanks for all the comments about beam containers.
>> After considering various opinions and investigating gcr and docker hub, 
>> we decided to push images to docker hub.
>>
>> Each image will have two tags, {version}_rc and {version}. {version} tag 
>> will be added after the release candidate image is verified.
>> Meanwhile, we will have latest tag for each repository, which always 
>> points to the most recent verified release image, so users can pull it 
>> by default.
>>
>> Docker hub doesn't support leveled repository, which means we should 
>> follow repository:tag format.
>> it's too general if we use {language_version} as repository for SDK 
>> images. (version is added when we support multiple versions.)
>> So I would like to include sdk to repository. Images generated at local 
>> will also have the same name.
>> Here are some examples:
>>
>> python2.7_sdk:2.15.0
>> java_sdk:2.15.0_rc
>> go_sdk:latest
>>
>> I will proceed with this format if there is no strong opposition by 
>> tomorrow noon(PST).
>>
>> To PMC members:
>> Permission control will follow the pypi model. All interested PMC 
>> members will be added as admins and release managers will be granted 
>> push permission.
>> Please let me know your docker id if you want to be added as an admin.
>>
>> Thanks,
>> Hannah
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>>>
>>> This will greatly simplify trying out portable runners: 
>>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>>
>>> Can't wait for following to disappear from the instructions page: 
>>> ./gradlew :sdks:python:container:docker
>>>
>>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:

 Awesome, thank you!


 On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang  
 wrote:
>
> Hi Thomas
>
> I created snapshot images from head as of around 2PM today.
> You can pull images from 
> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>
> Thanks,
> Hannah
>
> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>>
>> Hi Hannah,
>>
>> Thank you, I know how to build the containers locally, but not how 
>> to publish them!
>>
>> The cwiki says "Publishing images to gcr.io/beam requires 
>> permissions in apache-beam-testing project."
>>
>> Can I get access to the testing project (at least temporarily) and 
>> what would I need to setup to run the publish target that is shown 
>> on cwiki?
>>
>> Thanks,
>> Thomas

Re: Improve container support

[Kyle Weaver]

+1 Thanks for bringing that up Chad, I had the same problem locating the
docker images on Docker hub (searching "apachebeam" is the only way that
seems to work).

Another little thing I noticed is that https://hub.docker.com/u/apachebeam is
mostly empty. It'd be good to at least add the Beam logo as a profile
picture and a link to the website in the description.

On Tue, Nov 19, 2019 at 10:21 AM Chad Dombrova  wrote:

> Hi all,
> I think it might be good to update the description of the beam docker
> images and add some descriptive tags, because searching for "apache beam"
> in docker hub does not turn up anything:
> https://hub.docker.com/search?q=apache%20beam=image.
>
> I clicked through 10 pages worth and couldn't find it.  Maybe I missed
> something, but it clearly shouldn't be this hard.  I did eventually manage
> to find it through the docs.
>
> Also, googling "apache beam python docker" also does not yield anything
> useful.  In fact, it turns up an *unofficial* apache beam docker hub
> image.
>
> One thing I noticed is that images designated as "Official Images" get
> listed first, so it would be good to get that done as well.
>
> thanks!
> chad
>
>
>
>
> On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang 
> wrote:
>
>> Hi team
>>
>> I haven't received any objections, so will proceed with settings
>> mentioned in a previous email.
>>
>> A reminder to PMC members, please let me know your docker hub id if you
>> want to be an admin.
>>
>> Thanks,
>> Hannah
>>
>> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:
>>
>>> Please ignore the previous email. I was looking at the older document in
>>> the mail thread.
>>>
>>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>>>
 I think sdk in the name is obsolete as they are all under sdks name
 space.

 On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
 wrote:

> Hi Team
>
> Thanks for all the comments about beam containers.
> After considering various opinions and investigating gcr and docker
> hub, we decided to push images to docker hub.
>
> Each image will have two tags, {version}_rc and {version}. {version}
> tag will be added after the release candidate image is verified.
> Meanwhile, we will have* latest* tag for each repository, which
> always points to the most recent verified release image, so users can pull
> it by default.
>
> Docker hub doesn't support leveled repository, which means we should
> follow *repository:tag* format.
> it's too general if we use {language_version} as repository for SDK
> images. (version is added when we support multiple versions.)
> So I would like to include *sdk* to repository. Images generated at
> local will also have the same name.
> Here are some examples:
>
>- python2.7_sdk:2.15.0
>- java_sdk:2.15.0_rc
>- go_sdk:latest
>
> I will proceed with this format if there is no strong opposition by
> tomorrow noon(PST).
>
> *To PMC members*:
> Permission control will follow the pypi model. All interested PMC
> members will be added as admins and release managers will be granted push
> permission.
> Please let me know your *docker id* if you want to be added as an
> admin.
>
> Thanks,
> Hannah
>
>
>
>
>
>
>
>
> On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:
>
>> This will greatly simplify trying out portable runners:
>> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>>
>> Can't wait for following to disappear from the instructions page: 
>> ./gradlew
>> :sdks:python:container:docker
>>
>> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>>
>>> Awesome, thank you!
>>>
>>>
>>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>>> wrote:
>>>
 Hi Thomas

 I created snapshot images from head as of around 2PM today.
 You can pull images from
 gcr.io/apache-beam-testing/beam/sdks/snapshot.

 Thanks,
 Hannah

 On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:

> Hi Hannah,
>
> Thank you, I know how to build the containers locally, but not how
> to publish them!
>
> The cwiki says "Publishing images to gcr.io/beam requires
> permissions in apache-beam-testing project."
>
> Can I get access to the testing project (at least temporarily) and
> what would I need to setup to run the publish target that is shown on 
> cwiki?
>
> Thanks,
> Thomas
>
>
> On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <
> hannahji...@google.com> wrote:
>
>> Hi Thomas
>>
>> I haven't uploaded any snapshot images yet. Here is how you can
>> create one from head.

Re: Improve container support

[Chad Dombrova]

Hi all,
I think it might be good to update the description of the beam docker
images and add some descriptive tags, because searching for "apache beam"
in docker hub does not turn up anything:
https://hub.docker.com/search?q=apache%20beam=image.

I clicked through 10 pages worth and couldn't find it.  Maybe I missed
something, but it clearly shouldn't be this hard.  I did eventually manage
to find it through the docs.

Also, googling "apache beam python docker" also does not yield anything
useful.  In fact, it turns up an *unofficial* apache beam docker hub image.

One thing I noticed is that images designated as "Official Images" get
listed first, so it would be good to get that done as well.

thanks!
chad




On Fri, Sep 6, 2019 at 1:21 PM Hannah Jiang  wrote:

> Hi team
>
> I haven't received any objections, so will proceed with settings mentioned
> in a previous email.
>
> A reminder to PMC members, please let me know your docker hub id if you
> want to be an admin.
>
> Thanks,
> Hannah
>
> On Thu, Sep 5, 2019 at 5:02 PM Ankur Goenka  wrote:
>
>> Please ignore the previous email. I was looking at the older document in
>> the mail thread.
>>
>> On Thu, Sep 5, 2019 at 4:58 PM Ankur Goenka  wrote:
>>
>>> I think sdk in the name is obsolete as they are all under sdks name
>>> space.
>>>
>>> On Thu, Sep 5, 2019 at 3:26 PM Hannah Jiang 
>>> wrote:
>>>
 Hi Team

 Thanks for all the comments about beam containers.
 After considering various opinions and investigating gcr and docker
 hub, we decided to push images to docker hub.

 Each image will have two tags, {version}_rc and {version}. {version}
 tag will be added after the release candidate image is verified.
 Meanwhile, we will have* latest* tag for each repository, which always
 points to the most recent verified release image, so users can pull it by
 default.

 Docker hub doesn't support leveled repository, which means we should
 follow *repository:tag* format.
 it's too general if we use {language_version} as repository for SDK
 images. (version is added when we support multiple versions.)
 So I would like to include *sdk* to repository. Images generated at
 local will also have the same name.
 Here are some examples:

- python2.7_sdk:2.15.0
- java_sdk:2.15.0_rc
- go_sdk:latest

 I will proceed with this format if there is no strong opposition by
 tomorrow noon(PST).

 *To PMC members*:
 Permission control will follow the pypi model. All interested PMC
 members will be added as admins and release managers will be granted push
 permission.
 Please let me know your *docker id* if you want to be added as an
 admin.

 Thanks,
 Hannah








 On Wed, Sep 4, 2019 at 3:47 PM Thomas Weise  wrote:

> This will greatly simplify trying out portable runners:
> https://beam.apache.org/documentation/runners/flink/#executing-a-beam-pipeline-on-a-flink-cluster
>
> Can't wait for following to disappear from the instructions page: 
> ./gradlew
> :sdks:python:container:docker
>
> On Wed, Sep 4, 2019 at 3:35 PM Thomas Weise  wrote:
>
>> Awesome, thank you!
>>
>>
>> On Wed, Sep 4, 2019 at 3:22 PM Hannah Jiang 
>> wrote:
>>
>>> Hi Thomas
>>>
>>> I created snapshot images from head as of around 2PM today.
>>> You can pull images from
>>> gcr.io/apache-beam-testing/beam/sdks/snapshot.
>>>
>>> Thanks,
>>> Hannah
>>>
>>> On Wed, Sep 4, 2019 at 1:41 PM Thomas Weise  wrote:
>>>
 Hi Hannah,

 Thank you, I know how to build the containers locally, but not how
 to publish them!

 The cwiki says "Publishing images to gcr.io/beam requires
 permissions in apache-beam-testing project."

 Can I get access to the testing project (at least temporarily) and
 what would I need to setup to run the publish target that is shown on 
 cwiki?

 Thanks,
 Thomas


 On Wed, Sep 4, 2019 at 11:06 AM Hannah Jiang <
 hannahji...@google.com> wrote:

> Hi Thomas
>
> I haven't uploaded any snapshot images yet. Here is how you can
> create one from head.
> > cd [...]/beam/
> # For Python
> > ./gradlew :sdks:python:container:py{version}:docker *where
> version is {2,35,36,37}*
> # For Java
> > ./gradlew -p sdks/java/container docker
> # For Go
> > ./gradlew -p sdks/go/container docker
>
> The 2.15 one is just for testing, not a real 2.15.0, nor a
> snapshot from head.
>
> Please let me know if you have any questions.
> Hannah
>
> On Wed, Sep 4, 2019 at 10:57 AM Thomas Weise 
> wrote:
>

Re: Beam Dependency Check Report (2019-11-19)

[Tomo Suzuki]

It’s empty because I’m verifying latest changes. Give me some more time.

On Tue, Nov 19, 2019 at 12:41 Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> ERROR: File
> 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not
> exist

-- 
Regards,
Tomo

Beam Dependency Check Report (2019-11-19)

[Apache Jenkins Server]

ERROR: File 'src/build/dependencyUpdates/beam-dependency-check-report.html' does not exist

Re: Library to Parse Thrift Files for ThriftIO

[Ryan Skraba]

For info: https://github.com/airlift/drift has forked and maintained
the code over the last few years.

On Fri, Nov 15, 2019 at 7:23 PM Reuven Lax  wrote:
>
> At a quick glance, the license is Apache which is fine (though we'd have to 
> check dependencies as well). I do notice that git repro is no longer 
> maintained; is there a different one that is better supported?
>
> Reuven
>
> On Wed, Nov 13, 2019 at 7:04 AM Christopher Larsen 
>  wrote:
>>
>> Hey everyone,
>>
>> In regards to the library that will be used to parse the .thrift files for 
>> ThriftIO we are thinking about using some of the code from the library found 
>> here and updating it to use Beam friendly packages. We would love to get the 
>> community's feedback on this approach.
>>
>> Best,
>> Chris
>>
>> This message contains information that may be privileged or confidential and 
>> is the property of the Quantiphi Inc and/or its affiliates. It is intended 
>> only for the person to whom it is addressed. If you are not the intended 
>> recipient, any review, dissemination, distribution, copying, storage or 
>> other use of all or any portion of this message is strictly prohibited. If 
>> you received this message in error, please immediately notify the sender by 
>> reply e-mail and delete this message in its entirety