[Input needed] Capability Matrix Visual Redesign for extended version

[Griselda Cuevas]

Hi folks, another page that's getting a refresh this time around is the
Capability Matrix, which is one of the most critical pages for users as
they evaluate the current support for each of the Beam runners.

The situation we'd like to get your input on is: How do we optimize the
expanded version of the capability matrix, which explains the level of
support in each of the functions?

Right now the text gets in the way of analyzing the table and makes reading
hard. You can see a screenshot in the Beam wiki here [1], the file is
titled current_CapMatExt.

One of the proposed solutions is that after clicking the link "(click to
expand details)", we load a new page that has the corresponding table to
the click (what, where, when, how) at the top, and all the content of each
runner/function gets displayed at the bottom of the page, the file with the
proposed design is also in the Beam wiki here [1] and the file's name is
proposed_CapMatExt. This solution isn't perfect either, since we'd need to
move too much text under the table and reading isn't much easier.

Do you have suggestions/ideas in how to condense the extended version?

Share with us your feedback through this week,
Thanks!
G


[1] https://cwiki.apache.org/confluence/display/BEAM/Website+Redesign+Files

[Input Needed] Updating the documentation left navigation (was:Website Revamp update & navigation for documentation page)

[Griselda Cuevas]

Hi dev@ community,


I wanted to fork this conversation to make it more visible. We'd like to
get your feedback on the proposed update for the documentation page in the
website.


The proposed change is that the left navigation (which is the website main
navigation) stays high level and only shows titles, leaving the detailed
hierarchy in the right navigation, which appears when you click on a
Documentation topic or when you land directly on it.


You can see a screenshot of the current and proposed designs in the wiki
repo with the other website designs [1]. The names of the files are:
current_DocsNav for the current design, and proposed_DocsNav for the
proposed design.


Please let us know your preference or suggestions through this week.


Gris


[1] https://cwiki.apache.org/confluence/display/BEAM/Website+Redesign+Files

Re: Website Revamp update & navigation for documentation page

[Griselda Cuevas]

Ack, thanks Ahmet for the suggestion.

I think it's important to bring attention to this discussion separately, so
I am going to for the thread to start one specific about the navigation
design.


On Mon, 14 Dec 2020 at 17:49, Ahmet Altay  wrote:

>
>
> On Mon, Dec 14, 2020 at 4:32 PM Griselda Cuevas  wrote:
>
>> Hi Ahmet, adding response inline.
>>
>> A question about the content. Design adds items for use cases, do we have
>>> existing content that fits into that template?
>>>
>> Yes, the content will be curated from the use cases we got presented at
>> Beam Summits. We are tracking who has given permission to do this in Jira:
>> BEAM-11225  [1]
>>
>
> Nice!
>
>
>>
>> Re: left navigation on the docs page. The fonts here are more readable to
>>> me and the search feature is great. One question, the current version has
>>> hierarchical and collapsible items on the left navigation. Would that still
>>> be possible. IMO visual grouping, and hiding detailed items makes the
>>> documentation easier to navigate.
>>>
>>
>> This is the point we would like to hear feedback on, the proposal is to
>> remove the full detail in the left navigation and leave the hierarchy view
>> on the right navigation, which appears on the right when you click on a
>> topic. Doing this will allow us to keep the left navigation organized and
>> focused on the high level topics, hoping to make the experience smoother
>> and neater for people and do not overwhelm them with so many
>> topics/subtopics.
>>
>
> Got it. Having a simplified left nav, and a more detailed right nav make
> sense to me.
>
> My _very personal opinion_ is that the left nav is useful, and the right
> nav is usually cluttered and has duplicate information. For example in this
> page (
> https://beam.apache.org/documentation/transforms/python/overview/#element-wise),
> left nav has richer and collapsible information. Right nav has partial
> information. Somewhat of an opposite example in this page (
> https://beam.apache.org/documentation/programming-guide/#triggers). I
> wonder if we can have a single nav menu in a more elegant way?
>
>
>>
>> For context, experienced UX and UI designers are proposing these changes
>> and have optimized for a better discovery and troubleshooting experience,
>> hence the recommendation.
>>
>> [1]
>> https://issues.apache.org/jira/browse/BEAM-11225?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22permission%20use%20cases%22
>>
>>>

Re: Website Revamp update & navigation for documentation page

[Ahmet Altay]

On Mon, Dec 14, 2020 at 4:32 PM Griselda Cuevas  wrote:

> Hi Ahmet, adding response inline.
>
> A question about the content. Design adds items for use cases, do we have
>> existing content that fits into that template?
>>
> Yes, the content will be curated from the use cases we got presented at
> Beam Summits. We are tracking who has given permission to do this in Jira:
> BEAM-11225  [1]
>

Nice!


>
> Re: left navigation on the docs page. The fonts here are more readable to
>> me and the search feature is great. One question, the current version has
>> hierarchical and collapsible items on the left navigation. Would that still
>> be possible. IMO visual grouping, and hiding detailed items makes the
>> documentation easier to navigate.
>>
>
> This is the point we would like to hear feedback on, the proposal is to
> remove the full detail in the left navigation and leave the hierarchy view
> on the right navigation, which appears on the right when you click on a
> topic. Doing this will allow us to keep the left navigation organized and
> focused on the high level topics, hoping to make the experience smoother
> and neater for people and do not overwhelm them with so many
> topics/subtopics.
>

Got it. Having a simplified left nav, and a more detailed right nav make
sense to me.

My _very personal opinion_ is that the left nav is useful, and the right
nav is usually cluttered and has duplicate information. For example in this
page (
https://beam.apache.org/documentation/transforms/python/overview/#element-wise),
left nav has richer and collapsible information. Right nav has partial
information. Somewhat of an opposite example in this page (
https://beam.apache.org/documentation/programming-guide/#triggers). I
wonder if we can have a single nav menu in a more elegant way?


>
> For context, experienced UX and UI designers are proposing these changes
> and have optimized for a better discovery and troubleshooting experience,
> hence the recommendation.
>
> [1]
> https://issues.apache.org/jira/browse/BEAM-11225?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22permission%20use%20cases%22
>
>>

Re: Website Revamp update & navigation for documentation page

[Griselda Cuevas]

Hi Ahmet, adding response inline.

A question about the content. Design adds items for use cases, do we have
> existing content that fits into that template?
>
Yes, the content will be curated from the use cases we got presented at
Beam Summits. We are tracking who has given permission to do this in Jira:
BEAM-11225  [1]

Re: left navigation on the docs page. The fonts here are more readable to
> me and the search feature is great. One question, the current version has
> hierarchical and collapsible items on the left navigation. Would that still
> be possible. IMO visual grouping, and hiding detailed items makes the
> documentation easier to navigate.
>

This is the point we would like to hear feedback on, the proposal is to
remove the full detail in the left navigation and leave the hierarchy view
on the right navigation, which appears on the right when you click on a
topic. Doing this will allow us to keep the left navigation organized and
focused on the high level topics, hoping to make the experience smoother
and neater for people and do not overwhelm them with so many
topics/subtopics.

For context, experienced UX and UI designers are proposing these changes
and have optimized for a better discovery and troubleshooting experience,
hence the recommendation.

[1]
https://issues.apache.org/jira/browse/BEAM-11225?jql=project%20%3D%20BEAM%20AND%20text%20~%20%22permission%20use%20cases%22

>

Re: Website Revamp update & navigation for documentation page

[Ahmet Altay]

On Mon, Dec 14, 2020 at 6:34 AM Agnieszka Sell 
wrote:

> Hi Folks!
>
> Last week we had an 8th sprint review of the Beam Website Revamp project.
> The summary can be found here [1] and presentation from the meeting here
> [2]. During the meeting we discussed following UI designs:
>
>1. Documentation page updates (added expandable section with runners).
>2. Blog updates (updated blog cards).
>3. Powered By Apache Beam (created icons for commercial use cases and
>open source projects).
>
> A question about the content. Design adds items for use cases, do we have
existing content that fits into that template?

>
>1. Updated top navigation (added edit button and search bar).
>
> You can check all these designs in a presentation [2]. Additionally, I'd
> like to ask you to take a look at the UX design for the left navigation on
> the Documentation page (attached). How do you like it? Your feedback will
> be more than helpful :)
>

Thanks a lot for sharing these.

Re: left navigation on the docs page. The fonts here are more readable to
me and the search feature is great. One question, the current version has
hierarchical and collapsible items on the left navigation. Would that still
be possible. IMO visual grouping, and hiding detailed items makes the
documentation easier to navigate.


>
> One more thing is that we received some messages from people asking about
> the accessibility aspects of the revamped page – thank you for that! I'd
> like you to know that while redesigning the page we're following Google
> guidelines concerning accessibility. Let me know if you have any questions
> about it!
>
> Kind regards,
>
> Agnieszka
>
> [1]
> https://docs.google.com/document/d/1CqssBpRt1EjpV0nCBaS9WGaGHt_4stKXbwMz9gYfmwc/edit
> [2]
> https://docs.google.com/presentation/d/1asXPHDtW8Y4oHfCZ8QWsPf9mTVlTaxCHRMniiKPWfWQ/edit?usp=sharing
>
>
>
> --
>
> Agnieszka Sell
> Polidea  | Project Manager
>
> M: *+48 504 901 334* <+48504901334>
> E: agnieszka.s...@polidea.com
> [image: Polidea] 
>
> Check out our projects! 
> [image: Github]  [image: Facebook]
>  [image: Twitter]
>  [image: Linkedin]
>  [image: Instagram]
> 
>
> Unique Tech
> Check out our projects! 
>

Re: "org.apache.kafka:kafka-clients:5.3.2-ccs" dependency issue.

[Sonam Ramchand]

Thanks for your quick response. But, updating updating kafka-clients:1.0.0
to 2.4.1 on the PR does not resolve the issue sadly. Seems like there is
some other issue.

On Tue, Dec 15, 2020 at 2:03 AM Kyle Weaver  wrote:

> We recently upgraded kafka and kafka-clients to version 2.4.1 [1]. It
> looks like there are a couple places in your PR that use the old version
> kafka-clients:1.0.0 [2]. You will need to update your PR to use version
> 2.4.1 instead.
>
> [1]
> https://github.com/apache/beam/commit/8e6dae8105c7d8abaabf71f6529c604884c879d3
> [2] https://github.com/apache/beam/pull/12938
>
> On Mon, Dec 14, 2020 at 12:24 PM Sonam Ramchand <
> sonam.ramch...@venturedive.com> wrote:
>
>> Please refer to the link to understand the problem better
>> https://gradle.com/s/zaqcnvh2uiwga.
>>
>> Thanks!
>>
>> On Tue, Dec 15, 2020 at 1:03 AM Sonam Ramchand <
>> sonam.ramch...@venturedive.com> wrote:
>>
>>> Hi Devs,
>>> I have been getting:Could not resolve all dependencies for
>>> configuration ':sdks:java:container:dockerDependency'.
>>> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
>>> Searched in the following locations:
>>> -
>>> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> Required by:
>>> project :sdks:java:container
>>> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
>>> Searched in the following locations:
>>> -
>>> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> Required by:
>>> project :sdks:java:container > project :sdks:java:io:kafka
>>> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
>>> Searched in the following locations:
>>> -
>>> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> -
>>> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>>> Required by:
>>> project :sdks:java:container > project :sdks:java:io:kafka
>>> project :sdks:java:container > project :sdks:java:io:kafka >
>>> io.confluent:kafka-schema-registry-client:5.3.2Even after I add compile
>>> "org.apache.kafka:kafka-clients:5.3.2-ccs" to sdks:java:io:kafka
>>> build.gradle file, i get no luck.Any sort of quick help will be really
>>> appreciated.
>>>
>>> --
>>>
>>> Regards,
>>> *Sonam*
>>> Software Engineer
>>> Mobile: +92 3088337296 <+92%20308%208337296>
>>>
>>> 
>>>
>>
>>
>> --
>>
>> Regards,
>> *Sonam*
>> Software Engineer
>> Mobile: +92 3088337296 <+92%20308%208337296>
>>
>> 
>>
>

-- 

Regards,
*Sonam*
Software Engineer

Re: "org.apache.kafka:kafka-clients:5.3.2-ccs" dependency issue.

[Kyle Weaver]

We recently upgraded kafka and kafka-clients to version 2.4.1 [1]. It looks
like there are a couple places in your PR that use the old version
kafka-clients:1.0.0 [2]. You will need to update your PR to use version
2.4.1 instead.

[1]
https://github.com/apache/beam/commit/8e6dae8105c7d8abaabf71f6529c604884c879d3
[2] https://github.com/apache/beam/pull/12938

On Mon, Dec 14, 2020 at 12:24 PM Sonam Ramchand <
sonam.ramch...@venturedive.com> wrote:

> Please refer to the link to understand the problem better
> https://gradle.com/s/zaqcnvh2uiwga.
>
> Thanks!
>
> On Tue, Dec 15, 2020 at 1:03 AM Sonam Ramchand <
> sonam.ramch...@venturedive.com> wrote:
>
>> Hi Devs,
>> I have been getting:Could not resolve all dependencies for configuration
>> ':sdks:java:container:dockerDependency'.
>> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
>> Searched in the following locations:
>> -
>> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> Required by:
>> project :sdks:java:container
>> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
>> Searched in the following locations:
>> -
>> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> Required by:
>> project :sdks:java:container > project :sdks:java:io:kafka
>> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
>> Searched in the following locations:
>> -
>> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> -
>> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
>> Required by:
>> project :sdks:java:container > project :sdks:java:io:kafka
>> project :sdks:java:container > project :sdks:java:io:kafka >
>> io.confluent:kafka-schema-registry-client:5.3.2Even after I add compile
>> "org.apache.kafka:kafka-clients:5.3.2-ccs" to sdks:java:io:kafka
>> build.gradle file, i get no luck.Any sort of quick help will be really
>> appreciated.
>>
>> --
>>
>> Regards,
>> *Sonam*
>> Software Engineer
>> Mobile: +92 3088337296 <+92%20308%208337296>
>>
>> 
>>
>
>
> --
>
> Regards,
> *Sonam*
> Software Engineer
> Mobile: +92 3088337296 <+92%20308%208337296>
>
> 
>

Re: Docker Development Environment

[Brian Hulette]

Thanks Alex, I'll take a look

On Mon, Dec 14, 2020 at 12:48 PM Alex Kosolapov 
wrote:

> Hi!
>
>
>
> Wanted to send a note that PR with changes submitted
> https://github.com/apache/beam/pull/13545 and ready for code review
>
>
>
> Thanks,
>
> Alex
>
> *From: *Alex Kosolapov 
> *Date: *Thursday, December 10, 2020 at 3:44 PM
> *To: *"dev@beam.apache.org" 
> *Subject: *Re: Docker Development Environment
>
>
>
> Hi Brian,
>
>
>
> Yes - will update the script, created JIRA to track
> https://issues.apache.org/jira/browse/BEAM-11438
>
>
>
> I was thinking about dscl initially too - and dscl will require user to
> create docker group in this case as you noted. For some users that might be
> an extra manual step to do (group not created automatically by Docker
> install).
>
>
>
> Can approaches be combined?
>
>1. If docker group created – use docker group’s gid
>2. If docker group not created – assign some default, e.g. 1000 or
>some other safe way of handling “no docker group” case
>
>
>
> Thanks,
>
> Alex
>
>
>
> *From: *Brian Hulette 
> *Reply-To: *"dev@beam.apache.org" 
> *Date: *Wednesday, December 9, 2020 at 12:21 PM
> *To: *dev 
> *Subject: *Re: Docker Development Environment
>
>
>
> Hey Alex,
> Thanks for trying this out on macOS! Some responses inline
>
>
>
> On Tue, Dec 8, 2020 at 11:42 AM Alex Kosolapov 
> wrote:
>
> Hi!
>
>
>
> Thank you for creating Docker build environment - makes build environment
> setup so much easier!
>
>
>
> I ran start-build-env.sh on a macOS, and I ran into some items that wanted
> to share + propose how to improve Docker build environment for macOS
> support:
>
>1. ./start-build-env.sh: line 75: getent: command not found and script
>build error downstream.
>
>
>
> Step 26/26 : RUN echo '. /scripts/beam_env_checks.sh' >>
> /root/.bash_aliases
>
>  ---> Using cache
>
>  ---> fe48b8b26e91
>
> Successfully built fe48b8b26e91
>
> Successfully tagged beam-build:latest
>
> *./start-build-env.sh: line 75: getent: command not found*
>
> Sending build context to Docker daemon   2.56kB
>
> Step 1/10 : FROM beam-build
>
>  ---> fe48b8b26e91
>
> Step 2/10 : RUN rm -f /var/log/faillog /var/log/lastlog
>
>  ---> Using cache
>
>  ---> 11ea31f2099e
>
> Step 3/10 : RUN groupadd --non-unique -g 100 alex
>
>  ---> Using cache
>
>  ---> cf3f56f51d9f
>
> *Step 4/10 : RUN groupmod -g  docker*
>
> * ---> Running in 5d5bc473be3b*
>
> *groupmod: invalid group ID 'docker'*
>
> *The command '/bin/bash -o pipefail -c groupmod -g  docker' returned a
> non-zero code: 3*
>
>
>
> I investigated the issues and found that start-build-env.sh expected group
> "docker" created in the system groups. Creating a docker group is an
> optional post install step (
> https://docs.docker.com/engine/install/linux-postinstall/), another relevant
> reference to forum 
> gives a hint of using staff group on macOS. Solution that worked for me was
> pick another GID for DOCKER_GROUP_ID on Mac, e.g.
>
> #DOCKER_GROUP_ID=$(getent group docker | cut -d':' -f3)
>
>
>
> if [ "$(uname -s)" = "Linux" ]; then
>
> DOCKER_GROUP_ID=$(getent group docker | cut -d':' -f3)
>
> fi
>
> if [ "$(uname -s)" = "Darwin" ]; then
>
> DOCKER_GROUP_ID=1000
>
> fi
>
>
>
> I think everyone using this container should do that postinstall step so
> that docker doesn't have to be run as root. We actually point to it in an
> error message if we detect the script is run as root. I think the real
> issue here is that getent doesn't exist on MacOS, it looks like there is an
> alternative approach though:
> https://stackoverflow.com/questions/10910096/what-is-the-command-to-get-groupid-of-a-group-name-in-mac-or-linux
>
> Would you be ok updating the script to use that approach on macOS? This
> would still require users to make a docker group.
>
>
>
>
>
>1. Step 10/10 : RUN chown -R alex: /Users/alex/.cache
>
>  ---> Running in f09e2bb0e045
>
> chown: cannot access '/Users/alex/.cache': No such file or directory
>
> The command '/bin/bash -o pipefail -c chown -R alex: /Users/alex/.cache'
> returned a non-zero code: 1
>
>
>
> Fix for this was to change from HOME to DOCKER_HOME_DIR:
>
>   RUN chown -R ${USER_NAME}:${GROUP_ID}
> ${DOCKER_HOME_DIR}/.cache
>
> ENV GOPATH
> ${DOCKER_HOME_DIR}/beam/sdks/go/examples/.gogradle/project_gopath
>
> Ah yes this definitely looks like a bug. It just happened to work when
> running from a machine where the HOME dir was the same as in the container.
> This one definitely makes sense.
>
>
>
> If these proposed solutions make sense I will create JIRA ticket and
> submit these improvements for Docker build support on macOS.
>
>
>
> Thank you,
>
> Alex Kosolapov
>
>
>
> *From: *Brian Hulette 
> *Reply-To: *"dev@beam.apache.org" 
> *Date: *Friday, December 4, 2020 at 1:15 PM
> *To: *dev 
> *Cc: *Omar Ismail 
> *Subject: *Re: Docker Development Environment
>
>
>
> I think 

Re: Docker Development Environment

[Alex Kosolapov]

Hi!

Wanted to send a note that PR with changes submitted 
https://github.com/apache/beam/pull/13545 and ready for code review

Thanks,
Alex
From: Alex Kosolapov 
Date: Thursday, December 10, 2020 at 3:44 PM
To: "dev@beam.apache.org" 
Subject: Re: Docker Development Environment

Hi Brian,

Yes - will update the script, created JIRA to track 
https://issues.apache.org/jira/browse/BEAM-11438

I was thinking about dscl initially too - and dscl will require user to create 
docker group in this case as you noted. For some users that might be an extra 
manual step to do (group not created automatically by Docker install).

Can approaches be combined?

  1.  If docker group created – use docker group’s gid
  2.  If docker group not created – assign some default, e.g. 1000 or some 
other safe way of handling “no docker group” case

Thanks,
Alex

From: Brian Hulette 
Reply-To: "dev@beam.apache.org" 
Date: Wednesday, December 9, 2020 at 12:21 PM
To: dev 
Subject: Re: Docker Development Environment

Hey Alex,
Thanks for trying this out on macOS! Some responses inline

On Tue, Dec 8, 2020 at 11:42 AM Alex Kosolapov 
mailto:alex.kosola...@akvelon.com>> wrote:

Hi!



Thank you for creating Docker build environment - makes build environment setup 
so much easier!



I ran start-build-env.sh on a macOS, and I ran into some items that wanted to 
share + propose how to improve Docker build environment for macOS support:

  1.  ./start-build-env.sh: line 75: getent: command not found and script build 
error downstream.



Step 26/26 : RUN echo '. /scripts/beam_env_checks.sh' >> /root/.bash_aliases

 ---> Using cache

 ---> fe48b8b26e91

Successfully built fe48b8b26e91

Successfully tagged beam-build:latest

./start-build-env.sh: line 75: getent: command not found

Sending build context to Docker daemon   2.56kB

Step 1/10 : FROM beam-build

 ---> fe48b8b26e91

Step 2/10 : RUN rm -f /var/log/faillog /var/log/lastlog

 ---> Using cache

 ---> 11ea31f2099e

Step 3/10 : RUN groupadd --non-unique -g 100 alex

 ---> Using cache

 ---> cf3f56f51d9f

Step 4/10 : RUN groupmod -g  docker

 ---> Running in 5d5bc473be3b

groupmod: invalid group ID 'docker'

The command '/bin/bash -o pipefail -c groupmod -g  docker' returned a non-zero 
code: 3



I investigated the issues and found that start-build-env.sh expected group 
"docker" created in the system groups. Creating a docker group is an optional 
post install step (https://docs.docker.com/engine/install/linux-postinstall/), 
another relevant reference to 
forum gives a hint of 
using staff group on macOS. Solution that worked for me was pick another GID 
for DOCKER_GROUP_ID on Mac, e.g.

#DOCKER_GROUP_ID=$(getent group docker | cut -d':' -f3)



if [ "$(uname -s)" = "Linux" ]; then

DOCKER_GROUP_ID=$(getent group docker | cut -d':' -f3)

fi

if [ "$(uname -s)" = "Darwin" ]; then

DOCKER_GROUP_ID=1000

fi

I think everyone using this container should do that postinstall step so that 
docker doesn't have to be run as root. We actually point to it in an error 
message if we detect the script is run as root. I think the real issue here is 
that getent doesn't exist on MacOS, it looks like there is an alternative 
approach though: 
https://stackoverflow.com/questions/10910096/what-is-the-command-to-get-groupid-of-a-group-name-in-mac-or-linux

Would you be ok updating the script to use that approach on macOS? This would 
still require users to make a docker group.





  1.  Step 10/10 : RUN chown -R alex: /Users/alex/.cache

 ---> Running in f09e2bb0e045

chown: cannot access '/Users/alex/.cache': No such file or directory

The command '/bin/bash -o pipefail -c chown -R alex: /Users/alex/.cache' 
returned a non-zero code: 1



Fix for this was to change from HOME to DOCKER_HOME_DIR:

  RUN chown -R ${USER_NAME}:${GROUP_ID} 
${DOCKER_HOME_DIR}/.cache

ENV GOPATH ${DOCKER_HOME_DIR}/beam/sdks/go/examples/.gogradle/project_gopath
Ah yes this definitely looks like a bug. It just happened to work when running 
from a machine where the HOME dir was the same as in the container. This one 
definitely makes sense.



If these proposed solutions make sense I will create JIRA ticket and submit 
these improvements for Docker build support on macOS.



Thank you,

Alex Kosolapov


From: Brian Hulette mailto:bhule...@google.com>>
Reply-To: "dev@beam.apache.org" 
mailto:dev@beam.apache.org>>
Date: Friday, December 4, 2020 at 1:15 PM
To: dev mailto:dev@beam.apache.org>>
Cc: Omar Ismail mailto:omarism...@google.com>>
Subject: Re: Docker Development Environment

I think https://github.com/apache/beam/pull/13308 is about ready to merge. One 
question was whether or not to install pyenv in the container - I think we 
should try to do without it. Users of this environment will already be 
operating within a container, so they shouldn't need pyenv to create isolated 
python 

Re: "org.apache.kafka:kafka-clients:5.3.2-ccs" dependency issue.

[Sonam Ramchand]

Please refer to the link to understand the problem better
https://gradle.com/s/zaqcnvh2uiwga.

Thanks!

On Tue, Dec 15, 2020 at 1:03 AM Sonam Ramchand <
sonam.ramch...@venturedive.com> wrote:

> Hi Devs,
> I have been getting:Could not resolve all dependencies for configuration
> ':sdks:java:container:dockerDependency'.
> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
> Searched in the following locations:
> -
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> Required by:
> project :sdks:java:container
> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
> Searched in the following locations:
> -
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> Required by:
> project :sdks:java:container > project :sdks:java:io:kafka
> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
> Searched in the following locations:
> -
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> Required by:
> project :sdks:java:container > project :sdks:java:io:kafka
> project :sdks:java:container > project :sdks:java:io:kafka >
> io.confluent:kafka-schema-registry-client:5.3.2Even after I add compile
> "org.apache.kafka:kafka-clients:5.3.2-ccs" to sdks:java:io:kafka
> build.gradle file, i get no luck.Any sort of quick help will be really
> appreciated.
>
> --
>
> Regards,
> *Sonam*
> Software Engineer
> Mobile: +92 3088337296
>
> 
>


-- 

Regards,
*Sonam*
Software Engineer
Mobile: +92 3088337296

Re: "org.apache.kafka:kafka-clients:5.3.2-ccs" dependency issue.

[Ahmet Altay]

+Emily Ye  +Tyson Hamilton  - do
you know what might cause this?

On Mon, Dec 14, 2020 at 12:04 PM Sonam Ramchand <
sonam.ramch...@venturedive.com> wrote:

> Hi Devs,
> I have been getting:Could not resolve all dependencies for configuration
> ':sdks:java:container:dockerDependency'.
> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
> Searched in the following locations:
> -
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> Required by:
> project :sdks:java:container
> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
> Searched in the following locations:
> -
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> Required by:
> project :sdks:java:container > project :sdks:java:io:kafka
> > Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
> Searched in the following locations:
> -
> file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> -
> https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
> Required by:
> project :sdks:java:container > project :sdks:java:io:kafka
> project :sdks:java:container > project :sdks:java:io:kafka >
> io.confluent:kafka-schema-registry-client:5.3.2Even after I add compile
> "org.apache.kafka:kafka-clients:5.3.2-ccs" to sdks:java:io:kafka
> build.gradle file, i get no luck.Any sort of quick help will be really
> appreciated.
>
> --
>
> Regards,
> *Sonam*
> Software Engineer
> Mobile: +92 3088337296 <+92%20308%208337296>
>
> 
>

"org.apache.kafka:kafka-clients:5.3.2-ccs" dependency issue.

[Sonam Ramchand]

Hi Devs,
I have been getting:Could not resolve all dependencies for configuration
':sdks:java:container:dockerDependency'.
> Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
Searched in the following locations:
-
file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
Required by:
project :sdks:java:container
> Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
Searched in the following locations:
-
file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
Required by:
project :sdks:java:container > project :sdks:java:io:kafka
> Could not find org.apache.kafka:kafka-clients:5.3.2-ccs.
Searched in the following locations:
-
file:/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_PVR_Flink_Phrase/src/sdks/java/container/offline-repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
file:/home/jenkins/.m2/repository/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://jcenter.bintray.com/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://oss.sonatype.org/content/repositories/staging/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repository.apache.org/snapshots/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
-
https://repository.apache.org/content/repositories/releases/org/apache/kafka/kafka-clients/5.3.2-ccs/kafka-clients-5.3.2-ccs.pom
Required by:
project :sdks:java:container > project :sdks:java:io:kafka
project :sdks:java:container > project :sdks:java:io:kafka >
io.confluent:kafka-schema-registry-client:5.3.2Even after I add compile
"org.apache.kafka:kafka-clients:5.3.2-ccs" to sdks:java:io:kafka
build.gradle file, i get no luck.Any sort of quick help will be really
appreciated.

-- 

Regards,
*Sonam*
Software Engineer
Mobile: +92 3088337296

Re: Help measuring upcoming performance increase in flink runner on production systems

[Rion Williams]

Hi Teodor,

Although I’m sure you’ve come across it, this might have some valuable 
resources or methodologies to consider as you explore this a bit more:

https://arxiv.org/pdf/1907.08302.pdf

I’m looking forward to reading about your finding, especially using a more 
recent iteration of Beam!

Rion

> On Dec 14, 2020, at 6:37 AM, Teodor Spæren  wrote:
> 
> Just bumping this so people see it now that 2.26.0 is out :)
> 
>> On Wed, Nov 25, 2020 at 11:09:52AM +0100, Teodor Spæren wrote:
>> Hey!
>> 
>> My name is Teodor Spæren and I'm writing a master thesis investigating the 
>> performance overhead of using Beam instead of using the underlying systems 
>> directly. My focus has been on Flink and I've made a discovery about some 
>> unnecessary copying between operators in the Flink runner[1][2]. I wrote a 
>> fixed for this and it got accepted and merged,
>> and will be in the upcoming 2.26.0 release[3].
>> 
>> I'm writing this email to ask if anyone on these mailing lists would be 
>> willing to send me some result of applying this option when the new version 
>> of beam releases. Anything will be very much appreciated, stories, 
>> screenshots of performance monitoring before and after, hard numbers, 
>> anything! If you include the cluster size and the workload that would be 
>> awesome too! My master thesis is set to be complete the coming summer, so 
>> there is no real hurry :)
>> 
>> The thesis will be freely accessible[4] and I hope that these findings will 
>> be of help to the beam community. If anyone wishes to submit stories, but 
>> remain anonymous that is also ok :)
>> 
>> The best way to contact me would be to send an email my way here, or on 
>> teod...@mail.uio.no.
>> 
>> Any help is appreciated, thanks for your attention!
>> 
>> Best regards,
>> Teodor Spæren
>> 
>> 
>> [1]: 
>> https://lists.apache.org/thread.html/r24129dba98782e1cf4d18ec738ab9714dceb05ac23f13adfac5baad1%40%3Cdev.beam.apache.org%3E
>> [2]: https://issues.apache.org/jira/browse/BEAM-11146
>> [3]: https://github.com/apache/beam/pull/13240
>> [4]: https://www.duo.uio.no/

Beam Dependency Check Report (2020-12-14)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
chardet
3.0.4
4.0.0
None
2020-12-14BEAM-11445
chromedriver-binary
87.0.4280.88.0
88.0.4324.27.1
2020-12-07
2020-12-07BEAM-10426
dill
0.3.1.1
0.3.3
2019-10-07
2020-11-02BEAM-11167
google-cloud-bigquery
1.28.0
2.6.1
2020-10-05
2020-12-14BEAM-5537
google-cloud-build
2.0.0
3.0.0
2020-11-09
2020-11-09BEAM-11204
google-cloud-datastore
1.15.3
2.1.0
2020-11-16
2020-12-07BEAM-8443
google-cloud-dlp
1.0.0
3.0.0
2020-06-29
2020-12-07BEAM-10344
google-cloud-language
1.3.0
2.0.0
2020-10-26
2020-10-26BEAM-8
google-cloud-pubsub
1.7.0
2.2.0
2020-07-20
2020-12-07BEAM-5539
google-cloud-spanner
1.19.1
2.1.0
2020-11-16
2020-12-07BEAM-10345
google-cloud-videointelligence
1.16.1
2.0.0
2020-11-23
2020-11-23BEAM-11319
google-cloud-vision
1.0.0
2.0.0
2020-03-24
2020-10-05BEAM-9581
grpcio-tools
1.30.0
1.34.0
2020-06-29
2020-12-07BEAM-9582
mock
2.0.0
4.0.3
2019-05-20
2020-12-14BEAM-7369
mypy-protobuf
1.18
1.23
2020-03-24
2020-06-29BEAM-10346
nbconvert
5.6.1
6.0.7
2020-10-05
2020-10-05BEAM-11007
Pillow
7.2.0
8.0.1
2020-10-19
2020-10-26BEAM-11071
PyHamcrest
1.10.1
2.0.2
2020-01-20
2020-07-08BEAM-9155
pytest
4.6.11
6.2.0
2020-07-08
2020-12-14BEAM-8606
pytest-xdist
1.34.0
2.1.0
2020-08-17
2020-08-28BEAM-10713
setuptools
50.3.2
51.0.0
2020-10-19
2020-12-07BEAM-10714
tenacity
5.1.5
6.2.0
2019-11-11
2020-06-29BEAM-8607
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.datastax.cassandra:cassandra-driver-core
3.10.2
4.0.0
2020-08-26
2019-03-18BEAM-8674
com.esotericsoftware:kryo
4.0.2
5.0.2
2018-03-20
2020-12-01BEAM-5809
com.esotericsoftware.kryo:kryo
2.21
2.24.0
2013-02-27
2014-05-04BEAM-5574
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.33.0
0.36.0
2020-09-14
2020-11-09BEAM-6645
com.github.jk1.dependency-license-report:com.github.jk1.dependency-license-report.gradle.plugin
1.13
1.16
2020-06-29
2020-10-26BEAM-11120
com.google.api.grpc:grpc-google-common-protos
1.18.1
2.0.1
2020-08-11
2020-11-02BEAM-8633
com.google.api.grpc:proto-google-cloud-bigtable-v2
1.16.1
1.19.0
2020-10-08
2020-12-03BEAM-8679
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1
2.0.2
3.1.2
2020-10-02
2020-12-14BEAM-8682
com.google.api.grpc:proto-google-common-protos
1.18.1
2.0.1
2020-08-11
2020-11-02BEAM-6899
com.google.apis:google-api-services-bigquery
v2-rev20200916-1.30.10
v2-rev20201030-1.31.0
2020-09-30
2020-12-04BEAM-8684
com.google.apis:google-api-services-clouddebugger
v2-rev20200501-1.30.10
v2-rev20200807-1.31.0
2020-07-14
2020-12-03BEAM-8750
com.google.apis:google-api-services-cloudresourcemanager
v1-rev20200720-1.30.10
v2-rev2020-1.31.0
2020-07-25
2020-12-03BEAM-8751
com.google.apis:google-api-services-dataflow
v1b3-rev20200713-1.30.10
v1beta3-rev12-1.20.0
2020-07-25
2015-04-29BEAM-8752
com.google.apis:google-api-services-healthcare
v1beta1-rev20200713-1.30.10
v1-rev20201202-1.31.0
2020-07-24
2020-12-10BEAM-10349
com.google.apis:google-api-services-pubsub
v1-rev20200713-1.30.10
v1-rev20201201-1.31.0
2020-07-25
2020-12-11BEAM-8753
com.google.apis:google-api-services-storage
v1-rev20200927-1.30.10
v1-rev20201112-1.31.0

Re: Help measuring upcoming performance increase in flink runner on production systems

[Teodor Spæren]

Just bumping this so people see it now that 2.26.0 is out :)

On Wed, Nov 25, 2020 at 11:09:52AM +0100, Teodor Spæren wrote:

Hey!

My name is Teodor Spæren and I'm writing a master thesis investigating 
the performance overhead of using Beam instead of using the underlying 
systems directly. My focus has been on Flink and I've made a discovery 
about some unnecessary copying between operators in the Flink 
runner[1][2]. I wrote a fixed for this and it got accepted and merged,

and will be in the upcoming 2.26.0 release[3].

I'm writing this email to ask if anyone on these mailing lists would 
be willing to send me some result of applying this option when the new 
version of beam releases. Anything will be very much appreciated, 
stories, screenshots of performance monitoring before and after, hard 
numbers, anything! If you include the cluster size and the workload 
that would be awesome too! My master thesis is set to be complete the 
coming summer, so there is no real hurry :)


The thesis will be freely accessible[4] and I hope that these findings 
will be of help to the beam community. If anyone wishes to submit 
stories, but remain anonymous that is also ok :)


The best way to contact me would be to send an email my way here, or 
on teod...@mail.uio.no.


Any help is appreciated, thanks for your attention!

Best regards,
Teodor Spæren


[1]: 
https://lists.apache.org/thread.html/r24129dba98782e1cf4d18ec738ab9714dceb05ac23f13adfac5baad1%40%3Cdev.beam.apache.org%3E
[2]: https://issues.apache.org/jira/browse/BEAM-11146
[3]: https://github.com/apache/beam/pull/13240
[4]: https://www.duo.uio.no/