Re: [DISCUSS] Dependency management for Python SDK Container

- The purpose of install_requires in setup.py is to define the maximally permissive set of requirements for a package[1]. We don't pin a version in setup.py without a strong reason, instead we typically pick up a lower bound we have tested, and set an upper bound to be next major version. - The

beam_PreCommit_Java_Commit is broken

Hi all, Looks like https://builds.apache.org/job/beam_PreCommit_Java_Commit/ is broken. It fails with: FAILURE: Build failed with an exception. * Where: Build file

Re: beam_PreCommit_Java_Commit is broken

This is tracked here: https://issues.apache.org/jira/browse/BEAM-7892 /cc +Valentyn Tymofieiev +Udi Meiri On Mon, Aug 5, 2019 at 9:18 AM Alexey Romanenko wrote: > According to "git bisect”, it seems like this was introduced with commit > “149153b525236327badb138b09235ff735045adf” in PR >

Re: beam_PreCommit_Java_Commit is broken

It's being tracked by https://issues.apache.org/jira/browse/BEAM-7892. -Rui On Mon, Aug 5, 2019 at 9:18 AM Alexey Romanenko wrote: > According to "git bisect”, it seems like this was introduced with commit > “149153b525236327badb138b09235ff735045adf” in PR >

Re: [Update] Beam 2.15 Release Progress

Hi, I've verified release branch, and all Pre/Post-commits passed. The next step would be verifying the javadoc. We still have a few blocking issues,

Re: beam_PreCommit_Java_Commit is broken

According to "git bisect”, it seems like this was introduced with commit “149153b525236327badb138b09235ff735045adf” in PR https://github.com/apache/beam/pull/9223 (“[BEAM-7060] Introduce Python3-only test modules (#9223)”) Could some Python people

Re: Snapshots not been updated

I think https://issues.apache.org/jira/browse/BEAM-7892 is the reason. On Sun, Aug 4, 2019 at 11:02 PM Ismaël Mejía wrote: > Hello, > > It seems the 2.16.0-SNAPSHOTS are not been updated since last friday 02/08. > >

Beam Dependency Check Report (2019-08-05)

High Priority Dependency Updates Of Beam Python SDK: Dependency Name Current Version Latest Version Release Date Of the Current Used Version Release Date Of The Latest Release JIRA Issue google-cloud-pubsub 0.39.1 0.45.0

Re: Write-through-cache in State logic

I believe the intent is to add a new state API call telling the runner that it is blocked waiting for a response (BEAM-7000). This should allow the runner to wait till it sees one of these I'm blocked requests and then merge + batch any state calls it may have at that point in time allowing it to

Re: Write-through-cache in State logic

Now I see what you mean. On Mon, Aug 5, 2019 at 5:42 PM Thomas Weise wrote: > Hi Luke, > > I guess the answer is that it depends on the state backend. If a set > operation in the state backend is available that is more efficient than > clear+append, then it would be beneficial to have a

Re: Write-through-cache in State logic

That would add a synchronization point that forces extra latency especially in streaming mode. Wouldn't it be possible for the runner to assign the token when starting the bundle and for the SDK to pass it along the state requests? That way, there would be no need to batch and wait for a flush.

Re: Write-through-cache in State logic

Hi Luke, I guess the answer is that it depends on the state backend. If a set operation in the state backend is available that is more efficient than clear+append, then it would be beneficial to have a dedicated fn api operation to allow for such optimization. That's something that needs to be

Re: Write-through-cache in State logic

Thomas, why do you think a single round trip is needed? clear + append can be done blindly from the SDK side and it has total knowledge of the state at that point in time till the end of the bundle at which point you want to wait to get the cache token back from the runner for the append call so

Re: Latency of Google Dataflow with Pubsub

+dev On Mon, Aug 5, 2019 at 12:49 PM Dmitry Minaev wrote: > Hi there, > > I'm building streaming pipelines in Beam (using Google Dataflow runner) > and using Google Pubsub as a message broker. I've made a couple of > experiments with a very simple pipeline: consume events from Pubsub >

Re: [DISCUSS] Dependency management for Python SDK Container

On Mon, Aug 5, 2019 at 1:43 AM Valentyn Tymofieiev wrote: > - The purpose of install_requires in setup.py is to define the maximally > permissive set of requirements for a package[1]. We don't pin a version in > setup.py without a strong reason, instead we typically pick up a lower > bound we

Re: [DISCUSS] Dependency management for Python SDK Container

On Tue, Aug 6, 2019 at 2:29 AM Ahmet Altay wrote: > > > On Mon, Aug 5, 2019 at 1:43 AM Valentyn Tymofieiev > wrote: > >> - The purpose of install_requires in setup.py is to define the maximally >> permissive set of requirements for a package[1]. We don't pin a version in >> setup.py without a