Re: Proposal: Dynamic timer support (BEAM-6857)

[Reuven Lax]

Just to circle back around, after the discussion on this thread I propose
modifying the proposed API as follows:

class MyDoFn extends DoFn {
  @TimerFamily("timers") TimerSpec timers =
TimerSpecs.timerFamily(TimeDomain(EVENT_TIME));

  @ProcessElement
  public void process(@Element String e, @TimerFamily("timers") TimerMap
timers)) {
timers.set("timer1", ...);
timers.set("timer2", ...);
  }

  @OnTimer("timer") public void onTimer(@TimerId String timerFired,
@Timestamp Instant timerTs, @TimerFamily("timers") TimerMap timers) { ... }
}

Discussions around exposing DoFnSignature and DoFnInvoker to DSL authors
are a bit independent (though not completely so, as it does relate), so I
suggest splitting that into a separate discussion.

Reuven

On Mon, Oct 28, 2019 at 10:52 PM Reuven Lax  wrote:

>
>
> On Wed, Oct 23, 2019 at 1:21 AM Jan Lukavský  wrote:
>
>> Hi Reuven,
>>
>> yes, if this change is intended to be used by end users, then
>> DoFnSignatures cannot be used, agree on that. Regarding the relationship
>> with dynamic state - I agree that this is separate problem, but because it
>> is close enough, we should know how we want to deal with that. Because
>> state and timers share some functionality (after all timers need state to
>> be fault tolerant), these API should IMO share the same logic. Whatever
>> solution chosen to expose dynamic timers, it should extend to dynamic state.
>>
>> I'd like to stop a little with the premise that users want dynamic timers
>> (that is timers whose *name* - and therefore behavior - is determined by
>> incoming data). Could this case be modeled so that the timer actually has
>> one more (implicit) state variable that actually holds collection of tuples
>> (timestamp, name)? Then the timer would be invoked at given (minimum of all
>> currently set) timestamps with respective name? The question here probably
>> is - can this have performance impact? That is to say - can any runner
>> actually do anything different from this in the sense of time complexity of
>> the algorithm?
>>
>
> Yes - you could always multiplex many timers one one. This is what some
> users do today, but it tends to be very inefficient and also complex. The
> Beam model requires runners to support dynamic timers per key (e.g. that
> how windowing is implemented - each window has a separate timer), so
> there's no reason not to expose this to users.
>
>> I'm a little afraid if actually letting users define data-driven timers
>> might not be too restrictive for some runners. Yes, runners that don't have
>> this option would probably be able to resort to the logic described above,
>> but if this work could be reasonably done for all runners, then we wouldn't
>> force runners to actually implement it. And, the API could look as if the
>> timers were actually dynamic.
>>
>> Jan
>>
>> P.S. If dynamic (and therefore any number) of timers can be actually
>> implemented using single timer, that might be interesting pattern, because
>> single timer per (window, key) has many nice properties, like it implicitly
>> avoids situation where timer invocation is not ordered ([BEAM-7520]), which
>> seems to issue for multiple runners (samza, portable flink).
>>
> BEAM-7520 is simply an implementation bug. I don't think it makes sense to
> fix a bug by restricting the model.
>
>
>> On 10/22/19 6:52 PM, Reuven Lax wrote:
>>
>> Kenn:
>> +1 to using TimerFamily instead of TimerId and TimerMap.
>>
>> Jan:
>> This is definitely not just for DSLs. I've definitely seen cases where
>> the user wants different timers based on input data, so they cannot be
>> defined statically. As a thought experiment: one stated goal of state +
>> timers was to provide the low-level tools we use to implement windowing.
>> However to implement windowing you need a dynamic set of timers, not just a
>> single one. Now most users don't need to reimplement windowing (though we
>> have had some users who had that need, when they wanted something slightly
>> different than what native Beam windowing provided), however the need for
>> dynamic timers is not unheard of.
>>
>> +1 to allowing dynamic state. However I think this is separate enough
>> from timers that it doesn't need to be coupled in this discussion. Dynamic
>> state also raises the wrinkle of pipeline compatibility (as you mentioned),
>> which I think is a bit less of an issue for dynamic timers.
>>
>> Allowing a DSL to specify a DoFnSignature does not quite solve this
>> problem. The DSL still needs a way to set and process the timers. It also
>> does not solve the problem where the timers are based on input data
>> elements, so cannot be known at pipeline construction time. However what
>> might be more important is statically defining the timer families, and a
>> DSL could do this by specifying a DoFnSignature (and something similar
>> could be done with state). Also as mentioned above, this is useful to
>> normal Beam users as well, and we shouldn't force normal users to start

Re: Proposal: Dynamic timer support (BEAM-6857)

[Reuven Lax]

On Wed, Oct 23, 2019 at 1:21 AM Jan Lukavský  wrote:

> Hi Reuven,
>
> yes, if this change is intended to be used by end users, then
> DoFnSignatures cannot be used, agree on that. Regarding the relationship
> with dynamic state - I agree that this is separate problem, but because it
> is close enough, we should know how we want to deal with that. Because
> state and timers share some functionality (after all timers need state to
> be fault tolerant), these API should IMO share the same logic. Whatever
> solution chosen to expose dynamic timers, it should extend to dynamic state.
>
> I'd like to stop a little with the premise that users want dynamic timers
> (that is timers whose *name* - and therefore behavior - is determined by
> incoming data). Could this case be modeled so that the timer actually has
> one more (implicit) state variable that actually holds collection of tuples
> (timestamp, name)? Then the timer would be invoked at given (minimum of all
> currently set) timestamps with respective name? The question here probably
> is - can this have performance impact? That is to say - can any runner
> actually do anything different from this in the sense of time complexity of
> the algorithm?
>

Yes - you could always multiplex many timers one one. This is what some
users do today, but it tends to be very inefficient and also complex. The
Beam model requires runners to support dynamic timers per key (e.g. that
how windowing is implemented - each window has a separate timer), so
there's no reason not to expose this to users.

> I'm a little afraid if actually letting users define data-driven timers
> might not be too restrictive for some runners. Yes, runners that don't have
> this option would probably be able to resort to the logic described above,
> but if this work could be reasonably done for all runners, then we wouldn't
> force runners to actually implement it. And, the API could look as if the
> timers were actually dynamic.
>
> Jan
>
> P.S. If dynamic (and therefore any number) of timers can be actually
> implemented using single timer, that might be interesting pattern, because
> single timer per (window, key) has many nice properties, like it implicitly
> avoids situation where timer invocation is not ordered ([BEAM-7520]), which
> seems to issue for multiple runners (samza, portable flink).
>
BEAM-7520 is simply an implementation bug. I don't think it makes sense to
fix a bug by restricting the model.


> On 10/22/19 6:52 PM, Reuven Lax wrote:
>
> Kenn:
> +1 to using TimerFamily instead of TimerId and TimerMap.
>
> Jan:
> This is definitely not just for DSLs. I've definitely seen cases where the
> user wants different timers based on input data, so they cannot be defined
> statically. As a thought experiment: one stated goal of state + timers was
> to provide the low-level tools we use to implement windowing. However to
> implement windowing you need a dynamic set of timers, not just a single
> one. Now most users don't need to reimplement windowing (though we have had
> some users who had that need, when they wanted something slightly different
> than what native Beam windowing provided), however the need for dynamic
> timers is not unheard of.
>
> +1 to allowing dynamic state. However I think this is separate enough from
> timers that it doesn't need to be coupled in this discussion. Dynamic state
> also raises the wrinkle of pipeline compatibility (as you mentioned),
> which I think is a bit less of an issue for dynamic timers.
>
> Allowing a DSL to specify a DoFnSignature does not quite solve this
> problem. The DSL still needs a way to set and process the timers. It also
> does not solve the problem where the timers are based on input data
> elements, so cannot be known at pipeline construction time. However what
> might be more important is statically defining the timer families, and a
> DSL could do this by specifying a DoFnSignature (and something similar
> could be done with state). Also as mentioned above, this is useful to
> normal Beam users as well, and we shouldn't force normal users to start
> dealing with DoFnSignatures and DoFnInvokers.
>
>
>
>
>
>
> On Tue, Oct 22, 2019 at 7:56 AM Jan Lukavský  wrote:
>
>> Hi Max,
>>
>> wouldn't that be actually the same as
>>
>> class MyDoFn extends DoFn {
>>
>>
>>@ProcessElement
>>public void process(
>>ProcessContext context) {
>>  // "get" would register a new TimerSpec
>>  Timer timer1 = context.getTimer("timer1");
>>  Timer timer2 = context.getTimer("timer2");
>>  timers.set(...);
>>  timers.set(...);
>>}
>>
>> That is - no need to declare anything? One more concern about that - if
>> we allow registration of timers (or even state) dynamically like that it
>> might be harder to perform validation of pipeline upon upgrades.
>>
>> Jan
>>
>> On 10/22/19 4:47 PM, Maximilian Michels wrote:
>> > The idea makes sense to me. I really like that Beam gives upfront
>> > specs for timer and state, but it 

Re: Rethinking the Flink Runner modes

[Thomas Weise]

The current semantics of flink_master are tied to the Flink Java API. The
Flink client / Java API isn't a "REST API". It now uses the REST API
somewhere deep in RemoteEnvironment when the flink_master value is
host:port, but it does a lot of other things as well, such are parsing
config files and running local clusters.

A rest client to me is a thin wrapper around
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/rest_api.html


Different ways used in Beam to submit jobs to Flink:

- classic Flink runner, using Flink Java client
- job server, using Flink Java client
- generated jar, using Flink Java client
- Flink CLI with generated jar, using Flink Java client
- Python "FlinkRunner" using *REST API*

Since the last item does not submit through the Flink Java client, there is
a problem with the flink_master pipeline option. Though we cannot change
the option in a way that breaks existing setups. Like Robert suggests, we
could allow for optional URL syntax so that it can be used with the REST
API, in which case the code path that goes through the Java client will
have to disassemble such URL before handing it to Flink.

I would find it acceptable to interpret absence of the option as "[auto]",
which really means use CLI or REST API context when present, or local. I
would prefer to not have an empty string value default (but rather
None/null) and no additional magic values.

Thomas


On Mon, Oct 28, 2019 at 4:16 PM Kyle Weaver  wrote:

> Filed https://issues.apache.org/jira/browse/BEAM-8507 for the issue I
> mentioned.
>
> On Mon, Oct 28, 2019 at 4:12 PM Kyle Weaver  wrote:
>
>> > I'd like to see this issue resolved before 2.17 as changing the public
>> API once it's released will be harder.
>>
>> +1. In particular, I misunderstood that [auto] is not supported by
>> `FlinkUberJarJobServer`. Since [auto] is now the default, it's broken for
>> Python 3.6+.
>>
>> requests.exceptions.InvalidURL: Failed to parse: [auto]/v1/config
>>
>> We definitely should fix that, if nothing else.
>>
>> > One concern with this is that just supplying host:port is the existing
>> > behavior, so we can't start requiring the http://.
>>
>> The user shouldn't have to specify a protocol for Python, I think it's
>> preferable and reasonable to handle that for them in order to maintain
>> existing behavior and align with Java SDK.
>>
>> > 2. Deprecate the "[auto]" and "[local]" values. It should be sufficient
>> > to have either a non-empty address string or an empty one. The empty
>> > string would either mean local execution or, in the context of the Flink
>> > CLI tool, loading the master address from the config. The non-empty
>> > string would be interpreted as a cluster address.
>>
>> Looks like we also have a [collection] configuration value [1].
>>
>> If we're using [auto] as the default, I don't think this really makes so
>> much of a difference (as long as we're supporting and documenting these
>> properly, of course). I'm not sure there's a compelling reason to change
>> this?
>>
>> > always run locally (the least surprising to me
>>
>> I agree a local cluster should remain the default, whether that is
>> achieved through [local] or [auto] or some new mechanism such as the above.
>>
>> [1]
>> https://github.com/apache/beam/blob/2f1b56ccc506054e40afe4793a8b556e872e1865/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L80
>>
>> On Mon, Oct 28, 2019 at 6:35 AM Maximilian Michels 
>> wrote:
>>
>>> Hi,
>>>
>>> Robert and Kyle have been doing great work to simplify submitting
>>> portable pipelines with the Flink Runner. Part of this is having a
>>> Python "FlinkRunner" which handles bringing up a Beam job server and
>>> submitting the pipeline directly via the Flink REST API. One building
>>> block is the creation of "executable Jars" which contain the
>>> materialized / translated Flink pipeline and do not require the Beam job
>>> server or the Python driver anymore.
>>>
>>> While unifying a newly introduced option "flink_master_url" with the
>>> pre-existing "flink_master" [1][2], some questions came up about Flink's
>>> execution modes. (The two options are meant to do the same thing:
>>> provide the address of the Flink master to hand-over the translated
>>> pipeline.)
>>>
>>> Historically, Flink had a proprietary protocol for submitting pipelines,
>>> running on port 9091. This has since been replaced with a REST protocol
>>> at port 8081. To this date, this has implications how you submit
>>> programs, e.g. the Flink client libraries expects the address to be of
>>> form "host:port", without a protocol scheme. On the other hand, external
>>> Rest libraries typically expect a protocol scheme.
>>>
>>> But this is only half of the fun. There are also special addresses for
>>> "flink_master" that influence submission of the pipeline. If you specify
>>> "[local]" as the address, the pipeline won't be submitted but executed
>>> in a local in-process Flink cluster. If you 

Re: (mini-doc) Beam (Flink) portable job templates

[Chad Dombrova]

Thanks for the follow up, Thomas.

On Mon, Oct 28, 2019 at 7:55 PM Thomas Weise  wrote:

> Follow-up for users looking to run portable pipelines on Flink:
>
> After prototyping the generate-jar-file approach for internal deployment
> and some related discussion, the conclusion was that it is too limiting.
> The sticky point is that the jar file would need to be generated at
> container build time. That does not allow us to execute any logic in the
> Python driver program that depends on the deploy environment, such as
> retrieval of environment variables for configuration/credentials, setting a
> submission timestamp for stream positioning etc.
>
> What worked well was that no job server was required to submit the Flink
> job and the jar file could be used with the existing Flink tooling; there
> was no need to change the FlinkK8sOperator
>  at all.
>
> I then looked for a way to eliminate the build time translation and
> execute the Python driver program when the job is submitted, but still as a
> Flink entry point w/o extra job server deployment and client side
> dependencies. How can that work?
>
> https://issues.apache.org/jira/browse/BEAM-8471
>
> The main point was that there should be no requirement to install things
> on the client. FlinkK8sOperator is talking to the Flink REST API, w/o
> Python or Java. The Python dependencies need to be present on the Flink job
> manager host at the time the job is started through the REST API. That was
> something we had already solved for our container image build, and from
> conversation with few other folks this was their preferred container build
> approach also.
>
> In the future we may seek the ability to separate Flink and
> SDK/application bits into different images. For the SDK worker, this is
> intended via the external environment and sidecar container. For the client
> driver program, a similar approach could be implemented. Through an
> "external client environment", instead of a local process execution.
>
> The new Flink runner can be used as entry point for the REST API, the
> Flink CLI or standalone, especially for Flink centric automation. Of course
> portable pipelines can also be directly submitted through the SDK language
> client, via job server or other tooling, like the Python Flink client that
> Robert contributed recently.
>
> Thanks,
> Thomas
>
>
> On Thu, Aug 22, 2019 at 12:58 PM Kyle Weaver  wrote:
>
>> Following up on discussion in this morning's OSS runners meeting, I have
>> uploaded a draft PR for the full implementation (job creation + execution):
>> https://github.com/apache/beam/pull/9408
>>
>> Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com
>>
>>
>> On Tue, Aug 20, 2019 at 1:24 PM Robert Bradshaw 
>> wrote:
>>
>>> The point of expansion services is to run at pipeline construction
>>> time so that the caller can build on top of the outputs. E.g. we're
>>> hoping to expose Beam's SQL transforms to other languages via an
>>> expansion service and *not* duplicate the logic of parsing the SQL
>>> statements to determine the type(s) of the outputs. Even for simpler
>>> IOs, we would like to take advantage of schema information (e.g.
>>> looked up at construction time) to produce results and validate (or
>>> even inform) subsequent construction.
>>>
>>> I think we're also making a mistake in talking about "the" expansion
>>> service here, as if there was only one well defined service that all
>>> pipenes used. If we go the route of deferring some expansion to the
>>> runner, we need a way of naming expansion services. It seems like this
>>> proposal is simply isomorphic to defining new primitive transforms
>>> which some (all?) runners are just expected to understand.
>>>
>>> On Tue, Aug 20, 2019 at 10:11 AM Thomas Weise  wrote:
>>> >
>>> >
>>> >
>>> > On Tue, Aug 20, 2019 at 8:56 AM Lukasz Cwik  wrote:
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Aug 19, 2019 at 5:52 PM Ahmet Altay  wrote:
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Sun, Aug 18, 2019 at 12:34 PM Thomas Weise 
>>> wrote:
>>> 
>>>  There is a PR open for this:
>>> https://github.com/apache/beam/pull/9331
>>> 
>>>  (it wasn't tagged with the JIRA and therefore not linked)
>>> 
>>>  I think it is worthwhile to explore how we could further detangle
>>> the client side Python and Java dependencies.
>>> 
>>>  The expansion service is one more dependency to consider in a build
>>> environment. Is it really necessary to expand external transforms prior to
>>> submission to the job service?
>>> >>>
>>> >>>
>>> >>> +1, this will make it easier to use external transforms from the
>>> already familiar client environments.
>>> >>>
>>> >>
>>> >>
>>> >> The intent is to make it so that you CAN (not MUST) run an expansion
>>> service separate from a Runner. Creating a single endpoint that hosts both
>>> the Job and Expansion service is something that gRPC does very easily since
>>> you can host 

Re: (mini-doc) Beam (Flink) portable job templates

[Thomas Weise]

Follow-up for users looking to run portable pipelines on Flink:

After prototyping the generate-jar-file approach for internal deployment
and some related discussion, the conclusion was that it is too limiting.
The sticky point is that the jar file would need to be generated at
container build time. That does not allow us to execute any logic in the
Python driver program that depends on the deploy environment, such as
retrieval of environment variables for configuration/credentials, setting a
submission timestamp for stream positioning etc.

What worked well was that no job server was required to submit the Flink
job and the jar file could be used with the existing Flink tooling; there
was no need to change the FlinkK8sOperator
 at all.

I then looked for a way to eliminate the build time translation and execute
the Python driver program when the job is submitted, but still as a Flink
entry point w/o extra job server deployment and client side dependencies.
How can that work?

https://issues.apache.org/jira/browse/BEAM-8471

The main point was that there should be no requirement to install things on
the client. FlinkK8sOperator is talking to the Flink REST API, w/o Python
or Java. The Python dependencies need to be present on the Flink job
manager host at the time the job is started through the REST API. That was
something we had already solved for our container image build, and from
conversation with few other folks this was their preferred container build
approach also.

In the future we may seek the ability to separate Flink and SDK/application
bits into different images. For the SDK worker, this is intended via the
external environment and sidecar container. For the client driver program,
a similar approach could be implemented. Through an "external client
environment", instead of a local process execution.

The new Flink runner can be used as entry point for the REST API, the Flink
CLI or standalone, especially for Flink centric automation. Of course
portable pipelines can also be directly submitted through the SDK language
client, via job server or other tooling, like the Python Flink client that
Robert contributed recently.

Thanks,
Thomas


On Thu, Aug 22, 2019 at 12:58 PM Kyle Weaver  wrote:

> Following up on discussion in this morning's OSS runners meeting, I have
> uploaded a draft PR for the full implementation (job creation + execution):
> https://github.com/apache/beam/pull/9408
>
> Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com
>
>
> On Tue, Aug 20, 2019 at 1:24 PM Robert Bradshaw 
> wrote:
>
>> The point of expansion services is to run at pipeline construction
>> time so that the caller can build on top of the outputs. E.g. we're
>> hoping to expose Beam's SQL transforms to other languages via an
>> expansion service and *not* duplicate the logic of parsing the SQL
>> statements to determine the type(s) of the outputs. Even for simpler
>> IOs, we would like to take advantage of schema information (e.g.
>> looked up at construction time) to produce results and validate (or
>> even inform) subsequent construction.
>>
>> I think we're also making a mistake in talking about "the" expansion
>> service here, as if there was only one well defined service that all
>> pipenes used. If we go the route of deferring some expansion to the
>> runner, we need a way of naming expansion services. It seems like this
>> proposal is simply isomorphic to defining new primitive transforms
>> which some (all?) runners are just expected to understand.
>>
>> On Tue, Aug 20, 2019 at 10:11 AM Thomas Weise  wrote:
>> >
>> >
>> >
>> > On Tue, Aug 20, 2019 at 8:56 AM Lukasz Cwik  wrote:
>> >>
>> >>
>> >>
>> >> On Mon, Aug 19, 2019 at 5:52 PM Ahmet Altay  wrote:
>> >>>
>> >>>
>> >>>
>> >>> On Sun, Aug 18, 2019 at 12:34 PM Thomas Weise  wrote:
>> 
>>  There is a PR open for this:
>> https://github.com/apache/beam/pull/9331
>> 
>>  (it wasn't tagged with the JIRA and therefore not linked)
>> 
>>  I think it is worthwhile to explore how we could further detangle
>> the client side Python and Java dependencies.
>> 
>>  The expansion service is one more dependency to consider in a build
>> environment. Is it really necessary to expand external transforms prior to
>> submission to the job service?
>> >>>
>> >>>
>> >>> +1, this will make it easier to use external transforms from the
>> already familiar client environments.
>> >>>
>> >>
>> >>
>> >> The intent is to make it so that you CAN (not MUST) run an expansion
>> service separate from a Runner. Creating a single endpoint that hosts both
>> the Job and Expansion service is something that gRPC does very easily since
>> you can host multiple service definitions on a single port.
>> >
>> >
>> > Yes, that's fine. The point here is when the expansion occurs. I
>> believe the runner can also invoke the expansion service, thereby
>> eliminating the expansion service interaction 

Re: RFC: python static typing PR

[Robert Burke]

As someone who cribs from the Python SDK to make changes in the Go SDK,
this will make things much easier to follow! Thank you.

On Mon, Oct 28, 2019, 6:52 PM Chad Dombrova  wrote:

>
> Wow, that is an incredible amount of work!
>>
>
> Some people meditate.  I annotate ;)
>
> I'm definitely of the opinion that there's no viable counterargument to
>> the value of types, especially for large or complex codebases.
>>
>
> Agreed.  That's part of why I waited until I got the whole thing passing
> before really promoting the review of this PR.
>
> Robert and I have worked out a rough plan for merging:
>
>- I'll make a new PR with the foundations of the existing PR -- these
>are almost entirely type comments so no appreciable runtime changes --
>along with the mypy lint, which will be part of python precommit but not
>affect its failure status.   We'll get that merged first.
>- From there it should be easy for others to review and merge the
>remaining commits in parallel.
>- Once everything is in, we'll make a final commit to stop ignoring
>failures for the mypy job.
>
>
> For those concerned about yet another lint (a very reasonable concern!)
> the py37-mypy tox job completes in 37s on my macbook pro (excluding
> virtualenv setup time).
>
> -chad
>
>
>
>

Re: [DISCUSS] How to stopp SdkWorker in SdkHarness

[jincheng sun]

Sure, Thank you for your confirmation Luke! :)

Best,
Jincheng

Luke Cwik  于2019年10月29日周二 上午1:20写道：

> I would go with creating JIRAs and PRs directly since this doesn't seem to
> be contentious since you have received feedback from a few folks and they
> are all suggesting the same thing.
>
> On Sun, Oct 27, 2019 at 9:27 PM jincheng sun 
> wrote:
>
>> Hi all,
>>
>> Thanks a lot for your feedback. It seems that we have reached consensus
>> that both "Approach 2" and "Approach 3" are needed. "Approach 3" is a good
>> supplement for "Approach 2" and I also prefer "Approach 2" and "Approach 3"
>> for now.
>>
>> Do we need to vote on this discussion or I can create JIRAs and submit
>> the PRs directly?
>>
>> Best,
>> Jincheng
>>
>> Luke Cwik  于2019年10月26日周六 上午4:01写道：
>>
>>> Approach 3 is about caching the bundle descriptor forever but tearing
>>> down a "live" instance of the DoFns at some SDK chosen arbitrary point in
>>> time. This way if a future ProcessBundleRequest comes in, a new "live"
>>> instance can be constructed.
>>> Approach 2 is still needed so that when the workers are being
>>> shutdown all the "live" instances are torn down.
>>>
>>> On Fri, Oct 25, 2019 at 12:56 PM Robert Burke 
>>> wrote:
>>>
 Approach 2 isn't incompatible with approach 3. 3 simple sets down
 convention/configuration for the conditions when the SDK will do this after
 process bundle has completed.



 On Fri, Oct 25, 2019, 12:34 PM Robert Bradshaw 
 wrote:

> I think we'll still need approach (2) for when the pipeline finishes
> and a runner is tearing down workers.
>
> On Fri, Oct 25, 2019 at 10:36 AM Maximilian Michels 
> wrote:
> >
> > Hi Jincheng,
> >
> > Thanks for bringing this up and capturing the ideas in the doc.
> >
> > Intuitively, I would have also considered adding a new Proto message
> for
> > the teardown, but I think the idea to trigger this logic when the SDK
> > Harness evicts process bundle descriptors is more elegant.
> >
> > Thanks,
> > Max
> >
> > On 25.10.19 17:23, Luke Cwik wrote:
> > > I like approach 3 since it doesn't add additional complexity to
> the API
> > > and individual SDKs can choose to implement any clean-up strategy
> they
> > > want or none at all which is the simplest.
> > >
> > > On Thu, Oct 24, 2019 at 8:46 PM jincheng sun <
> sunjincheng...@gmail.com
> > > > wrote:
> > >
> > > Hi,
> > >
> > > Thanks for your comments in doc, I have add Approach 3 which
> you
> > > mentioned! @Luke
> > >
> > > For now, we should do a decision for Approach 3 and Approach 1.
> > > Detail can be found in doc [1]
> > >
> > > Welcome anyone's feedback :)
> > >
> > > Regards,
> > > Jincheng
> > >
> > > [1]
> > >
> https://docs.google.com/document/d/1sCgy9VQPf9zVXKRquK8P6N4x7aB62GEO8ozkujRSHZg/edit?usp=sharing
> > >
> > > jincheng sun  > > > 于2019年10月25日周五 上午10:40写道：
> > >
> > > Hi,
> > >
> > > Functionally capable of `abort`, but it will be called at
> the
> > > end of operator. So, I prefer `dispose` semantics. i.e.,
> all
> > > normal logic has been executed.
> > >
> > > Best,
> > > Jincheng
> > >
> > > Harsh Vardhan  anan...@google.com>>
> > > 于2019年10月23日周三 上午12:14写道：
> > >
> > > Would approach 1 be akin to abort semantics?
> > >
> > > On Mon, Oct 21, 2019 at 8:01 PM jincheng sun
> > >  sunjincheng...@gmail.com>>
> > > wrote:
> > >
> > > Hi Luke,
> > >
> > > Thanks a lot for your reply. Since it allows to
> share
> > > one SDK harness between multiple executable
> stages, the
> > > control service termination may occur much later
> than
> > > the completion of an executable stage. This is the
> main
> > > reason I prefer runners to control the teardown of
> DoFns.
> > >
> > > Regarding to "SDK harnesses can terminate
> instances any
> > > time they want and start new instances anytime as
> > > well.", personally I think it's not conflict with
> the
> > > proposed Approach 1 as the SDK harness could
> decide what
> > > to do when receiving the teardown request. It
> could do
> > > nothing if the DoFns has already been teared down
> and
> > > could also tear down the DoFns if needed.
> > >
> > > What do you think?
> > >
> > >   

Re: RFC: python static typing PR

[Chad Dombrova]

> Wow, that is an incredible amount of work!
>

Some people meditate.  I annotate ;)

I'm definitely of the opinion that there's no viable counterargument to the
> value of types, especially for large or complex codebases.
>

Agreed.  That's part of why I waited until I got the whole thing passing
before really promoting the review of this PR.

Robert and I have worked out a rough plan for merging:

   - I'll make a new PR with the foundations of the existing PR -- these
   are almost entirely type comments so no appreciable runtime changes --
   along with the mypy lint, which will be part of python precommit but not
   affect its failure status.   We'll get that merged first.
   - From there it should be easy for others to review and merge the
   remaining commits in parallel.
   - Once everything is in, we'll make a final commit to stop ignoring
   failures for the mypy job.


For those concerned about yet another lint (a very reasonable concern!) the
py37-mypy tox job completes in 37s on my macbook pro (excluding virtualenv
setup time).

-chad

Re: RFC: python static typing PR

[Ahmet Altay]

Thank you Chad, everyone else who helped with reviews so far. I think this
is a positive change.

I do share the worry of one more lint. I hope that we can hear from
majority of beam python contributors. Good tooling and educational
information would be helpful here. Ideally this will serve current
developers and future first time contributors equally well.

A question about this particular approach is the choice of type comments.
Assuming this was necessary for python 2 and that will eventually phase
out, what would be the next steps for converting this into annotations?

Ahmet

On Mon, Oct 28, 2019 at 6:28 PM Kenneth Knowles  wrote:

> Wow, that is an incredible amount of work!
>
> I'm definitely of the opinion that there's no viable counterargument to
> the value of types, especially for large or complex codebases.
>
> This kind of check must be in precommit or it will become perma-red very
> quickly.
>
> Kenn
>
> On Mon, Oct 28, 2019 at 4:21 PM Valentyn Tymofieiev 
> wrote:
>
>> Thanks a lot, Chad. Looking at the PR, I am incredibly happy to see
>> explicit type annotations throughout Beam codebase. I believe this is a
>> step in the right direction even if the tooling were not able to do any
>> inference at all. The effort required from developers to add annotations in
>> their code changes will be well compensated by reducing cognitive load
>> required to read the codebase, and will lower the entry barrier for new
>> contributions. Lack of docstrings in Beam internals in Beam codebase,
>> especially the lack of typing information, has repeatedly come up as a
>> source of frustration among several folks with whom I work.
>>
>> As Beam devs will be gaining more first-hand experience with the tooling,
>> we may need to add a style guide/best practices/FAQ to our contributor
>> guide to clarify known issues.
>>
>> Happy to help with reviewing individual commits once we have a merge plan
>> in place.
>>
>> On Mon, Oct 28, 2019 at 2:41 PM Robert Bradshaw 
>> wrote:
>>
>>> Thanks, Chad, this has been a herculean task. I'm excited for the
>>> additional tooling and documentation explicit types can bring to our
>>> code, even if tooling such as mypy isn't able to do as much inference
>>> for obvious cases as I would like.
>>>
>>> This will, of course, put another burden on developers in contributing
>>> code, similar to lint and writing unit tests. IMHO this will be worth
>>> the cost in terms of encouraging better code, but I think it's
>>> important to establish consensus if this is the direction we're
>>> moving. So if you have any thoughts or questions, positive or
>>> negative, please comment.
>>>
>>>
>>> On Mon, Oct 28, 2019 at 10:34 AM Chad Dombrova 
>>> wrote:
>>> >
>>> > Hi all,
>>> > I've been working on a PR to add static typing to the beam python sdk
>>> for the past 4 months or so.  This has been an epic journey which has
>>> required chasing down numerous fixes across several other projects (mypy,
>>> pylint, python-future), but the mypy tests are now passing!
>>> >
>>> > I'm not sure how much convincing I need to do with this group on the
>>> benefits of static typing, especially considering the heavy Java influence
>>> on this project, but for those who are curious, there's some good info
>>> here:  https://realpython.com/python-type-checking/#pros-and-cons.
>>> Suffice it to say, it's a game changer for projects like Beam (large code
>>> base, high level of quality, good testing infrastructure), and it dovetails
>>> perfectly into the efforts surrounding pipeline type analysis and
>>> serialization.  Anecdotally speaking, all of the developers I've worked
>>> with who were originally resistant to static typing in python ("it's
>>> ugly!"  "it's not pythonic!") have changed their tune and fully embraced it
>>> because of the time that it saves them every day.
>>> >
>>> > More details over at the PR:  https://github.com/apache/beam/pull/9056
>>> >
>>> > I look forward to your feedback!
>>> >
>>> > -chad
>>> >
>>>
>>

Re: [Discuss] Ideas for Apache Beam presence in social media

[Matthias Baetens]

Awesome Aizhamal :) Lmk if I can be of any help!

On Mon, Oct 28, 2019, 11:14 Aizhamal Nurmamat kyzy 
wrote:

> Thank you Matthias,
>
> I was supposed to write up the documentation.. sorry this got slipped
> through the cracks. I will prepare the PR until the end of the week.
>
> On Tue, Oct 22, 2019, 12:51 AM Matthias Baetens 
> wrote:
>
>> Thanks Thomas.
>>
>> Happy to help on the doc side when I find some time :) I'll give you a
>> ping when I have the PR ready!
>>
>> On Mon, Oct 14, 2019, 20:07 Thomas Weise  wrote:
>>
>>> Matthias,
>>>
>>> The process is already in use, but it would be nice to have it
>>> documented also.
>>>
>>> I gave you edit access to the spreadsheet, since working with the
>>> comments is cumbersome and sheets does not have suggestions.
>>>
>>> Thanks
>>>
>>>
>>> On Fri, Oct 11, 2019 at 11:59 PM Matthias Baetens <
>>> baetensmatth...@gmail.com> wrote:
>>>
 Hi all,

 Picking up this thread, since I wanted to use this facility and help
 drive this if necessary.

 I saw the sheet has now comment access enabled. Did we decide /
 document the desired process on the website? I am happy to testdrive that
 process and submit a PR if successful.

 Many thanks,
 Matthias

 On Tue, 13 Aug 2019 at 01:49, Thomas Weise  wrote:

> Yes, everyone should have comment access for this to make sense. Sorry
> for the confusion.
>
>
> On Mon, Aug 12, 2019 at 5:30 PM Kenneth Knowles 
> wrote:
>
>> Thanks for setting this up. It is nice to start building up a system
>> for this so everyone can participate.
>>
>> Regarding Jira versus notifications, how are people with only view
>> access to make suggestions for tweets? When I suggested gdocs, I meant 
>> for
>> everyone to have "comment" access, so then anyone can subscribe to all
>> comments, which would include suggestions. This allows anyone to suggest
>> tweets and anyone to subscribe to suggestions.
>>
>> Kenn
>>
>> On Wed, Aug 7, 2019 at 4:07 PM Aizhamal Nurmamat kyzy <
>> aizha...@google.com> wrote:
>>
>>> Thanks Thomas, changed the doc to view only and granted you and
>>> Ahmet edit access.
>>> @all - please send requests for access with your google accounts. I
>>> will update the thread once I document the process and submit the PR to 
>>> the
>>> website.
>>>
>>> Thank you,
>>> Aizhamal
>>>
>>> On Wed, Aug 7, 2019 at 3:12 PM Thomas Weise  wrote:
>>>
 I was able to subscribe now.

 Reminder for others that the spreadsheet of interest can be found
 here: s.apache.org/beam-tweets

 Aizhamal,

 Can you help with a couple changes to bring this closer to how
 similar gdoc resources are handled?

 * Make the document view only. *PMC members* that care to help
 with this can request edit access.
 * Document the process for other contributors. Maybe here?
 https://beam.apache.org/contribute/

 Thanks!



 On Wed, Aug 7, 2019 at 2:39 PM Ahmet Altay 
 wrote:

> I am able to subscribe to notifications now. Thomas does it work
> for you?
>
> On Wed, Aug 7, 2019 at 2:23 PM Aizhamal Nurmamat kyzy <
> aizha...@apache.org> wrote:
>
>> Hi all,
>>
>> I set the access to 'anyone can edit'. Let me know if
>> notifications work now.
>>
>> Thanks,
>>
>> On Wed, Aug 7, 2019 at 2:00 PM Ahmet Altay 
>> wrote:
>>
>>> You are probably right and it is an access issue.
>>>
>>> Aizhamal, could you give us edit access? And we can see if
>>> notifications work after that.
>>>
>>> On Wed, Aug 7, 2019 at 1:41 PM Thomas Weise 
>>> wrote:
>>>
 The use of JIRA was also suggested before, but why do the
 notifications not work? I wasn't able to subscribe and I suspect 
 that was
 due to not having sufficient access to the spreadsheet?



 On Wed, Aug 7, 2019 at 1:26 PM Ahmet Altay 
 wrote:

> As far as I understand we have not resolved this discussion
> and the sticking issue is that there is no good way of 
> subscribing to
> changes (i.e. proposals for tweets) for interested parties. The 
> method
> suggested in this thread (e.g. Tools and then Notification 
> rules.) does not
> work for some reason for a few of us including myself.
>
> Could we try to use any of our existing tools? For example,
> could the proposals be done in the form of filing a new JIRA 
> issue under a

Re: Quota issues again

[Kenneth Knowles]

It may also be advantageous to separate most submodules to not run a giant
generic Java precommit. Each IO really only needs its own, and to register
itself in the global Java precommit run only for the core. The bookkeeping
may become quite a lot, but this is the natural structure.

Kenn

On Mon, Oct 28, 2019 at 6:12 PM Chad Dombrova  wrote:

> Can we get more aggressive about separating tests into groups by those
> that are dependent on other languages and those that are not?  I think we
> could dramatically reduce our backlog if we didn’t run all of the Java
> tests every time a commit is made that only affects python code, and vice
> versa.
>
> -chad
>
>
> On Mon, Oct 28, 2019 at 3:05 PM Mikhail Gryzykhin 
> wrote:
>
>> Quota jira issue:
>> https://issues.apache.org/jira/browse/BEAM-8195
>>
>> On Mon, Oct 28, 2019 at 2:05 PM Mikhail Gryzykhin 
>> wrote:
>>
>>> Hi everyone,
>>>
>>>
>>> While validating release branch, I got failure due Quota again. Also, 
>>> current queue time for jobs is more than 1.5 hours.
>>>
>>>
>>> I'm not sure if it is worth starting another thread on tests efficiency, 
>>> but still want to keep this mail to highlight the issues.
>>>
>>>
>>> See PS for links.
>>>
>>>
>>> Regards,
>>>
>>> --Mikhail
>>>
>>>
>>> PS:
>>>
>>> https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull
>>>
>>> *13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed: googleapi: 
>>> Error 429: Quota exceeded for quota metric 
>>> 'dataflow.googleapis.com/create_requests' and limit 
>>> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for 
>>> consumer 'project_number:844138762903'., rateLimitExceeded
>>>
>>>
>>> Queue time:
>>>
>>> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>>>
>>>

Re: RFC: python static typing PR

[Kenneth Knowles]

Wow, that is an incredible amount of work!

I'm definitely of the opinion that there's no viable counterargument to the
value of types, especially for large or complex codebases.

This kind of check must be in precommit or it will become perma-red very
quickly.

Kenn

On Mon, Oct 28, 2019 at 4:21 PM Valentyn Tymofieiev 
wrote:

> Thanks a lot, Chad. Looking at the PR, I am incredibly happy to see
> explicit type annotations throughout Beam codebase. I believe this is a
> step in the right direction even if the tooling were not able to do any
> inference at all. The effort required from developers to add annotations in
> their code changes will be well compensated by reducing cognitive load
> required to read the codebase, and will lower the entry barrier for new
> contributions. Lack of docstrings in Beam internals in Beam codebase,
> especially the lack of typing information, has repeatedly come up as a
> source of frustration among several folks with whom I work.
>
> As Beam devs will be gaining more first-hand experience with the tooling,
> we may need to add a style guide/best practices/FAQ to our contributor
> guide to clarify known issues.
>
> Happy to help with reviewing individual commits once we have a merge plan
> in place.
>
> On Mon, Oct 28, 2019 at 2:41 PM Robert Bradshaw 
> wrote:
>
>> Thanks, Chad, this has been a herculean task. I'm excited for the
>> additional tooling and documentation explicit types can bring to our
>> code, even if tooling such as mypy isn't able to do as much inference
>> for obvious cases as I would like.
>>
>> This will, of course, put another burden on developers in contributing
>> code, similar to lint and writing unit tests. IMHO this will be worth
>> the cost in terms of encouraging better code, but I think it's
>> important to establish consensus if this is the direction we're
>> moving. So if you have any thoughts or questions, positive or
>> negative, please comment.
>>
>>
>> On Mon, Oct 28, 2019 at 10:34 AM Chad Dombrova  wrote:
>> >
>> > Hi all,
>> > I've been working on a PR to add static typing to the beam python sdk
>> for the past 4 months or so.  This has been an epic journey which has
>> required chasing down numerous fixes across several other projects (mypy,
>> pylint, python-future), but the mypy tests are now passing!
>> >
>> > I'm not sure how much convincing I need to do with this group on the
>> benefits of static typing, especially considering the heavy Java influence
>> on this project, but for those who are curious, there's some good info
>> here:  https://realpython.com/python-type-checking/#pros-and-cons.
>> Suffice it to say, it's a game changer for projects like Beam (large code
>> base, high level of quality, good testing infrastructure), and it dovetails
>> perfectly into the efforts surrounding pipeline type analysis and
>> serialization.  Anecdotally speaking, all of the developers I've worked
>> with who were originally resistant to static typing in python ("it's
>> ugly!"  "it's not pythonic!") have changed their tune and fully embraced it
>> because of the time that it saves them every day.
>> >
>> > More details over at the PR:  https://github.com/apache/beam/pull/9056
>> >
>> > I look forward to your feedback!
>> >
>> > -chad
>> >
>>
>

Re: Quota issues again

[Chad Dombrova]

Can we get more aggressive about separating tests into groups by those that
are dependent on other languages and those that are not?  I think we could
dramatically reduce our backlog if we didn’t run all of the Java tests
every time a commit is made that only affects python code, and vice versa.

-chad


On Mon, Oct 28, 2019 at 3:05 PM Mikhail Gryzykhin  wrote:

> Quota jira issue:
> https://issues.apache.org/jira/browse/BEAM-8195
>
> On Mon, Oct 28, 2019 at 2:05 PM Mikhail Gryzykhin 
> wrote:
>
>> Hi everyone,
>>
>>
>> While validating release branch, I got failure due Quota again. Also, 
>> current queue time for jobs is more than 1.5 hours.
>>
>>
>> I'm not sure if it is worth starting another thread on tests efficiency, but 
>> still want to keep this mail to highlight the issues.
>>
>>
>> See PS for links.
>>
>>
>> Regards,
>>
>> --Mikhail
>>
>>
>> PS:
>>
>> https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull
>>
>> *13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed: googleapi: 
>> Error 429: Quota exceeded for quota metric 
>> 'dataflow.googleapis.com/create_requests' and limit 
>> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for 
>> consumer 'project_number:844138762903'., rateLimitExceeded
>>
>>
>> Queue time:
>>
>> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>>
>>

Re: Python Precommit duration pushing 2 hours

[Pablo Estrada]

*not deciles, but 9-percentiles : )

On Mon, Oct 28, 2019 at 5:31 PM Pablo Estrada  wrote:

> I've ran the tests in Python 2 (without cython), and used a utility to
> track runtime for each test method. I found some of the following things:
> - Total test methods run: 2665
> - Total test runtime: 990 seconds
> - Deciles of time spent:
>   - 1949 tests run in the first 9% of time
>   - 173 in the 9-18% rang3e
>   - 130 in the 18-27% range
>   - 95 in the 27-36% range
>   - 77
>   - 66
>   - 55
>   - 46
>   - 37
>   - 24
>   - 13 tests run in the last 9% of time. This represents about 1 minute
> and a half.
>
> We may be able to look at the slowest X tests, and get gradual
> improvements from there. Although it seems .. not dramatic ones : )
>
> FWIW I uploaded the results here:
> https://storage.googleapis.com/apache-beam-website-pull-requests/python-tests/nosetimes.json
>
> The slowest 13 tests were:
>
>
> [('apache_beam.runners.interactive.pipeline_analyzer_test.PipelineAnalyzerTest.test_basic',
>   5.253582000732422),
>
>  
> ('apache_beam.runners.interactive.interactive_runner_test.InteractiveRunnerTest.test_wordcount',
>   7.907713890075684),
>
>  
> ('apache_beam.io.gcp.bigquery_test.PipelineBasedStreamingInsertTest.test_failure_has_same_insert_ids',
>   5.237942934036255),
>  ('apache_beam.transforms.combiners_test.CombineTest.test_global_sample',
>   5.563946008682251),
>
>  
> ('apache_beam.runners.worker.sideinputs_test.EmulatedCollectionsTest.test_large_iterable_values',
>   5.680700063705444),
>
>  
> ('apache_beam.io.parquetio_test.TestParquet.test_sink_transform_multiple_row_group',
>   6.111238956451416),
>
>  
> ('apache_beam.runners.worker.statesampler_test.StateSamplerTest.test_basic_sampler',
>   6.007534980773926),
>
>  
> ('apache_beam.runners.interactive.interactive_runner_test.InteractiveRunnerTest.test_basic',
>   13.993916988372803),
>
>  
> ('apache_beam.runners.interactive.pipeline_analyzer_test.PipelineAnalyzerTest.test_read_cache_expansion',
>   6.3383049964904785),
>
>  
> ('apache_beam.runners.interactive.pipeline_analyzer_test.PipelineAnalyzerTest.test_word_count',
>   9.157485008239746),
>
>  
> ('apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses.test_pardo_side_and_main_outputs',
>   5.191173076629639),
>
>  
> ('apache_beam.io.vcfio_test.VcfSourceTest.test_pipeline_read_file_pattern_large',
>   6.2221620082855225),
>
>  ('apache_beam.io.fileio_test.WriteFilesTest.test_streaming_complex_timing',
>   7.7187910079956055)]
>
> On Mon, Oct 28, 2019 at 3:10 PM Pablo Estrada  wrote:
>
>> I have written https://github.com/apache/beam/pull/9910 to reduce
>> FnApiRunnerTest variations.
>> I'm not in a rush to merge, but rather happy to start a discussion.
>> I'll also try to figure out if there are other tests slowing down the
>> suite significantly.
>> Best
>> -P.
>>
>> On Fri, Oct 25, 2019 at 7:41 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Thanks, Brian.
>>> +Udi Meiri 
>>> As next step, it would be good to know whether slowdown is caused by
>>> tests in this PR, or its effect on other tests, and to confirm that only
>>> Python 2 codepaths were affected.
>>>
>>> On Fri, Oct 25, 2019 at 6:35 PM Brian Hulette 
>>> wrote:
>>>
 I did a bisect based on the runtime of `./gradlew
 :sdks:python:test-suites:tox:py2:testPy2Gcp` around the commits between 9/1
 and 9/15 to see if I could find the source of the spike that happened
 around 9/6. It looks like it was due to PR#9283 [1]. I thought maybe this
 search would reveal some mis-guided configuration change, but as far as I
 can tell 9283 just added a well-tested feature. I don't think there's
 anything to learn from that... I just wanted to circle back about it in
 case others are curious about that spike.

 I'm +1 on bumping some FnApiRunner configurations.

 Brian

 [1] https://github.com/apache/beam/pull/9283

 On Fri, Oct 25, 2019 at 4:49 PM Pablo Estrada 
 wrote:

> I think it makes sense to remove some of the extra FnApiRunner
> configurations. Perhaps some of the multiworkers and some of the grpc
> versions?
> Best
> -P.
>
> On Fri, Oct 25, 2019 at 12:27 PM Robert Bradshaw 
> wrote:
>
>> It looks like fn_api_runner_test.py is quite expensive, taking 10-15+
>> minutes on each version of Python. This test consists of a base class
>> that is basically a validates runner suite, and is then run in several
>> configurations, many more of which (including some expensive ones)
>> have been added lately.
>>
>> class FnApiRunnerTest(unittest.TestCase):
>> class FnApiRunnerTestWithGrpc(FnApiRunnerTest):
>> class FnApiRunnerTestWithGrpcMultiThreaded(FnApiRunnerTest):
>> class FnApiRunnerTestWithDisabledCaching(FnApiRunnerTest):
>> class FnApiRunnerTestWithMultiWorkers(FnApiRunnerTest):
>> class FnApiRunnerTestWithGrpcAndMultiWorkers(FnApiRunnerTest):

Re: Python Precommit duration pushing 2 hours

[Pablo Estrada]

I've ran the tests in Python 2 (without cython), and used a utility to
track runtime for each test method. I found some of the following things:
- Total test methods run: 2665
- Total test runtime: 990 seconds
- Deciles of time spent:
  - 1949 tests run in the first 9% of time
  - 173 in the 9-18% rang3e
  - 130 in the 18-27% range
  - 95 in the 27-36% range
  - 77
  - 66
  - 55
  - 46
  - 37
  - 24
  - 13 tests run in the last 9% of time. This represents about 1 minute and
a half.

We may be able to look at the slowest X tests, and get gradual improvements
from there. Although it seems .. not dramatic ones : )

FWIW I uploaded the results here:
https://storage.googleapis.com/apache-beam-website-pull-requests/python-tests/nosetimes.json

The slowest 13 tests were:

[('apache_beam.runners.interactive.pipeline_analyzer_test.PipelineAnalyzerTest.test_basic',
  5.253582000732422),
 
('apache_beam.runners.interactive.interactive_runner_test.InteractiveRunnerTest.test_wordcount',
  7.907713890075684),
 
('apache_beam.io.gcp.bigquery_test.PipelineBasedStreamingInsertTest.test_failure_has_same_insert_ids',
  5.237942934036255),
 ('apache_beam.transforms.combiners_test.CombineTest.test_global_sample',
  5.563946008682251),
 
('apache_beam.runners.worker.sideinputs_test.EmulatedCollectionsTest.test_large_iterable_values',
  5.680700063705444),
 
('apache_beam.io.parquetio_test.TestParquet.test_sink_transform_multiple_row_group',
  6.111238956451416),
 
('apache_beam.runners.worker.statesampler_test.StateSamplerTest.test_basic_sampler',
  6.007534980773926),
 
('apache_beam.runners.interactive.interactive_runner_test.InteractiveRunnerTest.test_basic',
  13.993916988372803),
 
('apache_beam.runners.interactive.pipeline_analyzer_test.PipelineAnalyzerTest.test_read_cache_expansion',
  6.3383049964904785),
 
('apache_beam.runners.interactive.pipeline_analyzer_test.PipelineAnalyzerTest.test_word_count',
  9.157485008239746),
 
('apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses.test_pardo_side_and_main_outputs',
  5.191173076629639),
 
('apache_beam.io.vcfio_test.VcfSourceTest.test_pipeline_read_file_pattern_large',
  6.2221620082855225),
 ('apache_beam.io.fileio_test.WriteFilesTest.test_streaming_complex_timing',
  7.7187910079956055)]

On Mon, Oct 28, 2019 at 3:10 PM Pablo Estrada  wrote:

> I have written https://github.com/apache/beam/pull/9910 to reduce
> FnApiRunnerTest variations.
> I'm not in a rush to merge, but rather happy to start a discussion.
> I'll also try to figure out if there are other tests slowing down the
> suite significantly.
> Best
> -P.
>
> On Fri, Oct 25, 2019 at 7:41 PM Valentyn Tymofieiev 
> wrote:
>
>> Thanks, Brian.
>> +Udi Meiri 
>> As next step, it would be good to know whether slowdown is caused by
>> tests in this PR, or its effect on other tests, and to confirm that only
>> Python 2 codepaths were affected.
>>
>> On Fri, Oct 25, 2019 at 6:35 PM Brian Hulette 
>> wrote:
>>
>>> I did a bisect based on the runtime of `./gradlew
>>> :sdks:python:test-suites:tox:py2:testPy2Gcp` around the commits between 9/1
>>> and 9/15 to see if I could find the source of the spike that happened
>>> around 9/6. It looks like it was due to PR#9283 [1]. I thought maybe this
>>> search would reveal some mis-guided configuration change, but as far as I
>>> can tell 9283 just added a well-tested feature. I don't think there's
>>> anything to learn from that... I just wanted to circle back about it in
>>> case others are curious about that spike.
>>>
>>> I'm +1 on bumping some FnApiRunner configurations.
>>>
>>> Brian
>>>
>>> [1] https://github.com/apache/beam/pull/9283
>>>
>>> On Fri, Oct 25, 2019 at 4:49 PM Pablo Estrada 
>>> wrote:
>>>
 I think it makes sense to remove some of the extra FnApiRunner
 configurations. Perhaps some of the multiworkers and some of the grpc
 versions?
 Best
 -P.

 On Fri, Oct 25, 2019 at 12:27 PM Robert Bradshaw 
 wrote:

> It looks like fn_api_runner_test.py is quite expensive, taking 10-15+
> minutes on each version of Python. This test consists of a base class
> that is basically a validates runner suite, and is then run in several
> configurations, many more of which (including some expensive ones)
> have been added lately.
>
> class FnApiRunnerTest(unittest.TestCase):
> class FnApiRunnerTestWithGrpc(FnApiRunnerTest):
> class FnApiRunnerTestWithGrpcMultiThreaded(FnApiRunnerTest):
> class FnApiRunnerTestWithDisabledCaching(FnApiRunnerTest):
> class FnApiRunnerTestWithMultiWorkers(FnApiRunnerTest):
> class FnApiRunnerTestWithGrpcAndMultiWorkers(FnApiRunnerTest):
> class FnApiRunnerTestWithBundleRepeat(FnApiRunnerTest):
> class FnApiRunnerTestWithBundleRepeatAndMultiWorkers(FnApiRunnerTest):
>
> I'm not convinced we need to run all of these permutations, or at
> least not all tests in all permutations.
>
> On Fri, Oct 25, 2019 at 

Re: Pipeline AttributeError on Python3

[Valentyn Tymofieiev]

+user@, bcc: dev@
https://issues.apache.org/jira/browse/BEAM-6158 may be contributing to this
issue, although we saw instances of this bug in exactly opposite scenarios
- when pipeline was defined *in one file*, but not in multiple files.

Could you try replacing instances of super() in aggregation_transform.py
as done in https://github.com/apache/beam/pull/9513 and see if this issue
is still reproducible?

If that doesn't work, I would try to get the dump of serialized_fn, and try
to reproduce the issue in isolated environment, such as:

form apache_beam.internal import pickler
serialized_fn = "..content.."
pickler.loads(serialized_fn)

then I would try to trim the doFn in the example to a
minimally-reproducible example. It could be another issue with dill
dependency.


On Mon, Oct 28, 2019 at 2:48 PM Rakesh Kumar  wrote:

> Hi All,
>
> We have noticed a weird intermittent issue on Python3 but we don't run
> into this issue on python2. Sometimes when we are trying to submit the
> pipeline, we get AttributeError (Check the stack trace below).  we have
> double-checked and we do find the attribute/methods are present in the
> right module and in right place but somehow the pipeline still complains
> about it. In some cases, we refer methods before their definition. We tried
> to reorder the method definition but that didn't help at all.
>
> We don't see the same issue when the entire pipeline is defined in one
> file. Also, note that this doesn't happen all the time when we submit the
> pipeline, so I feel it is some kind of race condition. When we enable the
> worker recycle logic it happens most of the time when sdk worker is
> recycled.
>
> Some more information about the environment:
> Python version: 3
> Beam version: 2.16
> Flink version: 1.8
>
> *Stack trace: *
>
>- :
>
> TimerException{java.lang.RuntimeException: Failed to finish remote bundle}
> at
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$RepeatedTriggerTask.run(SystemProcessingTimeService.java:335)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Failed to finish remote bundle
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner.finishBundle(ExecutableStageDoFnOperator.java:667)
> at
> org.apache.beam.runners.core.StatefulDoFnRunner.finishBundle(StatefulDoFnRunner.java:144)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$2.finishBundle(ExecutableStageDoFnOperator.java:754)
> at
> org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:86)
> at
> org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:118)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:750)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.checkInvokeFinishBundleByTime(DoFnOperator.java:744)
> at
> org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.lambda$open$1(DoFnOperator.java:460)
> at
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$RepeatedTriggerTask.run(SystemProcessingTimeService.java:330)
> ... 7 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error received from SDK harness for instruction
> 6: Traceback (most recent call last):
>   File
> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 307, in get
> processor = self.cached_bundle_processors[bundle_descriptor_id].pop()
> IndexError: pop from empty list
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>   File
> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/apache_beam/internal/pickler.py",
> line 261, in loads
> return dill.loads(s)
>   File
> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/dill/_dill.py",
> line 317, in loads
> return load(file, ignore)
>   File
> "/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/dill/_dill.py",
> line 305, in load
> obj = pik.load()
>   File
> 

Re: RFC: python static typing PR

[Valentyn Tymofieiev]

Thanks a lot, Chad. Looking at the PR, I am incredibly happy to see
explicit type annotations throughout Beam codebase. I believe this is a
step in the right direction even if the tooling were not able to do any
inference at all. The effort required from developers to add annotations in
their code changes will be well compensated by reducing cognitive load
required to read the codebase, and will lower the entry barrier for new
contributions. Lack of docstrings in Beam internals in Beam codebase,
especially the lack of typing information, has repeatedly come up as a
source of frustration among several folks with whom I work.

As Beam devs will be gaining more first-hand experience with the tooling,
we may need to add a style guide/best practices/FAQ to our contributor
guide to clarify known issues.

Happy to help with reviewing individual commits once we have a merge plan
in place.

On Mon, Oct 28, 2019 at 2:41 PM Robert Bradshaw  wrote:

> Thanks, Chad, this has been a herculean task. I'm excited for the
> additional tooling and documentation explicit types can bring to our
> code, even if tooling such as mypy isn't able to do as much inference
> for obvious cases as I would like.
>
> This will, of course, put another burden on developers in contributing
> code, similar to lint and writing unit tests. IMHO this will be worth
> the cost in terms of encouraging better code, but I think it's
> important to establish consensus if this is the direction we're
> moving. So if you have any thoughts or questions, positive or
> negative, please comment.
>
>
> On Mon, Oct 28, 2019 at 10:34 AM Chad Dombrova  wrote:
> >
> > Hi all,
> > I've been working on a PR to add static typing to the beam python sdk
> for the past 4 months or so.  This has been an epic journey which has
> required chasing down numerous fixes across several other projects (mypy,
> pylint, python-future), but the mypy tests are now passing!
> >
> > I'm not sure how much convincing I need to do with this group on the
> benefits of static typing, especially considering the heavy Java influence
> on this project, but for those who are curious, there's some good info
> here:  https://realpython.com/python-type-checking/#pros-and-cons.
> Suffice it to say, it's a game changer for projects like Beam (large code
> base, high level of quality, good testing infrastructure), and it dovetails
> perfectly into the efforts surrounding pipeline type analysis and
> serialization.  Anecdotally speaking, all of the developers I've worked
> with who were originally resistant to static typing in python ("it's
> ugly!"  "it's not pythonic!") have changed their tune and fully embraced it
> because of the time that it saves them every day.
> >
> > More details over at the PR:  https://github.com/apache/beam/pull/9056
> >
> > I look forward to your feedback!
> >
> > -chad
> >
>

Re: Rethinking the Flink Runner modes

[Kyle Weaver]

Filed https://issues.apache.org/jira/browse/BEAM-8507 for the issue I
mentioned.

On Mon, Oct 28, 2019 at 4:12 PM Kyle Weaver  wrote:

> > I'd like to see this issue resolved before 2.17 as changing the public
> API once it's released will be harder.
>
> +1. In particular, I misunderstood that [auto] is not supported by
> `FlinkUberJarJobServer`. Since [auto] is now the default, it's broken for
> Python 3.6+.
>
> requests.exceptions.InvalidURL: Failed to parse: [auto]/v1/config
>
> We definitely should fix that, if nothing else.
>
> > One concern with this is that just supplying host:port is the existing
> > behavior, so we can't start requiring the http://.
>
> The user shouldn't have to specify a protocol for Python, I think it's
> preferable and reasonable to handle that for them in order to maintain
> existing behavior and align with Java SDK.
>
> > 2. Deprecate the "[auto]" and "[local]" values. It should be sufficient
> > to have either a non-empty address string or an empty one. The empty
> > string would either mean local execution or, in the context of the Flink
> > CLI tool, loading the master address from the config. The non-empty
> > string would be interpreted as a cluster address.
>
> Looks like we also have a [collection] configuration value [1].
>
> If we're using [auto] as the default, I don't think this really makes so
> much of a difference (as long as we're supporting and documenting these
> properly, of course). I'm not sure there's a compelling reason to change
> this?
>
> > always run locally (the least surprising to me
>
> I agree a local cluster should remain the default, whether that is
> achieved through [local] or [auto] or some new mechanism such as the above.
>
> [1]
> https://github.com/apache/beam/blob/2f1b56ccc506054e40afe4793a8b556e872e1865/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L80
>
> On Mon, Oct 28, 2019 at 6:35 AM Maximilian Michels  wrote:
>
>> Hi,
>>
>> Robert and Kyle have been doing great work to simplify submitting
>> portable pipelines with the Flink Runner. Part of this is having a
>> Python "FlinkRunner" which handles bringing up a Beam job server and
>> submitting the pipeline directly via the Flink REST API. One building
>> block is the creation of "executable Jars" which contain the
>> materialized / translated Flink pipeline and do not require the Beam job
>> server or the Python driver anymore.
>>
>> While unifying a newly introduced option "flink_master_url" with the
>> pre-existing "flink_master" [1][2], some questions came up about Flink's
>> execution modes. (The two options are meant to do the same thing:
>> provide the address of the Flink master to hand-over the translated
>> pipeline.)
>>
>> Historically, Flink had a proprietary protocol for submitting pipelines,
>> running on port 9091. This has since been replaced with a REST protocol
>> at port 8081. To this date, this has implications how you submit
>> programs, e.g. the Flink client libraries expects the address to be of
>> form "host:port", without a protocol scheme. On the other hand, external
>> Rest libraries typically expect a protocol scheme.
>>
>> But this is only half of the fun. There are also special addresses for
>> "flink_master" that influence submission of the pipeline. If you specify
>> "[local]" as the address, the pipeline won't be submitted but executed
>> in a local in-process Flink cluster. If you specify "[auto]" and you use
>> the CLI tool that comes bundled with Flink, then the master address will
>> be loaded from the Flink config, including any configuration like SSL.
>> If none is found, then it falls back to "[local]".
>>
>> This is a bit odd, and after a discussion with Robert and Thomas in [1],
>> we figured that this needs to be changed:
>>
>> 1. Make the master address a URL. Add "http://; to "flink_master" in
>> Python if no scheme is specified. Similarly, remove any "http://; in
>> Java, since the Java rest client does not expect a scheme. In case of
>> "http_s_://", we have a special treatment to load the SSL settings from
>> the Flink config.
>>
>> 2. Deprecate the "[auto]" and "[local]" values. It should be sufficient
>> to have either a non-empty address string or an empty one. The empty
>> string would either mean local execution or, in the context of the Flink
>> CLI tool, loading the master address from the config. The non-empty
>> string would be interpreted as a cluster address.
>>
>>
>> Any opinions on this?
>>
>>
>> Thanks,
>> Max
>>
>>
>> [1] https://github.com/apache/beam/pull/9803
>> [2] https://github.com/apache/beam/pull/9844
>>
>

Re: Rethinking the Flink Runner modes

[Kyle Weaver]

> I'd like to see this issue resolved before 2.17 as changing the public
API once it's released will be harder.

+1. In particular, I misunderstood that [auto] is not supported by
`FlinkUberJarJobServer`. Since [auto] is now the default, it's broken for
Python 3.6+.

requests.exceptions.InvalidURL: Failed to parse: [auto]/v1/config

We definitely should fix that, if nothing else.

> One concern with this is that just supplying host:port is the existing
> behavior, so we can't start requiring the http://.

The user shouldn't have to specify a protocol for Python, I think it's
preferable and reasonable to handle that for them in order to maintain
existing behavior and align with Java SDK.

> 2. Deprecate the "[auto]" and "[local]" values. It should be sufficient
> to have either a non-empty address string or an empty one. The empty
> string would either mean local execution or, in the context of the Flink
> CLI tool, loading the master address from the config. The non-empty
> string would be interpreted as a cluster address.

Looks like we also have a [collection] configuration value [1].

If we're using [auto] as the default, I don't think this really makes so
much of a difference (as long as we're supporting and documenting these
properly, of course). I'm not sure there's a compelling reason to change
this?

> always run locally (the least surprising to me

I agree a local cluster should remain the default, whether that is achieved
through [local] or [auto] or some new mechanism such as the above.

[1]
https://github.com/apache/beam/blob/2f1b56ccc506054e40afe4793a8b556e872e1865/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkExecutionEnvironments.java#L80

On Mon, Oct 28, 2019 at 6:35 AM Maximilian Michels  wrote:

> Hi,
>
> Robert and Kyle have been doing great work to simplify submitting
> portable pipelines with the Flink Runner. Part of this is having a
> Python "FlinkRunner" which handles bringing up a Beam job server and
> submitting the pipeline directly via the Flink REST API. One building
> block is the creation of "executable Jars" which contain the
> materialized / translated Flink pipeline and do not require the Beam job
> server or the Python driver anymore.
>
> While unifying a newly introduced option "flink_master_url" with the
> pre-existing "flink_master" [1][2], some questions came up about Flink's
> execution modes. (The two options are meant to do the same thing:
> provide the address of the Flink master to hand-over the translated
> pipeline.)
>
> Historically, Flink had a proprietary protocol for submitting pipelines,
> running on port 9091. This has since been replaced with a REST protocol
> at port 8081. To this date, this has implications how you submit
> programs, e.g. the Flink client libraries expects the address to be of
> form "host:port", without a protocol scheme. On the other hand, external
> Rest libraries typically expect a protocol scheme.
>
> But this is only half of the fun. There are also special addresses for
> "flink_master" that influence submission of the pipeline. If you specify
> "[local]" as the address, the pipeline won't be submitted but executed
> in a local in-process Flink cluster. If you specify "[auto]" and you use
> the CLI tool that comes bundled with Flink, then the master address will
> be loaded from the Flink config, including any configuration like SSL.
> If none is found, then it falls back to "[local]".
>
> This is a bit odd, and after a discussion with Robert and Thomas in [1],
> we figured that this needs to be changed:
>
> 1. Make the master address a URL. Add "http://; to "flink_master" in
> Python if no scheme is specified. Similarly, remove any "http://; in
> Java, since the Java rest client does not expect a scheme. In case of
> "http_s_://", we have a special treatment to load the SSL settings from
> the Flink config.
>
> 2. Deprecate the "[auto]" and "[local]" values. It should be sufficient
> to have either a non-empty address string or an empty one. The empty
> string would either mean local execution or, in the context of the Flink
> CLI tool, loading the master address from the config. The non-empty
> string would be interpreted as a cluster address.
>
>
> Any opinions on this?
>
>
> Thanks,
> Max
>
>
> [1] https://github.com/apache/beam/pull/9803
> [2] https://github.com/apache/beam/pull/9844
>

Re: Python Precommit duration pushing 2 hours

[Pablo Estrada]

I have written https://github.com/apache/beam/pull/9910 to reduce
FnApiRunnerTest variations.
I'm not in a rush to merge, but rather happy to start a discussion.
I'll also try to figure out if there are other tests slowing down the suite
significantly.
Best
-P.

On Fri, Oct 25, 2019 at 7:41 PM Valentyn Tymofieiev 
wrote:

> Thanks, Brian.
> +Udi Meiri 
> As next step, it would be good to know whether slowdown is caused by tests
> in this PR, or its effect on other tests, and to confirm that only Python 2
> codepaths were affected.
>
> On Fri, Oct 25, 2019 at 6:35 PM Brian Hulette  wrote:
>
>> I did a bisect based on the runtime of `./gradlew
>> :sdks:python:test-suites:tox:py2:testPy2Gcp` around the commits between 9/1
>> and 9/15 to see if I could find the source of the spike that happened
>> around 9/6. It looks like it was due to PR#9283 [1]. I thought maybe this
>> search would reveal some mis-guided configuration change, but as far as I
>> can tell 9283 just added a well-tested feature. I don't think there's
>> anything to learn from that... I just wanted to circle back about it in
>> case others are curious about that spike.
>>
>> I'm +1 on bumping some FnApiRunner configurations.
>>
>> Brian
>>
>> [1] https://github.com/apache/beam/pull/9283
>>
>> On Fri, Oct 25, 2019 at 4:49 PM Pablo Estrada  wrote:
>>
>>> I think it makes sense to remove some of the extra FnApiRunner
>>> configurations. Perhaps some of the multiworkers and some of the grpc
>>> versions?
>>> Best
>>> -P.
>>>
>>> On Fri, Oct 25, 2019 at 12:27 PM Robert Bradshaw 
>>> wrote:
>>>
 It looks like fn_api_runner_test.py is quite expensive, taking 10-15+
 minutes on each version of Python. This test consists of a base class
 that is basically a validates runner suite, and is then run in several
 configurations, many more of which (including some expensive ones)
 have been added lately.

 class FnApiRunnerTest(unittest.TestCase):
 class FnApiRunnerTestWithGrpc(FnApiRunnerTest):
 class FnApiRunnerTestWithGrpcMultiThreaded(FnApiRunnerTest):
 class FnApiRunnerTestWithDisabledCaching(FnApiRunnerTest):
 class FnApiRunnerTestWithMultiWorkers(FnApiRunnerTest):
 class FnApiRunnerTestWithGrpcAndMultiWorkers(FnApiRunnerTest):
 class FnApiRunnerTestWithBundleRepeat(FnApiRunnerTest):
 class FnApiRunnerTestWithBundleRepeatAndMultiWorkers(FnApiRunnerTest):

 I'm not convinced we need to run all of these permutations, or at
 least not all tests in all permutations.

 On Fri, Oct 25, 2019 at 10:57 AM Valentyn Tymofieiev
  wrote:
 >
 > I took another look at this and precommit ITs are already running in
 parallel, albeit in the same suite. However it appears Python precommits
 became slower, especially Python 2 precommits [35 min per suite x 3
 suites], see [1]. Not sure yet what caused the increase, but precommits
 used to be faster. Perhaps we have added a slow test or a lot of new tests.
 >
 > [1]
 https://scans.gradle.com/s/jvcw5fpqfc64k/timeline?task=ancsbov425524
 >
 > On Thu, Oct 24, 2019 at 4:53 PM Ahmet Altay  wrote:
 >>
 >> Ack. Separating precommit ITs to a different suite sounds good.
 Anyone is interested in doing that?
 >>
 >> On Thu, Oct 24, 2019 at 2:41 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:
 >>>
 >>> This should not increase the queue time substantially, since
 precommit ITs are running sequentially with precommit tests, unlike
 multiple precommit tests which run in parallel to each other.
 >>>
 >>> The precommit ITs we run are batch and streaming wordcount tests on
 Py2 and one Py3 version, so it's not a lot of tests.
 >>>
 >>> On Thu, Oct 24, 2019 at 1:07 PM Ahmet Altay 
 wrote:
 
  +1 to separating ITs from precommit. Downside would be, when Chad
 tried to do something similar [1] it was noted that the total time to run
 all precommit tests would increase and also potentially increase the queue
 time.
 
  Another alternative, we could run a smaller set of IT tests in
 precommits and run the whole suite as part of post commit tests.
 
  [1] https://github.com/apache/beam/pull/9642
 
  On Thu, Oct 24, 2019 at 12:15 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:
 >
 > One improvement could be move to Precommit IT tests into a
 separate suite from precommit tests, and run it in parallel.
 >
 > On Thu, Oct 24, 2019 at 11:41 AM Brian Hulette <
 bhule...@google.com> wrote:
 >>
 >> Python Precommits are taking quite a while now [1]. Just
 visually it looks like the average length is 1.5h or so, but it spikes up
 to 2h. I've had several precommit runs get aborted due to the 2 hour limit.
 >>
 >> It looks like there was a spike up above 1h back on 9/6 and the
 duration has 

Re: Quota issues again

[Mikhail Gryzykhin]

Quota jira issue:
https://issues.apache.org/jira/browse/BEAM-8195

On Mon, Oct 28, 2019 at 2:05 PM Mikhail Gryzykhin  wrote:

> Hi everyone,
>
>
> While validating release branch, I got failure due Quota again. Also, current 
> queue time for jobs is more than 1.5 hours.
>
>
> I'm not sure if it is worth starting another thread on tests efficiency, but 
> still want to keep this mail to highlight the issues.
>
>
> See PS for links.
>
>
> Regards,
>
> --Mikhail
>
>
> PS:
>
> https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull
>
> *13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed: googleapi: 
> Error 429: Quota exceeded for quota metric 
> 'dataflow.googleapis.com/create_requests' and limit 
> 'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com' for 
> consumer 'project_number:844138762903'., rateLimitExceeded
>
>
> Queue time:
>
> http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1
>
>

Re: Rethinking the Flink Runner modes

[Robert Bradshaw]

Thanks for bringing this to the list. Some comments below, though it
would be good to get additional feedback beyond those that have been
participating on the PR, if any. I'd like to see this issue resolved
before 2.17 as changing the public API once it's released will be
harder.

On Mon, Oct 28, 2019 at 6:36 AM Maximilian Michels  wrote:
>
> Hi,
>
> Robert and Kyle have been doing great work to simplify submitting
> portable pipelines with the Flink Runner. Part of this is having a
> Python "FlinkRunner" which handles bringing up a Beam job server and
> submitting the pipeline directly via the Flink REST API. One building
> block is the creation of "executable Jars" which contain the
> materialized / translated Flink pipeline and do not require the Beam job
> server or the Python driver anymore.
>
> While unifying a newly introduced option "flink_master_url" with the
> pre-existing "flink_master" [1][2], some questions came up about Flink's
> execution modes. (The two options are meant to do the same thing:
> provide the address of the Flink master to hand-over the translated
> pipeline.)
>
> Historically, Flink had a proprietary protocol for submitting pipelines,
> running on port 9091. This has since been replaced with a REST protocol
> at port 8081. To this date, this has implications how you submit
> programs, e.g. the Flink client libraries expects the address to be of
> form "host:port", without a protocol scheme. On the other hand, external
> Rest libraries typically expect a protocol scheme.
>
> But this is only half of the fun. There are also special addresses for
> "flink_master" that influence submission of the pipeline. If you specify
> "[local]" as the address, the pipeline won't be submitted but executed
> in a local in-process Flink cluster. If you specify "[auto]" and you use
> the CLI tool that comes bundled with Flink, then the master address will
> be loaded from the Flink config, including any configuration like SSL.
> If none is found, then it falls back to "[local]".
>
> This is a bit odd, and after a discussion with Robert and Thomas in [1],
> we figured that this needs to be changed:
>
> 1. Make the master address a URL. Add "http://; to "flink_master" in
> Python if no scheme is specified. Similarly, remove any "http://; in
> Java, since the Java rest client does not expect a scheme. In case of
> "http_s_://", we have a special treatment to load the SSL settings from
> the Flink config.

One concern with this is that just supplying host:port is the existing
behavior, so we can't start requiring the http://. The question
becomes whether we should allow it, or infer http[s] (e.g. by trying
out https first). One question I have is if there are other
authentication parameters that may be required for speaking to a flink
endpoint that we should be aware of (that would normally be buried in
the config file).

> 2. Deprecate the "[auto]" and "[local]" values. It should be sufficient
> to have either a non-empty address string or an empty one. The empty
> string would either mean local execution or, in the context of the Flink
> CLI tool, loading the master address from the config. The non-empty
> string would be interpreted as a cluster address.

I do like being explicit with something like [local] rather than
treating the empty string in a magical way. Perhaps we could have
[config] be an alias for [auto] indicating to read the config to get
the parameter (if any). The tricky bit it seems is that if running as
CLI (where, as I understand it, the jar is then executed on the
cluster) one does not want to run it truly locally but on that
cluster.

For non-CLI, the two options for default are to always run locally
(the least surprising to me, but it'd be good to get feedback from
others) or to read the config file and submit it to the value there,
if any.

> Any opinions on this?
>
>
> Thanks,
> Max
>
>
> [1] https://github.com/apache/beam/pull/9803
> [2] https://github.com/apache/beam/pull/9844

Re: [Question] Cannot resolve symbol 'AutoValue_KafkaIO_WriteRecords'

[Kirill Kozlov]

I found this document [1] helpful when setting up an Intellij to work with
Beam.
Make sure that (File | Settings | Build, Execution, Deployment | Build
Tools | Gradle | Runner) has a "Delegate IDE build/run actions to Gradle"
checked and "Run test using" is set to "Gradle Test Runner".
Under (Project Structure | Problems) there should be no import problems, if
there are - try reimporting a project from scratch.
Also, try running the following target in the beam folder: ./gradlew idea
Hope that helps.

[1]
https://docs.google.com/document/d/18eXrO9IYll4oOnFb53EBhOtIfx-JLOinTWZSIBFkLk4/edit

On Sun, Oct 27, 2019 at 7:09 AM lan.liang  wrote:

> Hi Team:
> I Open Beam Project in IDEA, But it not working.
>
> Looks like there are some files missing,
>
> In org.apache.beam.sdk.io.kafka.KafkaIO,i just got err:
>
> 
>Cannot resolve symbol 'AutoValue_KafkaIO_WriteRecords'
>
>Cannot resolve symbol 'AutoValue_KafkaIO_Write'
>
> 
>
>  If i missed something,remind me please.
>
> Thanks!
>
>
>
>
> - lan.liang
>

Pipeline AttributeError on Python3

[Rakesh Kumar]

Hi All,

We have noticed a weird intermittent issue on Python3 but we don't run into
this issue on python2. Sometimes when we are trying to submit the pipeline,
we get AttributeError (Check the stack trace below).  we have
double-checked and we do find the attribute/methods are present in the
right module and in right place but somehow the pipeline still complains
about it. In some cases, we refer methods before their definition. We tried
to reorder the method definition but that didn't help at all.

We don't see the same issue when the entire pipeline is defined in one
file. Also, note that this doesn't happen all the time when we submit the
pipeline, so I feel it is some kind of race condition. When we enable the
worker recycle logic it happens most of the time when sdk worker is
recycled.

Some more information about the environment:
Python version: 3
Beam version: 2.16
Flink version: 1.8

*Stack trace: *

   - :

TimerException{java.lang.RuntimeException: Failed to finish remote bundle}
at
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$RepeatedTriggerTask.run(SystemProcessingTimeService.java:335)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to finish remote bundle
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner.finishBundle(ExecutableStageDoFnOperator.java:667)
at
org.apache.beam.runners.core.StatefulDoFnRunner.finishBundle(StatefulDoFnRunner.java:144)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$2.finishBundle(ExecutableStageDoFnOperator.java:754)
at
org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:86)
at
org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:118)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:750)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.checkInvokeFinishBundleByTime(DoFnOperator.java:744)
at
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.lambda$open$1(DoFnOperator.java:460)
at
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$RepeatedTriggerTask.run(SystemProcessingTimeService.java:330)
... 7 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: Error received from SDK harness for instruction
6: Traceback (most recent call last):
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 307, in get
processor = self.cached_bundle_processors[bundle_descriptor_id].pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/apache_beam/internal/pickler.py",
line 261, in loads
return dill.loads(s)
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/dill/_dill.py",
line 317, in loads
return load(file, ignore)
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/dill/_dill.py",
line 305, in load
obj = pik.load()
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/dill/_dill.py",
line 474, in find_class
return StockUnpickler.find_class(self, module, name)
*AttributeError: Can't get attribute '_timestamp_keyed_result' on *

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 165, in _execute
response = task()
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 198, in 
self._execute(lambda: worker.do_instruction(work), work)
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 351, in do_instruction
request.instruction_id)
  File
"/srv/venvs/service/trusty/service_venv_python3.6/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py",
line 371, in process_bundle

Re: RFC: python static typing PR

[Robert Bradshaw]

Thanks, Chad, this has been a herculean task. I'm excited for the
additional tooling and documentation explicit types can bring to our
code, even if tooling such as mypy isn't able to do as much inference
for obvious cases as I would like.

This will, of course, put another burden on developers in contributing
code, similar to lint and writing unit tests. IMHO this will be worth
the cost in terms of encouraging better code, but I think it's
important to establish consensus if this is the direction we're
moving. So if you have any thoughts or questions, positive or
negative, please comment.


On Mon, Oct 28, 2019 at 10:34 AM Chad Dombrova  wrote:
>
> Hi all,
> I've been working on a PR to add static typing to the beam python sdk for the 
> past 4 months or so.  This has been an epic journey which has required 
> chasing down numerous fixes across several other projects (mypy, pylint, 
> python-future), but the mypy tests are now passing!
>
> I'm not sure how much convincing I need to do with this group on the benefits 
> of static typing, especially considering the heavy Java influence on this 
> project, but for those who are curious, there's some good info here:  
> https://realpython.com/python-type-checking/#pros-and-cons.  Suffice it to 
> say, it's a game changer for projects like Beam (large code base, high level 
> of quality, good testing infrastructure), and it dovetails perfectly into the 
> efforts surrounding pipeline type analysis and serialization.  Anecdotally 
> speaking, all of the developers I've worked with who were originally 
> resistant to static typing in python ("it's ugly!"  "it's not pythonic!") 
> have changed their tune and fully embraced it because of the time that it 
> saves them every day.
>
> More details over at the PR:  https://github.com/apache/beam/pull/9056
>
> I look forward to your feedback!
>
> -chad
>

Quota issues again

[Mikhail Gryzykhin]

Hi everyone,


While validating release branch, I got failure due Quota again. Also,
current queue time for jobs is more than 1.5 hours.


I'm not sure if it is worth starting another thread on tests
efficiency, but still want to keep this mail to highlight the issues.


See PS for links.


Regards,

--Mikhail


PS:

https://builds.apache.org/job/beam_PostCommit_Go_PR/71/consoleFull

*13:46:25* 2019/10/28 20:46:25 Test wordcount:kinglear failed:
googleapi: Error 429: Quota exceeded for quota metric
'dataflow.googleapis.com/create_requests' and limit
'CreateRequestsPerMinutePerUser' of service 'dataflow.googleapis.com'
for consumer 'project_number:844138762903'., rateLimitExceeded


Queue time:

http://metrics.beam.apache.org/d/_TNndF2iz/pre-commit-test-latency?orgId=1

Re: [UPDATE] Preparing for Beam 2.17.0 release

[Ahmet Altay]

On Mon, Oct 28, 2019 at 12:44 PM Gleb Kanterov  wrote:

> It looks like BigQueryIO DIRECT_READ is broken since 2.16.0, I've added a
> ticket describing the problem and possible fix, see BEAM-8504
>  [1].
>

Should this be added to 2.16 blog post as a known issue?


>
> [1]: https://issues.apache.org/jira/browse/BEAM-8504
>
> On Wed, Oct 23, 2019 at 9:19 PM Kenneth Knowles  wrote:
>
>> I opened https://github.com/apache/beam/pull/9862 to raise the
>> documentation of Fix Version to the top level. It also includes the write
>> up of Jira priorities, to make clear that "Blocker" priority does not refer
>> to release blocking.
>>
>> On Wed, Oct 23, 2019 at 11:16 AM Kenneth Knowles  wrote:
>>
>>> I've gone over the tickets and removed Fix Version from many of them
>>> that do not seem to be critical defects. If I removed Fix Version from a
>>> ticket you care about, please feel free to add it back. I am not trying to
>>> decide what is in/out of the release, just trying to triage the Jira data
>>> to match expected practices.
>>>
>>> It should probably be documented somewhere outside of the release guide.
>>> As far as I can tell, the fact that we triage them down to zero is the only
>>> place we mention that it is used to indicate release blockers and not used
>>> for feature targets.
>>>
>>> Kenn
>>>
>>> On Wed, Oct 23, 2019 at 10:40 AM Kenneth Knowles 
>>> wrote:
>>>
  Wow, 28 release blocking tickets! That is the most I've ever seen, by
 far. Many appear to be feature requests, not release-blocking defects. I
 believe this is not according to our normal best practice. The release
 cadence should not wait for features in progress, with exceptions discussed
 on dev@. As a matter of best practice, I think we should triage
 feature requests to not have Fix Version set until it has been discussed on
 dev@.

 Kenn

 On Wed, Oct 23, 2019 at 9:55 AM Mikhail Gryzykhin 
 wrote:

> Hi all,
>
> Beam 2.17 release branch cut is scheduled today (2019/10/23)
> according to the release calendar [1].  I'll start working on the
> branch cutoff and later work on cherry picking blocker fixes.
>
> If you have release blocking issues for 2.17 please mark their "Fix
> Version" as 2.17.0 [2]. This tag is already created in JIRA in case you
> would like to move any non-blocking issues to that version.
>
> There is a decent amount of open bugs to be resolved in 2.17.0 [2] and
> only 4 [3] are marked as blockers. Please, review those if these bugs are
> actually to be resolved in 2.17.0 and prioritize fixes if possible.
>
> Any thoughts, comments, objections?
>
> Regards.
> Mikhail.
>
>
> [1]
> https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
> [2]
> https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.17.0
> [3]
> https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%202.17.0
>

Re: [UPDATE] Preparing for Beam 2.17.0 release

[Gleb Kanterov]

It looks like BigQueryIO DIRECT_READ is broken since 2.16.0, I've added a
ticket describing the problem and possible fix, see BEAM-8504
 [1].

[1]: https://issues.apache.org/jira/browse/BEAM-8504

On Wed, Oct 23, 2019 at 9:19 PM Kenneth Knowles  wrote:

> I opened https://github.com/apache/beam/pull/9862 to raise the
> documentation of Fix Version to the top level. It also includes the write
> up of Jira priorities, to make clear that "Blocker" priority does not refer
> to release blocking.
>
> On Wed, Oct 23, 2019 at 11:16 AM Kenneth Knowles  wrote:
>
>> I've gone over the tickets and removed Fix Version from many of them that
>> do not seem to be critical defects. If I removed Fix Version from a ticket
>> you care about, please feel free to add it back. I am not trying to decide
>> what is in/out of the release, just trying to triage the Jira data to match
>> expected practices.
>>
>> It should probably be documented somewhere outside of the release guide.
>> As far as I can tell, the fact that we triage them down to zero is the only
>> place we mention that it is used to indicate release blockers and not used
>> for feature targets.
>>
>> Kenn
>>
>> On Wed, Oct 23, 2019 at 10:40 AM Kenneth Knowles  wrote:
>>
>>>  Wow, 28 release blocking tickets! That is the most I've ever seen, by
>>> far. Many appear to be feature requests, not release-blocking defects. I
>>> believe this is not according to our normal best practice. The release
>>> cadence should not wait for features in progress, with exceptions discussed
>>> on dev@. As a matter of best practice, I think we should triage feature
>>> requests to not have Fix Version set until it has been discussed on dev@
>>> .
>>>
>>> Kenn
>>>
>>> On Wed, Oct 23, 2019 at 9:55 AM Mikhail Gryzykhin 
>>> wrote:
>>>
 Hi all,

 Beam 2.17 release branch cut is scheduled today (2019/10/23) according
 to the release calendar [1].  I'll start working on the branch cutoff
 and later work on cherry picking blocker fixes.

 If you have release blocking issues for 2.17 please mark their "Fix
 Version" as 2.17.0 [2]. This tag is already created in JIRA in case you
 would like to move any non-blocking issues to that version.

 There is a decent amount of open bugs to be resolved in 2.17.0 [2] and
 only 4 [3] are marked as blockers. Please, review those if these bugs are
 actually to be resolved in 2.17.0 and prioritize fixes if possible.

 Any thoughts, comments, objections?

 Regards.
 Mikhail.


 [1]
 https://calendar.google.com/calendar/embed?src=0p73sl034k80oob7seouanigd0%40group.calendar.google.com
 [2]
 https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20fixVersion%20%3D%202.17.0
 [3]
 https://issues.apache.org/jira/browse/BEAM-8457?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Reopened%2C%20Open%2C%20%22In%20Progress%22%2C%20%22Under%20Discussion%22%2C%20%22In%20Implementation%22%2C%20%22Triage%20Needed%22)%20AND%20priority%20%3D%20Blocker%20AND%20fixVersion%20%3D%202.17.0

>>>

Re: Is there good way to make Python SDK docs draft accessible?

[Udi Meiri]

I believe that generating pydoc for the website is still a manual process
(unlike the rest of the website?).
The reviewer will need to manually generate the docs (checkout the PR, run
tox -e docs).

On Mon, Oct 28, 2019 at 10:55 AM Yoshiki Obata 
wrote:

> Hi all.
>
> I'm working on enabling to generate Python SDK docs with Python3 [1]
> I have modified scripts and now reviewing generated docs in someone’s
> eyes is needed.
>
> But there seems to be no existing way to upload generated docs to
> where anyone can access unlike website html which can be uploaded to
> GCS via Jenkins job.
> Would anyone know good way to make generated docs accessible for
> anyone for convenience of reviewing them?
>
> [1] https://issues.apache.org/jira/browse/BEAM-7847
>
>
> Best regards,
> Yoshiki
>
>
> --
> Yoshiki Obata
> mail: yoshiki.ob...@gmail.com
> gh: https://github.com/lazylynx
>


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [Discuss] Ideas for Apache Beam presence in social media

[Aizhamal Nurmamat kyzy]

Thank you Matthias,

I was supposed to write up the documentation.. sorry this got slipped
through the cracks. I will prepare the PR until the end of the week.

On Tue, Oct 22, 2019, 12:51 AM Matthias Baetens 
wrote:

> Thanks Thomas.
>
> Happy to help on the doc side when I find some time :) I'll give you a
> ping when I have the PR ready!
>
> On Mon, Oct 14, 2019, 20:07 Thomas Weise  wrote:
>
>> Matthias,
>>
>> The process is already in use, but it would be nice to have it documented
>> also.
>>
>> I gave you edit access to the spreadsheet, since working with the
>> comments is cumbersome and sheets does not have suggestions.
>>
>> Thanks
>>
>>
>> On Fri, Oct 11, 2019 at 11:59 PM Matthias Baetens <
>> baetensmatth...@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> Picking up this thread, since I wanted to use this facility and help
>>> drive this if necessary.
>>>
>>> I saw the sheet has now comment access enabled. Did we decide / document
>>> the desired process on the website? I am happy to testdrive that process
>>> and submit a PR if successful.
>>>
>>> Many thanks,
>>> Matthias
>>>
>>> On Tue, 13 Aug 2019 at 01:49, Thomas Weise  wrote:
>>>
 Yes, everyone should have comment access for this to make sense. Sorry
 for the confusion.


 On Mon, Aug 12, 2019 at 5:30 PM Kenneth Knowles 
 wrote:

> Thanks for setting this up. It is nice to start building up a system
> for this so everyone can participate.
>
> Regarding Jira versus notifications, how are people with only view
> access to make suggestions for tweets? When I suggested gdocs, I meant for
> everyone to have "comment" access, so then anyone can subscribe to all
> comments, which would include suggestions. This allows anyone to suggest
> tweets and anyone to subscribe to suggestions.
>
> Kenn
>
> On Wed, Aug 7, 2019 at 4:07 PM Aizhamal Nurmamat kyzy <
> aizha...@google.com> wrote:
>
>> Thanks Thomas, changed the doc to view only and granted you and Ahmet
>> edit access.
>> @all - please send requests for access with your google accounts. I
>> will update the thread once I document the process and submit the PR to 
>> the
>> website.
>>
>> Thank you,
>> Aizhamal
>>
>> On Wed, Aug 7, 2019 at 3:12 PM Thomas Weise  wrote:
>>
>>> I was able to subscribe now.
>>>
>>> Reminder for others that the spreadsheet of interest can be found
>>> here: s.apache.org/beam-tweets
>>>
>>> Aizhamal,
>>>
>>> Can you help with a couple changes to bring this closer to how
>>> similar gdoc resources are handled?
>>>
>>> * Make the document view only. *PMC members* that care to help with
>>> this can request edit access.
>>> * Document the process for other contributors. Maybe here?
>>> https://beam.apache.org/contribute/
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Wed, Aug 7, 2019 at 2:39 PM Ahmet Altay  wrote:
>>>
 I am able to subscribe to notifications now. Thomas does it work
 for you?

 On Wed, Aug 7, 2019 at 2:23 PM Aizhamal Nurmamat kyzy <
 aizha...@apache.org> wrote:

> Hi all,
>
> I set the access to 'anyone can edit'. Let me know if
> notifications work now.
>
> Thanks,
>
> On Wed, Aug 7, 2019 at 2:00 PM Ahmet Altay 
> wrote:
>
>> You are probably right and it is an access issue.
>>
>> Aizhamal, could you give us edit access? And we can see if
>> notifications work after that.
>>
>> On Wed, Aug 7, 2019 at 1:41 PM Thomas Weise 
>> wrote:
>>
>>> The use of JIRA was also suggested before, but why do the
>>> notifications not work? I wasn't able to subscribe and I suspect 
>>> that was
>>> due to not having sufficient access to the spreadsheet?
>>>
>>>
>>>
>>> On Wed, Aug 7, 2019 at 1:26 PM Ahmet Altay 
>>> wrote:
>>>
 As far as I understand we have not resolved this discussion and
 the sticking issue is that there is no good way of subscribing to 
 changes
 (i.e. proposals for tweets) for interested parties. The method 
 suggested in
 this thread (e.g. Tools and then Notification rules.) does not 
 work for
 some reason for a few of us including myself.

 Could we try to use any of our existing tools? For example,
 could the proposals be done in the form of filing a new JIRA issue 
 under a
 specific component. All of us should be able to get notifications 
 for that.
 And then we can follow the lazy consensus and alternative approval 
 options
 as written down by Robert (1 week or 3 PMC +1s).

Is there good way to make Python SDK docs draft accessible?

[Yoshiki Obata]

Hi all.

I'm working on enabling to generate Python SDK docs with Python3 [1]
I have modified scripts and now reviewing generated docs in someone’s
eyes is needed.

But there seems to be no existing way to upload generated docs to
where anyone can access unlike website html which can be uploaded to
GCS via Jenkins job.
Would anyone know good way to make generated docs accessible for
anyone for convenience of reviewing them?

[1] https://issues.apache.org/jira/browse/BEAM-7847


Best regards,
Yoshiki


--
Yoshiki Obata
mail: yoshiki.ob...@gmail.com
gh: https://github.com/lazylynx

Re: Go IOs ?

[Chamikara Jayalath]

On Mon, Oct 28, 2019 at 9:15 AM Robert Burke  wrote:

> There are IOs that work [1], but they haven't been vetted for production
> use (performance, overhead etc) just yet. As such, I don't recommend
> putting them on the site at this time. Of course, folks a free to discuss
> and disagree with me on that :).
>

Makes sense.


>
> The Go SDK doesn't yet have a way of scaling transforms without putting in
> a reshards/GBK, which single threads reads and slows things down
> dramatically. The portable API to do so is usually called Splittable DoFn,
> and is presently being worked on for the Go SDK [2].
>

I think scaling through re-shards should be enough for many cases. We have
some transforms in Java/Python, that depend on reshards for scaling and
successfully used by my folks in production. For example,
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L224
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerIO.java#L424

It would be good if we can create JIRAs to capture work needed to make
existing Go IO connectors production ready.
For example,
(1) Add reshards to parallelize reading.
(2) Add performance tests to detect anomalies

Thanks,
Cham



> A slightly faster path to performant IOs is to leverage the Cross Language
> Transform work so the Go SDK can use the existing Java and Python IOs. IIRC
> the trick would be to integrate the client side work to communicate with
> the expansion server that Python uses to make that work.
>

+1

>
> [1] https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io
> [2] https://s.apache.org/beam-go-sdf
>
>
> On Sun, Oct 27, 2019, 10:10 AM Austin Bennett 
> wrote:
>
>> Hi All,
>>
>> https://beam.apache.org/documentation/io/built-in/
>>
>> Only includes Ios for Java and Python.  I have heard talk of a Go SDK :-)
>>  Are there working IOs?  If yes, should they also be on that page?
>>
>> Thanks,
>> Austin
>>
>>
>>

RFC: python static typing PR

[Chad Dombrova]

Hi all,
I've been working on a PR to add static typing to the beam python sdk for
the past 4 months or so.  This has been an epic journey which has required
chasing down numerous fixes across several other projects (mypy, pylint,
python-future), but the mypy tests are now passing!

I'm not sure how much convincing I need to do with this group on the
benefits of static typing, especially considering the heavy Java influence
on this project, but for those who are curious, there's some good info
here:  https://realpython.com/python-type-checking/#pros-and-cons.  Suffice
it to say, it's a game changer for projects like Beam (large code base,
high level of quality, good testing infrastructure), and it dovetails
perfectly into the efforts surrounding pipeline type analysis and
serialization.  Anecdotally speaking, all of the developers I've worked
with who were originally resistant to static typing in python ("it's
ugly!"  "it's not pythonic!") have changed their tune and fully embraced it
because of the time that it saves them every day.

More details over at the PR:  https://github.com/apache/beam/pull/9056

I look forward to your feedback!

-chad

Re: [DISCUSS] How to stopp SdkWorker in SdkHarness

[Luke Cwik]

I would go with creating JIRAs and PRs directly since this doesn't seem to
be contentious since you have received feedback from a few folks and they
are all suggesting the same thing.

On Sun, Oct 27, 2019 at 9:27 PM jincheng sun 
wrote:

> Hi all,
>
> Thanks a lot for your feedback. It seems that we have reached consensus
> that both "Approach 2" and "Approach 3" are needed. "Approach 3" is a good
> supplement for "Approach 2" and I also prefer "Approach 2" and "Approach 3"
> for now.
>
> Do we need to vote on this discussion or I can create JIRAs and submit the
> PRs directly?
>
> Best,
> Jincheng
>
> Luke Cwik  于2019年10月26日周六 上午4:01写道：
>
>> Approach 3 is about caching the bundle descriptor forever but tearing
>> down a "live" instance of the DoFns at some SDK chosen arbitrary point in
>> time. This way if a future ProcessBundleRequest comes in, a new "live"
>> instance can be constructed.
>> Approach 2 is still needed so that when the workers are being
>> shutdown all the "live" instances are torn down.
>>
>> On Fri, Oct 25, 2019 at 12:56 PM Robert Burke  wrote:
>>
>>> Approach 2 isn't incompatible with approach 3. 3 simple sets down
>>> convention/configuration for the conditions when the SDK will do this after
>>> process bundle has completed.
>>>
>>>
>>>
>>> On Fri, Oct 25, 2019, 12:34 PM Robert Bradshaw 
>>> wrote:
>>>
 I think we'll still need approach (2) for when the pipeline finishes
 and a runner is tearing down workers.

 On Fri, Oct 25, 2019 at 10:36 AM Maximilian Michels 
 wrote:
 >
 > Hi Jincheng,
 >
 > Thanks for bringing this up and capturing the ideas in the doc.
 >
 > Intuitively, I would have also considered adding a new Proto message
 for
 > the teardown, but I think the idea to trigger this logic when the SDK
 > Harness evicts process bundle descriptors is more elegant.
 >
 > Thanks,
 > Max
 >
 > On 25.10.19 17:23, Luke Cwik wrote:
 > > I like approach 3 since it doesn't add additional complexity to the
 API
 > > and individual SDKs can choose to implement any clean-up strategy
 they
 > > want or none at all which is the simplest.
 > >
 > > On Thu, Oct 24, 2019 at 8:46 PM jincheng sun <
 sunjincheng...@gmail.com
 > > > wrote:
 > >
 > > Hi,
 > >
 > > Thanks for your comments in doc, I have add Approach 3 which you
 > > mentioned! @Luke
 > >
 > > For now, we should do a decision for Approach 3 and Approach 1.
 > > Detail can be found in doc [1]
 > >
 > > Welcome anyone's feedback :)
 > >
 > > Regards,
 > > Jincheng
 > >
 > > [1]
 > >
 https://docs.google.com/document/d/1sCgy9VQPf9zVXKRquK8P6N4x7aB62GEO8ozkujRSHZg/edit?usp=sharing
 > >
 > > jincheng sun >>> > > > 于2019年10月25日周五 上午10:40写道：
 > >
 > > Hi,
 > >
 > > Functionally capable of `abort`, but it will be called at
 the
 > > end of operator. So, I prefer `dispose` semantics. i.e., all
 > > normal logic has been executed.
 > >
 > > Best,
 > > Jincheng
 > >
 > > Harsh Vardhan >>> anan...@google.com>>
 > > 于2019年10月23日周三 上午12:14写道：
 > >
 > > Would approach 1 be akin to abort semantics?
 > >
 > > On Mon, Oct 21, 2019 at 8:01 PM jincheng sun
 > > >>> sunjincheng...@gmail.com>>
 > > wrote:
 > >
 > > Hi Luke,
 > >
 > > Thanks a lot for your reply. Since it allows to
 share
 > > one SDK harness between multiple executable stages,
 the
 > > control service termination may occur much later
 than
 > > the completion of an executable stage. This is the
 main
 > > reason I prefer runners to control the teardown of
 DoFns.
 > >
 > > Regarding to "SDK harnesses can terminate instances
 any
 > > time they want and start new instances anytime as
 > > well.", personally I think it's not conflict with
 the
 > > proposed Approach 1 as the SDK harness could decide
 what
 > > to do when receiving the teardown request. It could
 do
 > > nothing if the DoFns has already been teared down
 and
 > > could also tear down the DoFns if needed.
 > >
 > > What do you think?
 > >
 > > Best,
 > > Jincheng
 > >
 > > Luke Cwik >>> lc...@google.com>>
 > > 于2019年10月22日周二 上午2:05写道：
 > >
 > > Approach 2 is currently the suggested
 approach[1]
 

Re: [spark structured streaming runner] merge to master?

[Alexey Romanenko]

Let me share some of my thoughts on this.
>> - shall we filter out the package name from the release? 
>> 
Until new runner is not ready to be used in production (or, at least, be used 
for beta testing but users should be clearly warned about that in this case), I 
believe we need to filter out its classes from published jar to avoid a 
confusion.
>> - should we release 2 jars: one for the old and one for the new ? 
>> 
>> - should we release 3 jars: one for the new, one for the new and one for 
>> both ?
>> 
Once new runner will be released, then I think we need to provide only one 
single jar and allow user to switch between different Spark runners with CLI 
option.
>> - should we create a special entry to the capability matrix ?
>> 

Sure, since it has its own uniq characteristics and implementation, but again, 
only once new runner will be "officially released".


> On 28 Oct 2019, at 10:27, Etienne Chauchot  wrote:
> 
> Hi guys,
> 
> Any opinions on the point2 communication to users ?
> 
> Etienne
> On 24/10/2019 15:44, Etienne Chauchot wrote:
>> Hi guys,
>> 
>> I'm glad to announce that the PR for the merge to master of the new runner 
>> based on Spark Structured Streaming framework is submitted:
>> 
>> https://github.com/apache/beam/pull/9866 
>> 
>> 
>> 1. Regarding the status of the runner: 
>> -the runner passes 93% of the validates runner tests in batch mode.
>> 
>> -Streaming mode is barely started (waiting for the multi-aggregations 
>> support in spark Structured Streaming framework from the Spark community)
>> 
>> -Runner can execute Nexmark
>> 
>> -Some things are not wired up yet
>> 
>>   -Beam Schemas not wired with Spark Schemas
>> 
>>   -Optional features of the model not implemented: state api, timer api, 
>> splittable doFn api, …
>> 
>> 2. Regarding the communication to users:
>> 
>> - for reasons explained by Ismael: the runner is in the same module as the 
>> "older" one. But it is in a different sub-package and both runners share the 
>> same build.  
>> - How should we communicate to users: 
>> - shall we filter out the package name from the release? 
>> - should we release 2 jars: one for the old and one for the new ? 
>> - should we release 3 jars: one for the new, one for the new and one for 
>> both ?
>> 
>> - should we create a special entry to the capability matrix ?
>> 
>> WDYT ?
>> Best
>> 
>> Etienne
>> 
>> On 23/10/2019 19:11, Mikhail Gryzykhin wrote:
>>> +1 to merge.
>>> 
>>> It is worth keeping things in master with explicitly marked status. It will 
>>> make effort more visible to users and easier to get feedback upon.
>>> 
>>> --Mikhail
>>> 
>>> On Wed, Oct 23, 2019 at 8:36 AM Etienne Chauchot >> > wrote:
>>> Hi guys,
>>> 
>>> The new spark runner now supports beam coders and passes 93% of the batch 
>>> validates runner tests (+4%). I think it is time to merge it to master. I 
>>> will submit a PR in the coming days.
>>> 
>>> next steps: support schemas and thus better leverage catalyst optimizer 
>>> (among other things optims based on data), port perfs optims that were done 
>>> in the current runner.
>>> Best
>>> Etienne
>>> On 11/10/2019 22:48, Pablo Estrada wrote:
 +1 for merging : )
 
 On Fri, Oct 11, 2019 at 12:43 PM Robert Bradshaw >>> > wrote:
 Sounds like a good plan to me. 
 
 On Fri, Oct 11, 2019 at 6:20 AM Etienne Chauchot >>> > wrote:
 Comments inline
 On 10/10/2019 23:44, Ismaël Mejía wrote:
> +1
> 
> The earlier we get to master the better to encourage not only code
> contributions but as important to have early user feedback.
> 
>> Question is: do we keep the "old" spark runner for a while or not (or 
>> just keep on previous version/tag on git) ?
> It is still too early to even start discussing when to remove the
> classical runner given that the new runner is still a WIP. However the
> overall goal is that this runner becomes the de-facto one once the VR
> tests and the performance become at least equal to the classical
> runner, in the meantime the best for users is that they co-exist,
> let’s not forget that the other runner has been already battle tested
> for more than 3 years and has had lots of improvements in the last
> year.
 +1 on what Ismael says: no soon removal, 
 The plan I had in mind at first (that I showed at the apacheCon) was this 
 but I'm proposing moving the first gray label to before the red box. 
 
 
 
>> I don't think the number of commits should be an issue--we shouldn't
>> just squash years worth of history away. (OTOH, if this is a case of
>> this branch containing lots of little, irrelevant commits that would
>> have normally been squashed away in the normal review process we do
>> for the main branch, then, yes, some cleanup 

Re: Go IOs ?

[Robert Burke]

There are IOs that work [1], but they haven't been vetted for production
use (performance, overhead etc) just yet. As such, I don't recommend
putting them on the site at this time. Of course, folks a free to discuss
and disagree with me on that :).

The Go SDK doesn't yet have a way of scaling transforms without putting in
a reshards/GBK, which single threads reads and slows things down
dramatically. The portable API to do so is usually called Splittable DoFn,
and is presently being worked on for the Go SDK [2].

A slightly faster path to performant IOs is to leverage the Cross Language
Transform work so the Go SDK can use the existing Java and Python IOs. IIRC
the trick would be to integrate the client side work to communicate with
the expansion server that Python uses to make that work.

[1] https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io
[2] https://s.apache.org/beam-go-sdf


On Sun, Oct 27, 2019, 10:10 AM Austin Bennett 
wrote:

> Hi All,
>
> https://beam.apache.org/documentation/io/built-in/
>
> Only includes Ios for Java and Python.  I have heard talk of a Go SDK :-)
>  Are there working IOs?  If yes, should they also be on that page?
>
> Thanks,
> Austin
>
>
>

Re: Go IOs ?

[Chamikara Jayalath]

Seems like we have several file-based IO connectors and datastore here:
https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/io
We should definitely add these to the list.

I guess bulk of IOs will be supported through cross-language transforms
framework when we have that for Go SDK.

Thanks,
Cham

On Sun, Oct 27, 2019 at 10:10 AM Austin Bennett 
wrote:

> Hi All,
>
> https://beam.apache.org/documentation/io/built-in/
>
> Only includes Ios for Java and Python.  I have heard talk of a Go SDK :-)
>  Are there working IOs?  If yes, should they also be on that page?
>
> Thanks,
> Austin
>
>
>

Re: Apache Pulsar connector for Beam

[Yijie Shen]

I've written pulsar-flink and pulsar-spark connectors before.

Please feel free to ping me if you need some help.

Best,
Yijie

On Mon, Oct 28, 2019 at 1:21 AM Sijie Guo  wrote:

> Add Yijie in the loop. He can help from Pulsar side.
>
> - Sijie
>
> On Sat, Oct 26, 2019 at 7:23 PM Taher Koitawala 
> wrote:
>
>> Awesome. Thanks Max. Will definitely keep the community posted on this.
>>
>> On Sat, Oct 26, 2019, 4:21 PM Maximilian Michels  wrote:
>>
>>> Awesome. I've made you a contributor, so you can also create issues and
>>> assign yourself now.
>>>
>>> Please let us know about the progress once you start writing the
>>> connector.
>>>
>>> Thanks,
>>> Max
>>>
>>> On 26.10.19 12:36, Taher Koitawala wrote:
>>> > Thank you Alex and Max,
>>> >  My jira id is taherk77. Please add me.
>>> >
>>> > Regards,
>>> > Taher Koitawala
>>> >
>>> > On Sat, Oct 26, 2019, 3:53 PM Maximilian Michels >> > > wrote:
>>> >
>>> > That sounds great. How about you start looking into this Taher? If
>>> > necessary, Sijie could provide additional insight into Pulsar.
>>> >
>>> > Please create a JIRA account so we can assign you to
>>> > https://issues.apache.org/jira/browse/BEAM-8218
>>> >
>>> > Thanks,
>>> > Max
>>> >
>>> > On 26.10.19 12:08, Alex Van Boxel wrote:
>>> >  > Hey Taher, do you have a Jira account? Then I will assign the
>>> > ticket to
>>> >  > you. I made the ticket because we should have one, feel free to
>>> > take the
>>> >  > lead on this one.
>>> >  >
>>> >  >   _/
>>> >  > _/ Alex Van Boxel
>>> >  >
>>> >  >
>>> >  > On Fri, Oct 25, 2019 at 9:35 PM Taher Koitawala
>>> > mailto:taher...@gmail.com>
>>> >  > >> wrote:
>>> >  >
>>> >  > I would be interested in contributing to the Pulsar Beam
>>> > connector.
>>> >  > That's one of the reasons i started the email thread.
>>> >  >
>>> >  >
>>> >  > Regards,
>>> >  > Taher Koitawala
>>> >  >
>>> >  > On Sat, Oct 26, 2019, 9:41 AM Sijie Guo >> > 
>>> >  > >>
>>> wrote:
>>> >  >
>>> >  > This is Sijie Guo from StreamNative and Pulsar PMC.
>>> >  >
>>> >  > Maximilian - thank you for adding us in the email
>>> thread!
>>> >  >
>>> >  > We do have one roadmap item for adding a Beam connector
>>> for
>>> >  > Pulsar. It was planned for this quarter, but we haven’t
>>> > started
>>> >  > the implementation yet. If the Beam community is
>>> > interested in
>>> >  > it, we are happy to collaborate with Beam community.
>>> >  >
>>> >  > Thanks,
>>> >  > Sijie
>>> >  >
>>> >  > On Sat, Oct 26, 2019 at 12:36 AM Maximilian Michels
>>> >  > mailto:m...@apache.org>
>>> > >> wrote:
>>> >  >
>>> >  > It would be great to have a Pulsar connector. We
>>> > might want
>>> >  > to ask the
>>> >  > folks from StreamNative (in CC). Any plans? :)
>>> >  >
>>> >  > Cheers,
>>> >  > Max
>>> >  >
>>> >  > On 24.10.19 18:31, Pablo Estrada wrote:
>>> >  >  > There's a JIRA issue to track this:
>>> >  >  > https://issues.apache.org/jira/browse/BEAM-8218
>>> >  >  >
>>> >  >  > Alex was kind enough to file it. +Alex Van Boxel
>>> >  >  > > a...@vanboxel.be>
>>> > >> : )
>>> >  >  > Best
>>> >  >  > -P
>>> >  >  >
>>> >  >  > On Thu, Oct 24, 2019 at 12:01 AM Taher Koitawala
>>> >  > mailto:taher...@gmail.com>
>>> > >
>>> >  >  > >> >  >> > >> >  > wrote:
>>> >  >  >
>>> >  >  > Hi Reza,
>>> >  >  >   Thanks for your reply. However
>>> i
>>> > do not
>>> >  > see Pulsar
>>> >  >  > listed in there. Should we file a jira?
>>> >  >  >
>>> >  >  > On Thu, Oct 24, 2019, 12:16 PM Reza Rokni
>>> >  > mailto:r...@google.com>
>>> > >
>>> >  >  > > r...@google.com>
>>> > >> >  >  >
>>> >  >  

Re: Java PortableRunner package name

[Michał Walenia]

Hi,
thanks for the opinion. I think that's a very good argument against the
change. I'll stick with changing package names.

Have a good day,
Michal

On Mon, Oct 28, 2019 at 2:41 PM Maximilian Michels  wrote:

> Hi Michal,
>
> the package name looks good to me.
>
> -1 on the name change. For users, the current name "PortableRunner"
> reflects best what it does, running portable pipelines. The details of
> the translation and the submission process do not have to be reflected
> in the name.
>
> Cheers,
> Max
>
> On 28.10.19 13:06, Michał Walenia wrote:
> > Hi all,
> >
> > thank you for your replies and ideas. My proposition is to move
> > PortableRunner to package sdks.java.portability. I really like the idea
> > of renaming it - PortableRunnerClient looks like a good idea.
> >
> > WDYT?
> > Regards,
> > Michal
> >
> > On Wed, Oct 23, 2019 at 12:09 PM Ismaël Mejía  > > wrote:
> >
> > +Ankur Goenka
> >
> > Related JIRA. Maybe Ankur can chime in with more details on this and
> > other things he may have already thought.
> > https://issues.apache.org/jira/browse/BEAM-7303
> >
> >
> > On Tue, Oct 22, 2019 at 7:11 PM Maximilian Michels  > > wrote:
> >  >
> >  > +1 for moving. This is just a left-over from the fist "reference"
> > runner
> >  > implementation for portability.
> >  >
> >  > On 22.10.19 16:59, Łukasz Gajowy wrote:
> >  > > +1 for moving/renaming. I agree with Kyle and Michał - there
> indeed
> >  > > seems to be some confusion. The name "runners/reference"
> > suggests that
> >  > > it's a not production-ready "Runner" (it seems to be neither of
> > those).
> >  > > If possible, maybe sdks/java/portablility is a good place for
> this?
> >  > >
> >  > > Łukasz
> >  > >
> >  > > wt., 22 paź 2019 o 16:41 Kyle Weaver  > 
> >  > > >>
> > napisał(a):
> >  > >
> >  > > I agree this should be moved. PortableRunner.java is
> > analogous to
> >  > > portable_runner.py, which resides under
> >  > > sdks/python/apache_beam/runners/portability. Maybe
> >  > > PortableRunner.java should be moved to somewhere under
> > sdks/java, as
> >  > > it's not actually a runner itself. The nomenclature is
> >  > > confusing, PortableRunner could be more aptly named
> > something like
> >  > > `PortableRunnerClient`, or `JobClient` to better illustrate
> its
> >  > > relationship with `JobServer`.
> >  > >
> >  > > On Tue, Oct 22, 2019 at 4:11 PM Michał Walenia
> >  > >  > 
> >  > >> wrote:
> >  > >
> >  > > Hi,
> >  > >
> >  > > I found the Java PortableRunner class in
> >  > > org.apache.beam.runners.reference package, where
> > ReferenceRunner
> >  > > used to reside prior to its deletion. The PortableRunner
> >  > > implementation however is one that can be used with real
> >  > > JobServers in production code.
> >  > >
> >  > > *
> >  > > *
> >  > >
> >  > > It seems that this class shouldn’t be in the reference
> > package
> >  > > but somewhere else. I’d like to rename the package from
> >  > > org.apache.beam.runners.reference to
> >  > > org.apache.beam.runners.portability, as it contains
> > only classes
> >  > > related to the portable runner operation.
> >  > >
> >  > > *
> >  > > *
> >  > >
> >  > > What do you think? If nobody is strongly against the
> > change,
> >  > > I’ll make a pull request with the refactor.
> >  > >
> >  > > *
> >  > > *
> >  > >
> >  > > Have a good day,
> >  > >
> >  > > Michal
> >  > >
> >  > >
> >  > >
> >  > >
> >  > > --
> >  > >
> >  > > Michał Walenia
> >  > > Polidea  | Software Engineer
> >  > >
> >  > > M: +48 791 432 002 
> >  > > E: michal.wale...@polidea.com
> > 
> > >
> >  > >
> >  > > Unique Tech
> >  > > Check out our projects! <
> https://www.polidea.com/our-work>
> >  > >
> >
> >
> >
> > --
> >
> > Michał Walenia
> > Polidea  | Software Engineer
> >
> > M: +48 791 432 002 
> > E: michal.wale...@polidea.com 
> >
> > Unique Tech
> > Check out our projects! 
> >
>


-- 

Michał Walenia

Re: Java PortableRunner package name

[Maximilian Michels]

Hi Michal,

the package name looks good to me.

-1 on the name change. For users, the current name "PortableRunner" 
reflects best what it does, running portable pipelines. The details of 
the translation and the submission process do not have to be reflected 
in the name.


Cheers,
Max

On 28.10.19 13:06, Michał Walenia wrote:

Hi all,

thank you for your replies and ideas. My proposition is to move 
PortableRunner to package sdks.java.portability. I really like the idea 
of renaming it - PortableRunnerClient looks like a good idea.


WDYT?
Regards,
Michal

On Wed, Oct 23, 2019 at 12:09 PM Ismaël Mejía > wrote:


+Ankur Goenka

Related JIRA. Maybe Ankur can chime in with more details on this and
other things he may have already thought.
https://issues.apache.org/jira/browse/BEAM-7303


On Tue, Oct 22, 2019 at 7:11 PM Maximilian Michels mailto:m...@apache.org>> wrote:
 >
 > +1 for moving. This is just a left-over from the fist "reference"
runner
 > implementation for portability.
 >
 > On 22.10.19 16:59, Łukasz Gajowy wrote:
 > > +1 for moving/renaming. I agree with Kyle and Michał - there indeed
 > > seems to be some confusion. The name "runners/reference"
suggests that
 > > it's a not production-ready "Runner" (it seems to be neither of
those).
 > > If possible, maybe sdks/java/portablility is a good place for this?
 > >
 > > Łukasz
 > >
 > > wt., 22 paź 2019 o 16:41 Kyle Weaver mailto:kcwea...@google.com>
 > > >>
napisał(a):
 > >
 > >     I agree this should be moved. PortableRunner.java is
analogous to
 > >     portable_runner.py, which resides under
 > >     sdks/python/apache_beam/runners/portability. Maybe
 > >     PortableRunner.java should be moved to somewhere under
sdks/java, as
 > >     it's not actually a runner itself. The nomenclature is
 > >     confusing, PortableRunner could be more aptly named
something like
 > >     `PortableRunnerClient`, or `JobClient` to better illustrate its
 > >     relationship with `JobServer`.
 > >
 > >     On Tue, Oct 22, 2019 at 4:11 PM Michał Walenia
 > >     mailto:michal.wale...@polidea.com>
>> wrote:
 > >
 > >         Hi,
 > >
 > >         I found the Java PortableRunner class in
 > >         org.apache.beam.runners.reference package, where
ReferenceRunner
 > >         used to reside prior to its deletion. The PortableRunner
 > >         implementation however is one that can be used with real
 > >         JobServers in production code.
 > >
 > >         *
 > >         *
 > >
 > >         It seems that this class shouldn’t be in the reference
package
 > >         but somewhere else. I’d like to rename the package from
 > >         org.apache.beam.runners.reference to
 > >         org.apache.beam.runners.portability, as it contains
only classes
 > >         related to the portable runner operation.
 > >
 > >         *
 > >         *
 > >
 > >         What do you think? If nobody is strongly against the
change,
 > >         I’ll make a pull request with the refactor.
 > >
 > >         *
 > >         *
 > >
 > >         Have a good day,
 > >
 > >         Michal
 > >
 > >
 > >
 > >
 > >         --
 > >
 > >         Michał Walenia
 > >         Polidea  | Software Engineer
 > >
 > >         M: +48 791 432 002 
 > >         E: michal.wale...@polidea.com

>
 > >
 > >         Unique Tech
 > >         Check out our projects! 
 > >



--

Michał Walenia
Polidea  | Software Engineer

M: +48 791 432 002 
E: michal.wale...@polidea.com 

Unique Tech
Check out our projects! 

Rethinking the Flink Runner modes

[Maximilian Michels]

Hi,

Robert and Kyle have been doing great work to simplify submitting 
portable pipelines with the Flink Runner. Part of this is having a 
Python "FlinkRunner" which handles bringing up a Beam job server and 
submitting the pipeline directly via the Flink REST API. One building 
block is the creation of "executable Jars" which contain the 
materialized / translated Flink pipeline and do not require the Beam job 
server or the Python driver anymore.


While unifying a newly introduced option "flink_master_url" with the 
pre-existing "flink_master" [1][2], some questions came up about Flink's 
execution modes. (The two options are meant to do the same thing: 
provide the address of the Flink master to hand-over the translated 
pipeline.)


Historically, Flink had a proprietary protocol for submitting pipelines, 
running on port 9091. This has since been replaced with a REST protocol 
at port 8081. To this date, this has implications how you submit 
programs, e.g. the Flink client libraries expects the address to be of 
form "host:port", without a protocol scheme. On the other hand, external 
Rest libraries typically expect a protocol scheme.


But this is only half of the fun. There are also special addresses for 
"flink_master" that influence submission of the pipeline. If you specify 
"[local]" as the address, the pipeline won't be submitted but executed 
in a local in-process Flink cluster. If you specify "[auto]" and you use 
the CLI tool that comes bundled with Flink, then the master address will 
be loaded from the Flink config, including any configuration like SSL. 
If none is found, then it falls back to "[local]".


This is a bit odd, and after a discussion with Robert and Thomas in [1], 
we figured that this needs to be changed:


1. Make the master address a URL. Add "http://; to "flink_master" in 
Python if no scheme is specified. Similarly, remove any "http://; in 
Java, since the Java rest client does not expect a scheme. In case of 
"http_s_://", we have a special treatment to load the SSL settings from 
the Flink config.


2. Deprecate the "[auto]" and "[local]" values. It should be sufficient 
to have either a non-empty address string or an empty one. The empty 
string would either mean local execution or, in the context of the Flink 
CLI tool, loading the master address from the config. The non-empty 
string would be interpreted as a cluster address.



Any opinions on this?


Thanks,
Max


[1] https://github.com/apache/beam/pull/9803
[2] https://github.com/apache/beam/pull/9844

Re: Java PortableRunner package name

[Michał Walenia]

Hi all,

thank you for your replies and ideas. My proposition is to move
PortableRunner to package sdks.java.portability. I really like the idea of
renaming it - PortableRunnerClient looks like a good idea.

WDYT?
Regards,
Michal

On Wed, Oct 23, 2019 at 12:09 PM Ismaël Mejía  wrote:

> +Ankur Goenka
>
> Related JIRA. Maybe Ankur can chime in with more details on this and
> other things he may have already thought.
> https://issues.apache.org/jira/browse/BEAM-7303
>
>
> On Tue, Oct 22, 2019 at 7:11 PM Maximilian Michels  wrote:
> >
> > +1 for moving. This is just a left-over from the fist "reference" runner
> > implementation for portability.
> >
> > On 22.10.19 16:59, Łukasz Gajowy wrote:
> > > +1 for moving/renaming. I agree with Kyle and Michał - there indeed
> > > seems to be some confusion. The name "runners/reference" suggests that
> > > it's a not production-ready "Runner" (it seems to be neither of those).
> > > If possible, maybe sdks/java/portablility is a good place for this?
> > >
> > > Łukasz
> > >
> > > wt., 22 paź 2019 o 16:41 Kyle Weaver  > > > napisał(a):
> > >
> > > I agree this should be moved. PortableRunner.java is analogous to
> > > portable_runner.py, which resides under
> > > sdks/python/apache_beam/runners/portability. Maybe
> > > PortableRunner.java should be moved to somewhere under sdks/java,
> as
> > > it's not actually a runner itself. The nomenclature is
> > > confusing, PortableRunner could be more aptly named something like
> > > `PortableRunnerClient`, or `JobClient` to better illustrate its
> > > relationship with `JobServer`.
> > >
> > > On Tue, Oct 22, 2019 at 4:11 PM Michał Walenia
> > > mailto:michal.wale...@polidea.com>>
> wrote:
> > >
> > > Hi,
> > >
> > > I found the Java PortableRunner class in
> > > org.apache.beam.runners.reference package, where
> ReferenceRunner
> > > used to reside prior to its deletion. The PortableRunner
> > > implementation however is one that can be used with real
> > > JobServers in production code.
> > >
> > > *
> > > *
> > >
> > > It seems that this class shouldn’t be in the reference package
> > > but somewhere else. I’d like to rename the package from
> > > org.apache.beam.runners.reference to
> > > org.apache.beam.runners.portability, as it contains only
> classes
> > > related to the portable runner operation.
> > >
> > > *
> > > *
> > >
> > > What do you think? If nobody is strongly against the change,
> > > I’ll make a pull request with the refactor.
> > >
> > > *
> > > *
> > >
> > > Have a good day,
> > >
> > > Michal
> > >
> > >
> > >
> > >
> > > --
> > >
> > > Michał Walenia
> > > Polidea  | Software Engineer
> > >
> > > M: +48 791 432 002 
> > > E: michal.wale...@polidea.com  michal.wale...@polidea.com>
> > >
> > > Unique Tech
> > > Check out our projects! 
> > >
>


-- 

Michał Walenia
Polidea  | Software Engineer

M: +48 791 432 002 <+48791432002>
E: michal.wale...@polidea.com

Unique Tech
Check out our projects! 

Beam Dependency Check Report (2019-10-28)

[Apache Jenkins Server]

High Priority Dependency Updates Of Beam Python SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
mock
2.0.0
3.0.5
2019-05-20
2019-05-20BEAM-7369
oauth2client
3.0.0
4.1.3
2018-12-10
2018-12-10BEAM-6089
Sphinx
1.8.5
2.2.1
2019-05-20
2019-10-28BEAM-7370
High Priority Dependency Updates Of Beam Java SDK:


  Dependency Name
  Current Version
  Latest Version
  Release Date Of the Current Used Version
  Release Date Of The Latest Release
  JIRA Issue
  
com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
0.20.0
0.27.0
2019-02-11
2019-10-21BEAM-6645
com.github.spotbugs:spotbugs
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-7792
com.github.spotbugs:spotbugs-annotations
3.1.12
4.0.0-beta4
2019-03-01
2019-09-18BEAM-6951
javax.servlet:javax.servlet-api
3.1.0
4.0.1
2013-04-25
2018-04-20BEAM-5750
org.conscrypt:conscrypt-openjdk
1.1.3
2.2.1
2018-06-04
2019-08-08BEAM-5748
org.eclipse.jetty:jetty-server
9.2.10.v20150310
10.0.0-alpha0
2015-03-10
2019-07-11BEAM-5752
org.eclipse.jetty:jetty-servlet
9.2.10.v20150310
10.0.0-alpha0
2015-03-10
2019-07-11BEAM-5753
Gradle:
5.2.1 -> 5.6.3
6.0-rc-1
2019-10-21
2019-10-21BEAM-8002

 A dependency update is high priority if it satisfies one of following criteria: 

 It has major versions update available, e.g. org.assertj:assertj-core 2.5.0 -> 3.10.0; 


 It is over 3 minor versions behind the latest version, e.g. org.tukaani:xz 1.5 -> 1.8; 


 The current version is behind the later version for over 180 days, e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11. 

 In Beam, we make a best-effort attempt at keeping all dependencies up-to-date.
 In the future, issues will be filed and tracked for these automatically,
 but in the meantime you can search for existing issues or open a new one.

 For more information:  Beam Dependency Guide