Re: Removing documentation for old Beam versions

[Thomas Weise]

Hi Scott,

Thanks for bringing the discussion back here.

I agree that we should separate the changes for hosting of generated
java/pydocs from the rest of website automation so that we can make the
switch and fix the contributor headache soon.

But perhaps we can avoid adding 4m lines of generated code to the main beam
repository (and keep on adding with every release) if we continue to serve
the site from the old beam-site repo? (I left a comment the doc.)

About trying buildbot, as mentioned earlier I would be happy to help with
it. I prefer a setup that keeps the docs separate from the web site.

Thomas


On Fri, Sep 21, 2018 at 10:28 AM Scott Wegner  wrote:

> Re-opening this thread as it came up today in the discussion for PR#6458
> [1]. This PR is part of the work for Beam-Site Automation Reliability
> improvements; design doc here: https://s.apache.org/beam-site-automation
>
> The current plan is to keep generated javadoc/pydoc sources only on the
> asf-site branch, which is necessary for the current githubpubsub publishing
> mechanism. This maintains our current approach, the only change being that
> we're moving the asf-site branch from the retiring apache/beam-site
> repository into a new apache/beam repo branch.
>
> The concern for committing generated content is the extra overhead during
> git fetch. I did some analysis to measure the impact [2], and found that
> fetching a week of source + generated content history from apache/beam-site
> took 0.39 seconds.
>
> I like the idea of publishing javadoc/pydoc snapshots to an external
> location like Flink does with buildbot, but that work is separable and
> shouldn't be a prerequisite for this effort. The goal of this work is to
> improve the reliability of automation for contributing website changes. At
> last measure, only about half of beam-site PR merges use Mergebot without
> experiencing some reliability issue [3].
>
> I've opened BEAM-5459 [4] to track moving our generated docs out of git.
> Thomas, would you have bandwidth to look into this?
>
> [1] https://github.com/apache/beam/pull/6458#issuecomment-423406643
> [2]
> https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit#heading=h.uqzivheohd7j
> [3]
> https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit#heading=h.a208cwi78xmu
> [4] https://issues.apache.org/jira/browse/BEAM-5459
>
> On Fri, Aug 24, 2018 at 11:48 AM Thomas Weise  wrote:
>
>> Hi Udi,
>>
>> Good to know you will continue this work.
>>
>> Let me know if you want to try the buildbot route (which does not require
>> generated documentation to be checked into the repo). Happy to help with
>> that.
>>
>> Thomas
>>
>> On Fri, Aug 24, 2018 at 11:36 AM Udi Meiri  wrote:
>>
>>> I'm picking up the website migration. The plan is to not include
>>> generated files in the master branch.
>>>
>>> However, I've been told that even putting generated files a separate
>>> branch could blow up the git repository for all (e.g. make git pulls a lot
>>> longer?).
>>> Not sure if this is a real issue or not.
>>>
>>> On Mon, Aug 20, 2018 at 2:53 AM Robert Bradshaw 
>>> wrote:
>>>
 On Sun, Aug 5, 2018 at 5:28 AM Thomas Weise  wrote:
 >
 > Yes, I think the separation of generated code will need to occur
 prior to completing the merge and switching the web site to the main repo.
 >
 > There should be no reason to check generated documentation into
 either of the repos/branches.

 Huge +1 to this. Thomas, would have time to set something like this up
 for Beam? If not, could anyone else pick this up?

 > Please see as an example how this was solved in Flink, using the ASF
 buildbot infrastructure.
 >
 > Documentation per version/release, for example:
 >
 > https://ci.apache.org/projects/flink/flink-docs-release-1.5/
 >
 > The buildbot configuration is here (requires committer access):
 >
 >
 https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/flink.conf
 >
 > Thanks,
 > Thomas
 >
 > On Thu, Aug 2, 2018 at 6:46 PM Mikhail Gryzykhin 
 wrote:
 >>
 >> Last time I talked with Scott I brought this idea in. I believe the
 plan was either to publish compiled site to website directly, or keep it in
 separate storage from apache/beam repo.
 >>
 >> One of the main reasons not to check in compiled version of website
 is that every developer will have to pull all the versions of website every
 time they clone repo, which is not that good of an idea to do.
 >>
 >> Regards,
 >> --Mikhail
 >>
 >> Have feedback?
 >>
 >>
 >> On Thu, Aug 2, 2018 at 6:42 PM Udi Meiri  wrote:
 >>>
 >>> Pablo, the docs are generated into versioned paths, e.g.,
 https://beam.apache.org/documentation/sdks/javadoc/2.5.0/ so tags are
 not necessary?
 >>> Also, once apache/beam-site is merged with 

Re: Python SDK: .options deprecation

[Ahmet Altay]

If I remember correctly, this was related to change of the signature of run
on runners to run(pipeline, options) instead of just run(pipeline). A
runner accepts a pipeline and set of options to run that pipeline, instead
of a pipeline and options being a combined thing.

It would be good to update the comments or bug with some explanation.

On Fri, Sep 21, 2018 at 2:20 PM, Sindy Li  wrote:

> I think it's just deprecating referencing the option as
>pipeline.options
> , because it is made into a private variable, but we can still do
>pipeline.options()
>
> On Fri, Sep 21, 2018 at 2:11 PM Udi Meiri  wrote:
>
>> Hey, does anybody know why the pipeline.options property was deprecated?
>> I found this bug: https://issues.apache.org/jira/browse/BEAM-2124
>> but there's no explanation.
>>
>

Re: SplittableDoFn

[Lukasz Cwik]

It turned out that not providing direct access to the ByteKeyRangeTracker
and only to the RestrictionTracker prevented the usage of a "markDone"
method which through further investigation seems to have been hiding a bug
where we thought that lexicographically the byte key { 0x02 } was the
smallest key greater then { 0x01 } when it should have been { 0x01 0x00 }.
Will need to investigate further but it could be that we are actually
missing processing keys if your next key based upon the old calculation was
greater then the end key.

On Fri, Sep 21, 2018 at 3:41 PM Lukasz Cwik  wrote:

> The synchronization is related to Java thread safety since there is likely
> to be concurrent access needed to a restriction tracker to properly handle
> accessing the backlog and splitting concurrently from when the users DoFn
> is executing and updating the restriction tracker. This is similar to the
> Java thread safety needed in BoundedSource and UnboundedSource for fraction
> consumed, backlog bytes, and splitting.
>
> On Fri, Sep 21, 2018 at 2:38 PM Reuven Lax  wrote:
>
>> Can you give details on what the synchronization is per? Is it per key,
>> or global to each worker?
>>
>> On Fri, Sep 21, 2018 at 2:10 PM Lukasz Cwik  wrote:
>>
>>> As I was looking at the SplittableDoFn API while working towards making
>>> a proposal for how the backlog/splitting API could look, I found some sharp
>>> edges that could be improved.
>>>
>>> I noticed that:
>>> 1) We require users to write thread safe code, this is something that we
>>> haven't asked of users when writing a DoFn.
>>> 2) We "internal" methods within the RestrictionTracker that are not
>>> meant to be used by the runner.
>>>
>>> I can fix these issues by giving the user a forwarding restriction
>>> tracker[1] that provides an appropriate level of synchronization as needed
>>> and also provides the necessary observation hooks to see when a claim
>>> failed or succeeded.
>>>
>>> This requires a change to our experimental API since we need to pass
>>> a RestrictionTracker to the @ProcessElement method instead of a sub-type of
>>> RestrictionTracker.
>>> @ProcessElement
>>> processElement(ProcessContext context, OffsetRangeTracker tracker) { ...
>>> }
>>> becomes:
>>> @ProcessElement
>>> processElement(ProcessContext context, RestrictionTracker>> Long> tracker) { ... }
>>>
>>> This provides an additional benefit that it prevents users from working
>>> around the RestrictionTracker APIs and potentially making underlying
>>> changes to the tracker outside of the tryClaim call.
>>>
>>> Full implementation is available within this PR[2] and was wondering
>>> what people thought.
>>>
>>> 1:
>>> https://github.com/apache/beam/pull/6467/files#diff-ed95abb6bc30a9ed07faef5c3fea93f0R72
>>> 2: https://github.com/apache/beam/pull/6467
>>>
>>>
>>> On Mon, Sep 17, 2018 at 12:45 PM Lukasz Cwik  wrote:
>>>
 The changes to the API have not been proposed yet. So far it has all
 been about what is the representation and why.

 For splitting, the current idea has been about using the backlog as a
 way of telling the SplittableDoFn where to split, so it would be in terms
 of whatever the SDK decided to report.
 The runner always chooses a number for backlog that is relative to the
 SDKs reported backlog. It would be upto the SDK to round/clamp the number
 given by the Runner to represent something meaningful for itself.
 For example if the backlog that the SDK was reporting was bytes
 remaining in a file such as 500, then the Runner could provide some value
 like 212.2 which the SDK would then round to 212.
 If the backlog that the SDK was reporting 57 pubsub messages, then the
 Runner could provide a value like 300 which would mean to read 57 values
 and then another 243 as part of the current restriction.

 I believe that BoundedSource/UnboundedSource will have wrappers added
 that provide a basic SplittableDoFn implementation so existing IOs should
 be migrated over without API changes.

 On Mon, Sep 17, 2018 at 1:11 AM Ismaël Mejía  wrote:

> Thanks a lot Luke for bringing this back to the mailing list and Ryan
> for taking
> the notes.
>
> I would like to know if there was some discussion, or if you guys have
> given
> some thought to the required changes in the SDK (API) part. What will
> be the
> equivalent of `splitAtFraction` and what should IO authors do to
> support it..
>
> On Sat, Sep 15, 2018 at 1:52 AM Lukasz Cwik  wrote:
> >
> > Thanks to everyone who joined and for the questions asked.
> >
> > Ryan graciously collected notes of the discussion:
> https://docs.google.com/document/d/1kjJLGIiNAGvDiUCMEtQbw8tyOXESvwGeGZLL-0M06fQ/edit?usp=sharing
> >
> > The summary was that bringing BoundedSource/UnboundedSource into
> using a unified backlog-reporting mechanism with optional other signals
> that Dataflow has 

Re: Beam python sdk with gevent

[Ahmet Altay]

Thank you for the example it helps.

I still do not know what is wrong with gevent. Would you consider using
multiprocessing package? We are already using that to accomplish something
similar in file based sinks, and there is already utility function that
wraps it around similar to your example [1].

[1]
https://github.com/apache/beam/blob/59c85b44d156bb7b4462d80fcb5591f860235708/sdks/python/apache_beam/internal/util.py#L117

Ahmet


On Wed, Sep 19, 2018 at 10:25 PM, Rakesh Kumar  wrote:

>
> Gevent  is basically used to make parallel
> network calls. We are using gevent in one of the transformation methods to
> call internal services. The transformation method is making multiple
> network call in parallel. Here is the code snippet:
> /__init__.py
> import gevent.monkey
> gevent.monkey.patch_all()
>
> /transform.py
> from gevent import Greenlet
> from gevent import joinall
> def filter_out_invalid_users(events):
>key, user_id_data_pairs = events
>user_ids = [user_id for user_id, data in user_id_data_pairs]
>
>jobs = []
>id_chunks = utils.chunk_list_evenly(user_ids, BATCH_SIZE)
>for id_chunk in id_chunks:
>   jobs.append(Greenlet.spawn(_call_users_service, #
> _call_user_service_ method is making the network call.
>  list(id_chunk)))
>
>"""
>Here we increase the timeout based on the number of greenlets we are
> running, to account for yielding
>among greenlets
>"""
>join_timeout = GREENLET_TIMEOUT + len(jobs) * GREENLET_TIMEOUT * 0.1
>joinall(jobs, timeout=join_timeout)
>
>successful_jobs = [job for job in jobs if job.successful()]
>valid_user_ids = []
>for job in successful_jobs:
>   network_response = job.get()
>   valid_user_ids.append(network_response.user_id)
>yield valid_user_ids
>
> def _call_users_service(user_ids):
># make network call and return response
>..
>..
>return network_response
>
> On Tue, Sep 18, 2018 at 7:07 PM Ahmet Altay  wrote:
>
>> I am also not familiar with gevent. Could you explain what are you trying
>> to do and how do you plan to use gevent?
>>
>> On Tue, Sep 18, 2018 at 9:38 AM, Lukasz Cwik  wrote:
>>
>>> I don't think anyone has tried what your doing. The code that your
>>> working with is very new.
>>>
>>> On Mon, Sep 17, 2018 at 5:02 PM Micah Wylde  wrote:
>>>
 Hi all,

 We're using the Python SDK with the portable Flink runner and running
 into some problems integrating gevent. We're patching the gRPC runtime for
 gevent as described in [0] which allows pipelines to start and partially
 run. However the tasks produce a stream of gevent exceptions:

 Exception greenlet.error: error('cannot switch to a different thread',)
 in 'grpc._cython.cygrpc.run_loop' ignored
 Traceback (most recent call last):
   File "src/gevent/event.py", line 240, in gevent._event.Event.wait
   File "src/gevent/event.py", line 140, in gevent._event._
 AbstractLinkable._wait
   File "src/gevent/event.py", line 117, in gevent._event._
 AbstractLinkable._wait_core
   File "src/gevent/event.py", line 119, in gevent._event._
 AbstractLinkable._wait_core
   File "src/gevent/_greenlet_primitives.py", line 59, in
 gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
   File "src/gevent/_greenlet_primitives.py", line 59, in
 gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
   File "src/gevent/_greenlet_primitives.py", line 63, in
 gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
   File "src/gevent/__greenlet_primitives.pxd", line 35, in
 gevent.__greenlet_primitives._greenlet_switch
 greenlet.error: cannot switch to a different thread

 and do not make any progress.

 Has anybody else successfully used the portable python sdk with gevent?
 Or is there a recommended alternative for doing async IO in python
 pipelines?

 [0] https://github.com/grpc/grpc/issues/4629#issuecomment-376962677

 Micah

>>>
>> --
> Rakesh Kumar
> Software Engineer
> 510-761-1364 |
>
> 
>

Resolving Go SDK build/test failures when using gradle

[Robert Burke]

If you haven't run into :beam-sdks-go:buildLinuxAmd64 or similar failing
with "undefined: passert.Sum" recently, stop reading now.

The root cause is that the gogradle plugin doesn't clean up the vendor
directories  that it
sets up, in combination with trying to use gradle with Go, leads to gradle
vendoring the beam package in the sdks/go/test, sdks/go/examples,
sdks/go/container directories.
This vendoring is persistent on the local client, and isn't cleaned up by
the clean task.

The immediate fix is to navigate to each of the sdks/go, and
sdks/go/{test|examples|container} directories and delete the vendor and
.gogradle directories.

eg Run the following from your beam git root, if your'e using a *Unix or
similar:

rm -rf sdks/go/{vendor,.gogradle}
sdks/go/{test,examples,container}/{vendor,.gogradle}

Then try running your gradle command again.

Go gradle will create the same vendored directories, but will at least have
a more up to date version of the Go SDK.

The short term fix would be to fix the recursive clean tasks that affect
the go builds to also remove the vendor directories, if not targetting the
vendored copy of beam specifically.

This situation is awful and while the long term fix would probably be to
use Go Modules  directly (see
BEAM-5379 ), and replace
gogradle plugin with targeted bash scripts, allowing the go tool to manage
dependencies and build artifacts directly.

If you're familiar with gradle and could provide guidance on customizing a
task, so it's properly invoked on the general clean task, it would be much
appreciated. BEAM-5465  has
been filed to track that work.

Thank you for your patience,
Robert B

Re: Is Splittable DoFn suitable for fetch data from a socket server?

[Raghu Angadi]

> This in-house built socket server could accept multiple clients, but only
send messages to the first-connected client, and will send messages to the
second client if the first one disconnected.

Server sending messages to first client connection only is quite critical.
Even if you use Source API which honors 'Setup()' JavaDoc, it is not enough
in your case. Note that is says it reuses, but that does not guarantee
single DoFn instance or when it actually calls TearDown(). It is on
best-effort basis. The work could move to a different worker and the DoFn
instance on earlier worker can live for a long time. So if you held the
connection to server until TearDown() is called, you could be inadvertently
blocking reads from DoFn on the new worker. If you want to keep the
connection open across bundles, you need some way to close an idle
connection asynchronously (alternately your service might have timeout to
close an idle client connection, which is much better). Since you can't
afford to wait till TearDown(), you might as well have a singleton
connection that gets closed after some idle time.

Assuming you need to ack on the same connection that served the records,
finalize() functionality in UnboundedSource API is important case. You can
use UnboundeSource API for now.

On Thu, Sep 20, 2018 at 8:25 PM flyisland  wrote:

> Hi Reuven,
>
> There is no explicit ID in the message itself, and if there is
> information can be used as an ID is depend on use cases.
>
> On Fri, Sep 21, 2018 at 11:05 AM Reuven Lax  wrote:
>
>> Is there information in the message that can be used as an id, that can
>> be used for deduplication?
>>
>> On Thu, Sep 20, 2018 at 6:36 PM flyisland  wrote:
>>
>>> Hi Lukasz,
>>>
>>> With the current API we provided, messages cannot be acked from a
>>> different client.
>>>
>>> The server will re-send messages to the reconnected client if those
>>> messages were not acked. So there'll be duplicate messages, but with a
>>> "redeliver times" property in the header.
>>>
>>> Thanks for your helpful information, I'll check the UnboundedSources,
>>> thanks!
>>>
>>>
>>>
>>> On Fri, Sep 21, 2018 at 2:09 AM Lukasz Cwik  wrote:
>>>
 Are duplicate messages ok?

 Can you ack messages from a different client or are messages sticky to
 a single client (e.g. if one client loses connection, when it reconnects
 can it ack messages it received or are those messages automatically
 replayed)?

 UnboundedSources are the only current "source" type that supports
 finalization callbacks[1] that you will need to ack messages and
 deduplication[2]. SplittableDoFn will support both of these features but
 are not there yet.

 1:
 https://github.com/apache/beam/blob/256fcdfe3ab0f6827195a262932e1267c4d4bfba/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L129
 2:
 https://github.com/apache/beam/blob/256fcdfe3ab0f6827195a262932e1267c4d4bfba/sdks/java/core/src/main/java/org/apache/beam/sdk/io/UnboundedSource.java#L93


 On Wed, Sep 19, 2018 at 8:31 PM flyisland  wrote:

> Hi Lukasz,
>
> This socket server is like an MQTT server, it has queues inside it and
> the client should ack each message.
>
> > Is receiving and processing each message reliably important or is it
> ok to drop messages when things fail?
> A: Reliable is important, no messages should be lost.
>
> > Is there a message acknowledgement system or can you request a
> position within the message stream (e.g. send all messages from position X
> when connecting and if for whatever reason you need to reconnect you can
> say send messages from position X to replay past messages)?
> A: Client should ack each message it received, and the server will
> delete the acked message. If the client broked and the server do not
> receive an ack, the server will re-send the message to the client while it
> online again. And there is no "position" concept like kafka.
>
>
> On Thu, Sep 20, 2018 at 4:27 AM Lukasz Cwik  wrote:
>
>> Before getting into what you could use and the current state of
>> SplittableDoFn and its supported features, I was wondering what 
>> reliability
>> guarantees does the socket server have around messages?
>>
>> Is receiving and processing each message reliably important or is it
>> ok to drop messages when things fail?
>> Is there a message acknowledgement system or can you request a
>> position within the message stream (e.g. send all messages from position 
>> X
>> when connecting and if for whatever reason you need to reconnect you can
>> say send messages from position X to replay past messages)?
>>
>>
>>
>>
>> On Tue, Sep 18, 2018 at 5:00 PM flyisland 
>> wrote:
>>
>>>
>>> Hi Gurus,
>>>
>>> I'm trying to create an IO connector to fetch data from a socket
>>> server from 

Re: SplittableDoFn

[Lukasz Cwik]

The synchronization is related to Java thread safety since there is likely
to be concurrent access needed to a restriction tracker to properly handle
accessing the backlog and splitting concurrently from when the users DoFn
is executing and updating the restriction tracker. This is similar to the
Java thread safety needed in BoundedSource and UnboundedSource for fraction
consumed, backlog bytes, and splitting.

On Fri, Sep 21, 2018 at 2:38 PM Reuven Lax  wrote:

> Can you give details on what the synchronization is per? Is it per key, or
> global to each worker?
>
> On Fri, Sep 21, 2018 at 2:10 PM Lukasz Cwik  wrote:
>
>> As I was looking at the SplittableDoFn API while working towards making a
>> proposal for how the backlog/splitting API could look, I found some sharp
>> edges that could be improved.
>>
>> I noticed that:
>> 1) We require users to write thread safe code, this is something that we
>> haven't asked of users when writing a DoFn.
>> 2) We "internal" methods within the RestrictionTracker that are not meant
>> to be used by the runner.
>>
>> I can fix these issues by giving the user a forwarding restriction
>> tracker[1] that provides an appropriate level of synchronization as needed
>> and also provides the necessary observation hooks to see when a claim
>> failed or succeeded.
>>
>> This requires a change to our experimental API since we need to pass
>> a RestrictionTracker to the @ProcessElement method instead of a sub-type of
>> RestrictionTracker.
>> @ProcessElement
>> processElement(ProcessContext context, OffsetRangeTracker tracker) { ... }
>> becomes:
>> @ProcessElement
>> processElement(ProcessContext context, RestrictionTracker> Long> tracker) { ... }
>>
>> This provides an additional benefit that it prevents users from working
>> around the RestrictionTracker APIs and potentially making underlying
>> changes to the tracker outside of the tryClaim call.
>>
>> Full implementation is available within this PR[2] and was wondering what
>> people thought.
>>
>> 1:
>> https://github.com/apache/beam/pull/6467/files#diff-ed95abb6bc30a9ed07faef5c3fea93f0R72
>> 2: https://github.com/apache/beam/pull/6467
>>
>>
>> On Mon, Sep 17, 2018 at 12:45 PM Lukasz Cwik  wrote:
>>
>>> The changes to the API have not been proposed yet. So far it has all
>>> been about what is the representation and why.
>>>
>>> For splitting, the current idea has been about using the backlog as a
>>> way of telling the SplittableDoFn where to split, so it would be in terms
>>> of whatever the SDK decided to report.
>>> The runner always chooses a number for backlog that is relative to the
>>> SDKs reported backlog. It would be upto the SDK to round/clamp the number
>>> given by the Runner to represent something meaningful for itself.
>>> For example if the backlog that the SDK was reporting was bytes
>>> remaining in a file such as 500, then the Runner could provide some value
>>> like 212.2 which the SDK would then round to 212.
>>> If the backlog that the SDK was reporting 57 pubsub messages, then the
>>> Runner could provide a value like 300 which would mean to read 57 values
>>> and then another 243 as part of the current restriction.
>>>
>>> I believe that BoundedSource/UnboundedSource will have wrappers added
>>> that provide a basic SplittableDoFn implementation so existing IOs should
>>> be migrated over without API changes.
>>>
>>> On Mon, Sep 17, 2018 at 1:11 AM Ismaël Mejía  wrote:
>>>
 Thanks a lot Luke for bringing this back to the mailing list and Ryan
 for taking
 the notes.

 I would like to know if there was some discussion, or if you guys have
 given
 some thought to the required changes in the SDK (API) part. What will
 be the
 equivalent of `splitAtFraction` and what should IO authors do to
 support it..

 On Sat, Sep 15, 2018 at 1:52 AM Lukasz Cwik  wrote:
 >
 > Thanks to everyone who joined and for the questions asked.
 >
 > Ryan graciously collected notes of the discussion:
 https://docs.google.com/document/d/1kjJLGIiNAGvDiUCMEtQbw8tyOXESvwGeGZLL-0M06fQ/edit?usp=sharing
 >
 > The summary was that bringing BoundedSource/UnboundedSource into
 using a unified backlog-reporting mechanism with optional other signals
 that Dataflow has found useful (such as is the remaining restriction
 splittable (yes, no, unknown)). Other runners can use or not. SDFs should
 report backlog and watermark as minimum bar. The backlog should use an
 arbitrary precision float such as Java BigDecimal to prevent issues where
 limited precision removes the ability to compute delta efficiently.
 >
 >
 >
 > On Wed, Sep 12, 2018 at 3:54 PM Lukasz Cwik  wrote:
 >>
 >> Here is the link to join the discussion:
 https://meet.google.com/idc-japs-hwf
 >> Remember that it is this Friday Sept 14th from 11am-noon PST.
 >>
 >>
 >>
 >> On Mon, Sep 10, 2018 at 7:30 AM Maximilian Michels 
 

Re: SplittableDoFn

[Reuven Lax]

Can you give details on what the synchronization is per? Is it per key, or
global to each worker?

On Fri, Sep 21, 2018 at 2:10 PM Lukasz Cwik  wrote:

> As I was looking at the SplittableDoFn API while working towards making a
> proposal for how the backlog/splitting API could look, I found some sharp
> edges that could be improved.
>
> I noticed that:
> 1) We require users to write thread safe code, this is something that we
> haven't asked of users when writing a DoFn.
> 2) We "internal" methods within the RestrictionTracker that are not meant
> to be used by the runner.
>
> I can fix these issues by giving the user a forwarding restriction
> tracker[1] that provides an appropriate level of synchronization as needed
> and also provides the necessary observation hooks to see when a claim
> failed or succeeded.
>
> This requires a change to our experimental API since we need to pass
> a RestrictionTracker to the @ProcessElement method instead of a sub-type of
> RestrictionTracker.
> @ProcessElement
> processElement(ProcessContext context, OffsetRangeTracker tracker) { ... }
> becomes:
> @ProcessElement
> processElement(ProcessContext context, RestrictionTracker Long> tracker) { ... }
>
> This provides an additional benefit that it prevents users from working
> around the RestrictionTracker APIs and potentially making underlying
> changes to the tracker outside of the tryClaim call.
>
> Full implementation is available within this PR[2] and was wondering what
> people thought.
>
> 1:
> https://github.com/apache/beam/pull/6467/files#diff-ed95abb6bc30a9ed07faef5c3fea93f0R72
> 2: https://github.com/apache/beam/pull/6467
>
>
> On Mon, Sep 17, 2018 at 12:45 PM Lukasz Cwik  wrote:
>
>> The changes to the API have not been proposed yet. So far it has all been
>> about what is the representation and why.
>>
>> For splitting, the current idea has been about using the backlog as a way
>> of telling the SplittableDoFn where to split, so it would be in terms of
>> whatever the SDK decided to report.
>> The runner always chooses a number for backlog that is relative to the
>> SDKs reported backlog. It would be upto the SDK to round/clamp the number
>> given by the Runner to represent something meaningful for itself.
>> For example if the backlog that the SDK was reporting was bytes remaining
>> in a file such as 500, then the Runner could provide some value like 212.2
>> which the SDK would then round to 212.
>> If the backlog that the SDK was reporting 57 pubsub messages, then the
>> Runner could provide a value like 300 which would mean to read 57 values
>> and then another 243 as part of the current restriction.
>>
>> I believe that BoundedSource/UnboundedSource will have wrappers added
>> that provide a basic SplittableDoFn implementation so existing IOs should
>> be migrated over without API changes.
>>
>> On Mon, Sep 17, 2018 at 1:11 AM Ismaël Mejía  wrote:
>>
>>> Thanks a lot Luke for bringing this back to the mailing list and Ryan
>>> for taking
>>> the notes.
>>>
>>> I would like to know if there was some discussion, or if you guys have
>>> given
>>> some thought to the required changes in the SDK (API) part. What will be
>>> the
>>> equivalent of `splitAtFraction` and what should IO authors do to support
>>> it..
>>>
>>> On Sat, Sep 15, 2018 at 1:52 AM Lukasz Cwik  wrote:
>>> >
>>> > Thanks to everyone who joined and for the questions asked.
>>> >
>>> > Ryan graciously collected notes of the discussion:
>>> https://docs.google.com/document/d/1kjJLGIiNAGvDiUCMEtQbw8tyOXESvwGeGZLL-0M06fQ/edit?usp=sharing
>>> >
>>> > The summary was that bringing BoundedSource/UnboundedSource into using
>>> a unified backlog-reporting mechanism with optional other signals that
>>> Dataflow has found useful (such as is the remaining restriction splittable
>>> (yes, no, unknown)). Other runners can use or not. SDFs should report
>>> backlog and watermark as minimum bar. The backlog should use an arbitrary
>>> precision float such as Java BigDecimal to prevent issues where limited
>>> precision removes the ability to compute delta efficiently.
>>> >
>>> >
>>> >
>>> > On Wed, Sep 12, 2018 at 3:54 PM Lukasz Cwik  wrote:
>>> >>
>>> >> Here is the link to join the discussion:
>>> https://meet.google.com/idc-japs-hwf
>>> >> Remember that it is this Friday Sept 14th from 11am-noon PST.
>>> >>
>>> >>
>>> >>
>>> >> On Mon, Sep 10, 2018 at 7:30 AM Maximilian Michels 
>>> wrote:
>>> >>>
>>> >>> Thanks for moving forward with this, Lukasz!
>>> >>>
>>> >>> Unfortunately, can't make it on Friday but I'll sync with somebody on
>>> >>> the call (e.g. Ryan) about your discussion.
>>> >>>
>>> >>> On 08.09.18 02:00, Lukasz Cwik wrote:
>>> >>> > Thanks for everyone who wanted to fill out the doodle poll. The
>>> most
>>> >>> > popular time was Friday Sept 14th from 11am-noon PST. I'll send
>>> out a
>>> >>> > calendar invite and meeting link early next week.
>>> >>> >
>>> >>> > I have received a lot of feedback on the document and have
>>> addressed

Re: Python PreCommit broken

[Charles Chen]

I can speak to the breakage from https://github.com/apache/beam/pull/6151:
here, the root cause was that an interface in an unmodified file was
refactored in the interim.

On Fri, Sep 21, 2018 at 2:26 PM Ryan Williams  wrote:

>
> On Fri, Sep 21, 2018 at 5:08 PM Robert Burke  wrote:
>
>> The issue is time from commit to merge, and without manual intervention,
>> commits from other PRs aren't accounted for, if there's a lag between LGTM
>> and merge.
>>
>
> it's not between LGTM and merge, it's between [last commit to the PR's
> branch] and merge, right?
>
>
>>
>> On Fri, Sep 21, 2018, 1:52 PM Ahmet Altay  wrote:
>>
>>> I will suggest a rollback in this case, and in general as a good
>>> practice to unblock people.
>>>
>>> On Fri, Sep 21, 2018 at 1:02 PM, Charles Chen  wrote:
>>>
 Relatedly, https://github.com/apache/beam/pull/6151 also recently
 broke the build (
 https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1503/console)
 because the Precommits were very out of date when merged.

>>>
>>> Does not github change the test signals from green to yellow when new
>>> commits are added?
>>>
>>
> it does this if you add commits to your PR's branch, not if commits happen
> upstream.
>
> in the latter case, github only checks whether your PR can be merged
> cleanly (i.e. with no merge-conflicts) according to some line-diffing
> heuristic (presumably the same as git uses), but not whether the tests
> would still pass post-merge.
>
> the unwritten assumption is that a non-conflicting merge with upstream
> won't break any tests, but obviously there can be false-positives; thanks
> for documenting those cases, Valentyn and Charles.
>
> it would be nice to run precommits on potential post-merge commits, but
> i'd guess that our precommits take longer than the average time between
> upstream commits, so it would be hard to get anything merged that way!
>
> we'd need to get smarter about knowing exactly which tests need re-running
> when new commits happen upstream, and only re-run those tests, which we're
> pretty far from today.
>
> so, the possibility of this kind of breakage is always present,
> unfortunately.
>
> i would be curious to see exactly how the line-diffing heuristic was
> fooled in these cases. an easy way to construct such a case is to have one
> side rename a variable, and another side add a new reference to the
> variable's old name, in a newly-created file; each side can touch a
> disjoint set of files and have all its tests pass in isolation, and
> git{,hub} will let you merge them, but the tests will fail post-merge.
>
>
>>
>>>
>>>

 On Fri, Sep 21, 2018 at 12:50 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> The change https://github.com/apache/beam/pull/6424 was not deemed
> particularly risky, and it's purpose was adding more tests to precommit
> test suite.
> There was a green Precommit signal on Jenkins, and I believe
> Postcommit test suite (at the same time) wouldn't catch this.
>
> The reason the breakage was introduced is that
> https://github.com/apache/beam/commit/7689f12db5 was committed after
> the PR 6424 was reviewed, but before it was merged. A combination of both
> introduced the breakage.
>
> Had we re-run the tests closer to the merge, we should have caught
> this. Can we automatically re-run precommits tests at merge time,  when
> some of the files  a PR is touching have changed since last precommit run?
>
> I suggest we proceed with https://github.com/apache/beam/pull/6464 or
> revert  https://github.com/apache/beam/pull/6424 in the mean time,
> while we are iterating on the fix.
>
> On Fri, Sep 21, 2018 at 11:41 AM Charles Chen  wrote:
>
>> Do we happen to know the root cause for why this wasn't caught during
>> review / precommit?
>>
>> In the future, can we run manually run postcommits for risky changes
>> like these?  That is, trigger it by commenting "Run Python PostCommit"?
>>
>> On Fri, Sep 21, 2018 at 10:10 AM Pablo Estrada 
>> wrote:
>>
>>> Robbe has prepared a better fix on
>>> https://github.com/apache/beam/pull/6465
>>> Hopefully that'll be the last of this breakage : )
>>> -P.
>>>
>>> On Fri, Sep 21, 2018 at 9:13 AM Jean-Baptiste Onofré <
>>> j...@nanthrax.net> wrote:
>>>
 By the way, it fails for me on my machine as well.

 Regards
 JB

 On 21/09/2018 18:10, Pablo Estrada wrote:
 > I investigated. This failure comes from the activation of
 > apache_beam.pipeline_test in Python 3 unit tests.
 >
 > I have https://github.com/apache/beam/pull/6464 out to fix this.
 >
 > In any case, it's very exciting that we have some unit tests
 running on
 > Py3 : )
 > Best
 > -P.
 >
 > On Fri, Sep 21, 2018 at 

Re: Python PreCommit broken

[Ryan Williams]

On Fri, Sep 21, 2018 at 5:08 PM Robert Burke  wrote:

> The issue is time from commit to merge, and without manual intervention,
> commits from other PRs aren't accounted for, if there's a lag between LGTM
> and merge.
>

it's not between LGTM and merge, it's between [last commit to the PR's
branch] and merge, right?


>
> On Fri, Sep 21, 2018, 1:52 PM Ahmet Altay  wrote:
>
>> I will suggest a rollback in this case, and in general as a good practice
>> to unblock people.
>>
>> On Fri, Sep 21, 2018 at 1:02 PM, Charles Chen  wrote:
>>
>>> Relatedly, https://github.com/apache/beam/pull/6151 also recently broke
>>> the build (
>>> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1503/console)
>>> because the Precommits were very out of date when merged.
>>>
>>
>> Does not github change the test signals from green to yellow when new
>> commits are added?
>>
>
it does this if you add commits to your PR's branch, not if commits happen
upstream.

in the latter case, github only checks whether your PR can be merged
cleanly (i.e. with no merge-conflicts) according to some line-diffing
heuristic (presumably the same as git uses), but not whether the tests
would still pass post-merge.

the unwritten assumption is that a non-conflicting merge with upstream
won't break any tests, but obviously there can be false-positives; thanks
for documenting those cases, Valentyn and Charles.

it would be nice to run precommits on potential post-merge commits, but i'd
guess that our precommits take longer than the average time between
upstream commits, so it would be hard to get anything merged that way!

we'd need to get smarter about knowing exactly which tests need re-running
when new commits happen upstream, and only re-run those tests, which we're
pretty far from today.

so, the possibility of this kind of breakage is always present,
unfortunately.

i would be curious to see exactly how the line-diffing heuristic was fooled
in these cases. an easy way to construct such a case is to have one side
rename a variable, and another side add a new reference to the variable's
old name, in a newly-created file; each side can touch a disjoint set of
files and have all its tests pass in isolation, and git{,hub} will let you
merge them, but the tests will fail post-merge.


>
>>
>>
>>>
>>> On Fri, Sep 21, 2018 at 12:50 PM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 The change https://github.com/apache/beam/pull/6424 was not deemed
 particularly risky, and it's purpose was adding more tests to precommit
 test suite.
 There was a green Precommit signal on Jenkins, and I believe Postcommit
 test suite (at the same time) wouldn't catch this.

 The reason the breakage was introduced is that
 https://github.com/apache/beam/commit/7689f12db5 was committed after
 the PR 6424 was reviewed, but before it was merged. A combination of both
 introduced the breakage.

 Had we re-run the tests closer to the merge, we should have caught
 this. Can we automatically re-run precommits tests at merge time,  when
 some of the files  a PR is touching have changed since last precommit run?

 I suggest we proceed with https://github.com/apache/beam/pull/6464 or
 revert  https://github.com/apache/beam/pull/6424 in the mean time,
 while we are iterating on the fix.

 On Fri, Sep 21, 2018 at 11:41 AM Charles Chen  wrote:

> Do we happen to know the root cause for why this wasn't caught during
> review / precommit?
>
> In the future, can we run manually run postcommits for risky changes
> like these?  That is, trigger it by commenting "Run Python PostCommit"?
>
> On Fri, Sep 21, 2018 at 10:10 AM Pablo Estrada 
> wrote:
>
>> Robbe has prepared a better fix on
>> https://github.com/apache/beam/pull/6465
>> Hopefully that'll be the last of this breakage : )
>> -P.
>>
>> On Fri, Sep 21, 2018 at 9:13 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> By the way, it fails for me on my machine as well.
>>>
>>> Regards
>>> JB
>>>
>>> On 21/09/2018 18:10, Pablo Estrada wrote:
>>> > I investigated. This failure comes from the activation of
>>> > apache_beam.pipeline_test in Python 3 unit tests.
>>> >
>>> > I have https://github.com/apache/beam/pull/6464 out to fix this.
>>> >
>>> > In any case, it's very exciting that we have some unit tests
>>> running on
>>> > Py3 : )
>>> > Best
>>> > -P.
>>> >
>>> > On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels >> > > wrote:
>>> >
>>> > Hi,
>>> >
>>> > The Python PreCommit tests are currently broken. Is anybody
>>> looking
>>> > into
>>> > this?
>>> >
>>> > Example:
>>> >
>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
>>> > JIRA: 

Re: Python SDK: .options deprecation

[Sindy Li]

I think it's just deprecating referencing the option as
   pipeline.options
, because it is made into a private variable, but we can still do
   pipeline.options()

On Fri, Sep 21, 2018 at 2:11 PM Udi Meiri  wrote:

> Hey, does anybody know why the pipeline.options property was deprecated?
> I found this bug: https://issues.apache.org/jira/browse/BEAM-2124
> but there's no explanation.
>

Python SDK: .options deprecation

[Udi Meiri]

Hey, does anybody know why the pipeline.options property was deprecated?
I found this bug: https://issues.apache.org/jira/browse/BEAM-2124
but there's no explanation.


smime.p7s
Description: S/MIME Cryptographic Signature

Re: SplittableDoFn

[Lukasz Cwik]

As I was looking at the SplittableDoFn API while working towards making a
proposal for how the backlog/splitting API could look, I found some sharp
edges that could be improved.

I noticed that:
1) We require users to write thread safe code, this is something that we
haven't asked of users when writing a DoFn.
2) We "internal" methods within the RestrictionTracker that are not meant
to be used by the runner.

I can fix these issues by giving the user a forwarding restriction
tracker[1] that provides an appropriate level of synchronization as needed
and also provides the necessary observation hooks to see when a claim
failed or succeeded.

This requires a change to our experimental API since we need to pass
a RestrictionTracker to the @ProcessElement method instead of a sub-type of
RestrictionTracker.
@ProcessElement
processElement(ProcessContext context, OffsetRangeTracker tracker) { ... }
becomes:
@ProcessElement
processElement(ProcessContext context, RestrictionTracker tracker) { ... }

This provides an additional benefit that it prevents users from working
around the RestrictionTracker APIs and potentially making underlying
changes to the tracker outside of the tryClaim call.

Full implementation is available within this PR[2] and was wondering what
people thought.

1:
https://github.com/apache/beam/pull/6467/files#diff-ed95abb6bc30a9ed07faef5c3fea93f0R72
2: https://github.com/apache/beam/pull/6467


On Mon, Sep 17, 2018 at 12:45 PM Lukasz Cwik  wrote:

> The changes to the API have not been proposed yet. So far it has all been
> about what is the representation and why.
>
> For splitting, the current idea has been about using the backlog as a way
> of telling the SplittableDoFn where to split, so it would be in terms of
> whatever the SDK decided to report.
> The runner always chooses a number for backlog that is relative to the
> SDKs reported backlog. It would be upto the SDK to round/clamp the number
> given by the Runner to represent something meaningful for itself.
> For example if the backlog that the SDK was reporting was bytes remaining
> in a file such as 500, then the Runner could provide some value like 212.2
> which the SDK would then round to 212.
> If the backlog that the SDK was reporting 57 pubsub messages, then the
> Runner could provide a value like 300 which would mean to read 57 values
> and then another 243 as part of the current restriction.
>
> I believe that BoundedSource/UnboundedSource will have wrappers added that
> provide a basic SplittableDoFn implementation so existing IOs should be
> migrated over without API changes.
>
> On Mon, Sep 17, 2018 at 1:11 AM Ismaël Mejía  wrote:
>
>> Thanks a lot Luke for bringing this back to the mailing list and Ryan for
>> taking
>> the notes.
>>
>> I would like to know if there was some discussion, or if you guys have
>> given
>> some thought to the required changes in the SDK (API) part. What will be
>> the
>> equivalent of `splitAtFraction` and what should IO authors do to support
>> it..
>>
>> On Sat, Sep 15, 2018 at 1:52 AM Lukasz Cwik  wrote:
>> >
>> > Thanks to everyone who joined and for the questions asked.
>> >
>> > Ryan graciously collected notes of the discussion:
>> https://docs.google.com/document/d/1kjJLGIiNAGvDiUCMEtQbw8tyOXESvwGeGZLL-0M06fQ/edit?usp=sharing
>> >
>> > The summary was that bringing BoundedSource/UnboundedSource into using
>> a unified backlog-reporting mechanism with optional other signals that
>> Dataflow has found useful (such as is the remaining restriction splittable
>> (yes, no, unknown)). Other runners can use or not. SDFs should report
>> backlog and watermark as minimum bar. The backlog should use an arbitrary
>> precision float such as Java BigDecimal to prevent issues where limited
>> precision removes the ability to compute delta efficiently.
>> >
>> >
>> >
>> > On Wed, Sep 12, 2018 at 3:54 PM Lukasz Cwik  wrote:
>> >>
>> >> Here is the link to join the discussion:
>> https://meet.google.com/idc-japs-hwf
>> >> Remember that it is this Friday Sept 14th from 11am-noon PST.
>> >>
>> >>
>> >>
>> >> On Mon, Sep 10, 2018 at 7:30 AM Maximilian Michels 
>> wrote:
>> >>>
>> >>> Thanks for moving forward with this, Lukasz!
>> >>>
>> >>> Unfortunately, can't make it on Friday but I'll sync with somebody on
>> >>> the call (e.g. Ryan) about your discussion.
>> >>>
>> >>> On 08.09.18 02:00, Lukasz Cwik wrote:
>> >>> > Thanks for everyone who wanted to fill out the doodle poll. The most
>> >>> > popular time was Friday Sept 14th from 11am-noon PST. I'll send out
>> a
>> >>> > calendar invite and meeting link early next week.
>> >>> >
>> >>> > I have received a lot of feedback on the document and have addressed
>> >>> > some parts of it including:
>> >>> > * clarifying terminology
>> >>> > * processing skew due to some restrictions having their watermarks
>> much
>> >>> > further behind then others affecting scheduling of bundles by
>> runners
>> >>> > * external throttling & I/O wait overhead reporting to make 

Re: Python PreCommit broken

[Robert Burke]

The issue is time from commit to merge, and without manual intervention,
commits from other PRs aren't accounted for, if there's a lag between LGTM
and merge.

On Fri, Sep 21, 2018, 1:52 PM Ahmet Altay  wrote:

> I will suggest a rollback in this case, and in general as a good practice
> to unblock people.
>
> On Fri, Sep 21, 2018 at 1:02 PM, Charles Chen  wrote:
>
>> Relatedly, https://github.com/apache/beam/pull/6151 also recently broke
>> the build (
>> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1503/console)
>> because the Precommits were very out of date when merged.
>>
>
> Does not github change the test signals from green to yellow when new
> commits are added?
>
>
>>
>> On Fri, Sep 21, 2018 at 12:50 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> The change https://github.com/apache/beam/pull/6424 was not deemed
>>> particularly risky, and it's purpose was adding more tests to precommit
>>> test suite.
>>> There was a green Precommit signal on Jenkins, and I believe Postcommit
>>> test suite (at the same time) wouldn't catch this.
>>>
>>> The reason the breakage was introduced is that
>>> https://github.com/apache/beam/commit/7689f12db5 was committed after
>>> the PR 6424 was reviewed, but before it was merged. A combination of both
>>> introduced the breakage.
>>>
>>> Had we re-run the tests closer to the merge, we should have caught this.
>>> Can we automatically re-run precommits tests at merge time,  when some of
>>> the files  a PR is touching have changed since last precommit run?
>>>
>>> I suggest we proceed with https://github.com/apache/beam/pull/6464 or
>>> revert  https://github.com/apache/beam/pull/6424 in the mean time,
>>> while we are iterating on the fix.
>>>
>>> On Fri, Sep 21, 2018 at 11:41 AM Charles Chen  wrote:
>>>
 Do we happen to know the root cause for why this wasn't caught during
 review / precommit?

 In the future, can we run manually run postcommits for risky changes
 like these?  That is, trigger it by commenting "Run Python PostCommit"?

 On Fri, Sep 21, 2018 at 10:10 AM Pablo Estrada 
 wrote:

> Robbe has prepared a better fix on
> https://github.com/apache/beam/pull/6465
> Hopefully that'll be the last of this breakage : )
> -P.
>
> On Fri, Sep 21, 2018 at 9:13 AM Jean-Baptiste Onofré 
> wrote:
>
>> By the way, it fails for me on my machine as well.
>>
>> Regards
>> JB
>>
>> On 21/09/2018 18:10, Pablo Estrada wrote:
>> > I investigated. This failure comes from the activation of
>> > apache_beam.pipeline_test in Python 3 unit tests.
>> >
>> > I have https://github.com/apache/beam/pull/6464 out to fix this.
>> >
>> > In any case, it's very exciting that we have some unit tests
>> running on
>> > Py3 : )
>> > Best
>> > -P.
>> >
>> > On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels > > > wrote:
>> >
>> > Hi,
>> >
>> > The Python PreCommit tests are currently broken. Is anybody
>> looking
>> > into
>> > this?
>> >
>> > Example:
>> >
>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
>> > JIRA: https://issues.apache.org/jira/browse/BEAM-5458
>> >
>> > I'm sure this was just an accident. No big deal but let's make
>> sure
>> > PreCommit passes when merging. A broken PreCommit means that we
>> can't
>> > merge any other changes with confidence.
>> >
>> > Thanks,
>> > Max
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>

Re: Tracking and resolving release blocking bugs

[Ahmet Altay]

I agree with Ismaël, this process is generally working well. As an
improvement we could document it somewhere.

One other area I think we can improve is, issues that are related to the
release process. For example, at times we will have issues that require
doing additional manual steps in the release process, we will file a JIRA
and we tend to punt those JIRAs release after release as long as there is a
manual work around. However, these issues add to the cost of cutting
releases. I would propose prioritizing those issues that are identified
during the release and about the release process itself.

Ahmet

On Wed, Sep 19, 2018 at 4:41 AM, Ismaël Mejía  wrote:

> We have done this so far by letting the JIRA issues 'Open' with the
> 'Fix version' corresponding to the upcoming release and we track the
> progress between the branch cut and the first release candidate with
> the assigned parties, the process has been the de-facto standard since
> long time ago and has worked so far smoothly. More info here:
>
> https://beam.apache.org/contribute/release-guide/#triage-release-blocking-
> issues-in-jira
>
> Is there something missing? or do you have other ideas maybe to
> improve it in mind?
>
> On Wed, Sep 19, 2018 at 2:34 AM Connell O'Callaghan 
> wrote:
> >
> > Hi All
> >
> > In order to allow successful and smooth deployment of the latest BEAM
> releases, are the community OK that we track bugs blocking releases, with a
> goal to resolve such bugs within a week? If there is general agreement (or
> no major objections) on this we will edit the contributor page using
> similar language to the "Stale pull requests" section -early next week.
> >
> > Thank you all,
> > - Connell
>

Re: Python PreCommit broken

[Ahmet Altay]

I will suggest a rollback in this case, and in general as a good practice
to unblock people.

On Fri, Sep 21, 2018 at 1:02 PM, Charles Chen  wrote:

> Relatedly, https://github.com/apache/beam/pull/6151 also recently broke
> the build (https://builds.apache.org/view/A-D/view/Beam/job/beam_
> PostCommit_Java_GradleBuild/1503/console) because the Precommits were
> very out of date when merged.
>

Does not github change the test signals from green to yellow when new
commits are added?


>
> On Fri, Sep 21, 2018 at 12:50 PM Valentyn Tymofieiev 
> wrote:
>
>> The change https://github.com/apache/beam/pull/6424 was not deemed
>> particularly risky, and it's purpose was adding more tests to precommit
>> test suite.
>> There was a green Precommit signal on Jenkins, and I believe Postcommit
>> test suite (at the same time) wouldn't catch this.
>>
>> The reason the breakage was introduced is that https://github.com/
>> apache/beam/commit/7689f12db5 was committed after the PR 6424 was
>> reviewed, but before it was merged. A combination of both introduced the
>> breakage.
>>
>> Had we re-run the tests closer to the merge, we should have caught this.
>> Can we automatically re-run precommits tests at merge time,  when some of
>> the files  a PR is touching have changed since last precommit run?
>>
>> I suggest we proceed with https://github.com/apache/beam/pull/6464 or
>> revert  https://github.com/apache/beam/pull/6424 in the mean time, while
>> we are iterating on the fix.
>>
>> On Fri, Sep 21, 2018 at 11:41 AM Charles Chen  wrote:
>>
>>> Do we happen to know the root cause for why this wasn't caught during
>>> review / precommit?
>>>
>>> In the future, can we run manually run postcommits for risky changes
>>> like these?  That is, trigger it by commenting "Run Python PostCommit"?
>>>
>>> On Fri, Sep 21, 2018 at 10:10 AM Pablo Estrada 
>>> wrote:
>>>
 Robbe has prepared a better fix on https://github.com/apache/
 beam/pull/6465
 Hopefully that'll be the last of this breakage : )
 -P.

 On Fri, Sep 21, 2018 at 9:13 AM Jean-Baptiste Onofré 
 wrote:

> By the way, it fails for me on my machine as well.
>
> Regards
> JB
>
> On 21/09/2018 18:10, Pablo Estrada wrote:
> > I investigated. This failure comes from the activation of
> > apache_beam.pipeline_test in Python 3 unit tests.
> >
> > I have https://github.com/apache/beam/pull/6464 out to fix this.
> >
> > In any case, it's very exciting that we have some unit tests running
> on
> > Py3 : )
> > Best
> > -P.
> >
> > On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels  > > wrote:
> >
> > Hi,
> >
> > The Python PreCommit tests are currently broken. Is anybody
> looking
> > into
> > this?
> >
> > Example:
> > https://builds.apache.org/job/beam_PreCommit_Python_
> Commit/1308/#showFailuresLink
> > JIRA: https://issues.apache.org/jira/browse/BEAM-5458
> >
> > I'm sure this was just an accident. No big deal but let's make
> sure
> > PreCommit passes when merging. A broken PreCommit means that we
> can't
> > merge any other changes with confidence.
> >
> > Thanks,
> > Max
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Python PreCommit broken

[Charles Chen]

Relatedly, https://github.com/apache/beam/pull/6151 also recently broke the
build (
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1503/console)
because the Precommits were very out of date when merged.

On Fri, Sep 21, 2018 at 12:50 PM Valentyn Tymofieiev 
wrote:

> The change https://github.com/apache/beam/pull/6424 was not deemed
> particularly risky, and it's purpose was adding more tests to precommit
> test suite.
> There was a green Precommit signal on Jenkins, and I believe Postcommit
> test suite (at the same time) wouldn't catch this.
>
> The reason the breakage was introduced is that
> https://github.com/apache/beam/commit/7689f12db5 was committed after the
> PR 6424 was reviewed, but before it was merged. A combination of both
> introduced the breakage.
>
> Had we re-run the tests closer to the merge, we should have caught this.
> Can we automatically re-run precommits tests at merge time,  when some of
> the files  a PR is touching have changed since last precommit run?
>
> I suggest we proceed with https://github.com/apache/beam/pull/6464 or
> revert  https://github.com/apache/beam/pull/6424 in the mean time, while
> we are iterating on the fix.
>
> On Fri, Sep 21, 2018 at 11:41 AM Charles Chen  wrote:
>
>> Do we happen to know the root cause for why this wasn't caught during
>> review / precommit?
>>
>> In the future, can we run manually run postcommits for risky changes like
>> these?  That is, trigger it by commenting "Run Python PostCommit"?
>>
>> On Fri, Sep 21, 2018 at 10:10 AM Pablo Estrada 
>> wrote:
>>
>>> Robbe has prepared a better fix on
>>> https://github.com/apache/beam/pull/6465
>>> Hopefully that'll be the last of this breakage : )
>>> -P.
>>>
>>> On Fri, Sep 21, 2018 at 9:13 AM Jean-Baptiste Onofré 
>>> wrote:
>>>
 By the way, it fails for me on my machine as well.

 Regards
 JB

 On 21/09/2018 18:10, Pablo Estrada wrote:
 > I investigated. This failure comes from the activation of
 > apache_beam.pipeline_test in Python 3 unit tests.
 >
 > I have https://github.com/apache/beam/pull/6464 out to fix this.
 >
 > In any case, it's very exciting that we have some unit tests running
 on
 > Py3 : )
 > Best
 > -P.
 >
 > On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels >>> > > wrote:
 >
 > Hi,
 >
 > The Python PreCommit tests are currently broken. Is anybody
 looking
 > into
 > this?
 >
 > Example:
 >
 https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
 > JIRA: https://issues.apache.org/jira/browse/BEAM-5458
 >
 > I'm sure this was just an accident. No big deal but let's make
 sure
 > PreCommit passes when merging. A broken PreCommit means that we
 can't
 > merge any other changes with confidence.
 >
 > Thanks,
 > Max
 >

 --
 Jean-Baptiste Onofré
 jbono...@apache.org
 http://blog.nanthrax.net
 Talend - http://www.talend.com

>>>

Re: Python PreCommit broken

[Valentyn Tymofieiev]

The change https://github.com/apache/beam/pull/6424 was not deemed
particularly risky, and it's purpose was adding more tests to precommit
test suite.
There was a green Precommit signal on Jenkins, and I believe Postcommit
test suite (at the same time) wouldn't catch this.

The reason the breakage was introduced is that
https://github.com/apache/beam/commit/7689f12db5 was committed after the PR
6424 was reviewed, but before it was merged. A combination of both
introduced the breakage.

Had we re-run the tests closer to the merge, we should have caught this.
Can we automatically re-run precommits tests at merge time,  when some of
the files  a PR is touching have changed since last precommit run?

I suggest we proceed with https://github.com/apache/beam/pull/6464 or
revert  https://github.com/apache/beam/pull/6424 in the mean time, while we
are iterating on the fix.

On Fri, Sep 21, 2018 at 11:41 AM Charles Chen  wrote:

> Do we happen to know the root cause for why this wasn't caught during
> review / precommit?
>
> In the future, can we run manually run postcommits for risky changes like
> these?  That is, trigger it by commenting "Run Python PostCommit"?
>
> On Fri, Sep 21, 2018 at 10:10 AM Pablo Estrada  wrote:
>
>> Robbe has prepared a better fix on
>> https://github.com/apache/beam/pull/6465
>> Hopefully that'll be the last of this breakage : )
>> -P.
>>
>> On Fri, Sep 21, 2018 at 9:13 AM Jean-Baptiste Onofré 
>> wrote:
>>
>>> By the way, it fails for me on my machine as well.
>>>
>>> Regards
>>> JB
>>>
>>> On 21/09/2018 18:10, Pablo Estrada wrote:
>>> > I investigated. This failure comes from the activation of
>>> > apache_beam.pipeline_test in Python 3 unit tests.
>>> >
>>> > I have https://github.com/apache/beam/pull/6464 out to fix this.
>>> >
>>> > In any case, it's very exciting that we have some unit tests running on
>>> > Py3 : )
>>> > Best
>>> > -P.
>>> >
>>> > On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels >> > > wrote:
>>> >
>>> > Hi,
>>> >
>>> > The Python PreCommit tests are currently broken. Is anybody looking
>>> > into
>>> > this?
>>> >
>>> > Example:
>>> >
>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
>>> > JIRA: https://issues.apache.org/jira/browse/BEAM-5458
>>> >
>>> > I'm sure this was just an accident. No big deal but let's make sure
>>> > PreCommit passes when merging. A broken PreCommit means that we
>>> can't
>>> > merge any other changes with confidence.
>>> >
>>> > Thanks,
>>> > Max
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbono...@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>

Re: Metrics Pusher support on Dataflow

[Scott Wegner]

Hi Etienne, sorry for the delay on this. I just got back from leave and
found this discussion.

We haven't started implementing MetricsPusher in the Dataflow runner,
mostly because the Dataflow service has it's own rich Metrics REST API and
we haven't heard a need from Dataflow customers to push metrics to an
external backend. However, it would be nice to have this implemented across
all runners for feature parity.

I read through the discussion in JIRA [1], and the simplest implementation
for Dataflow may be to have a single thread periodically poll the Dataflow
REST API [2] for latest metric values, and push them to a configured sink.
This polling thread could be hosted in a separate docker container, within
the worker process, or perhaps a ParDo with timers that gets injected into
the pipeline during graph translation.

At any rate, I'm not aware of anybody currently working on this. But with
the Dataflow worker code being donated to Beam [3], soon it will be
possible for anybody to contribute.

[1] https://issues.apache.org/jira/browse/BEAM-3926
[2]
https://cloud.google.com/dataflow/docs/reference/rest/v1b3/projects.locations.jobs/getMetrics
[3]
https://lists.apache.org/thread.html/2bdc645659e2fbd7e29f3a2758941faefedb01148a2a11558dfe60f8@%3Cdev.beam.apache.org%3E

On Fri, Aug 17, 2018 at 4:26 PM Lukasz Cwik  wrote:

> I forwarded your request to a few people who work on the internal parts of
> Dataflow to see if they could help in some way.
>
> On Thu, Aug 16, 2018 at 6:22 AM Etienne Chauchot 
> wrote:
>
>> Hi all
>>
>> As we already discussed, it would be good to support Metrics Pusher [1]
>> in Dataflow (in other runners also, of course). Today, only Spark and Flink
>> support it. It requires a modification in C++ Dataflow code, so only Google
>> friends can do it.
>>
>> Is someone interested in doing it ?
>>
>> Here is the ticket https://issues.apache.org/jira/browse/BEAM-3926
>>
>> Besides, I wonder if this feature should be added to the capability
>> matrix.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/BEAM/Metrics+architecture+inside+the+runners
>>
>> Thanks
>> Etienne
>>
>

-- 




Got feedback? tinyurl.com/swegner-feedback

Re: [DISCUSS] Committer Guidelines / Hygene before merging PRs

[Daniel Oliveira]

As a non-committer I think some automated squashing of commits sounds best
since it lightens the load of regular contributors, by not having to always
remember to squash, and lightens the load of committers so it doesn't take
as long to have your PR approved by one.

But for now I think the second best route would be making it PR author's
responsibility to squash fixup commits. Having that expectation described
clearly in the Contributor's Guide, along with some simple step-by-step
instructions for how to do so should be enough. I mainly support this
because I've been doing the squashing myself since I saw a thread about it
here a few months ago. It's not nearly as huge a burden on me as it
probably is for committers who have to merge in many more PRs, it's very
easy to learn how to do, and it's one less barrier to having my code merged
in.

Of course I wouldn't expect that committers wait for PR authors to squash
their fixup commits, but I think leaving a message like "For future pull
requests you should squash any small fixup commits, as described here:
" should be fine.


> I was also thinking about the possibility of wanting to revert
> individual commits from a merge commit. The solution you propose works,
> but only if you want to revert everything.


Does this happen often? I might not have enough context since I'm not a
committer, but it seems to me that often the person performing a revert is
not the original author of a change and doesn't have the context or time to
pick out an individual commit to revert.

On Wed, Sep 19, 2018 at 1:32 PM Maximilian Michels  wrote:

> I tend to agree with you Lukasz. Of course we should try to follow the
> guide lines as much as possible but if it requires an extra back and
> forth with the PR author for a cosmetic change, it may not be worth the
> time.
>
> On 19.09.18 22:17, Lukasz Cwik wrote:
> > I have to say I'm guilty of not following the merge guidelines,
> > sometimes doing merges without rebasing/flatten commits.
> >
> > I find that it is a few extra mins of my time to fix someones PR history
> > if they have more then one logical commit they want to be separate and
> > it usually takes days for the PR author to do merging  with the extra
> > burden as a committer to keep track of another PR and its state (waiting
> > for clean-up) is taxing. I really liked the idea of the mergebot (even
> > though it didn't work out in practice) because it could do all the
> > policy work on my behalf.
> >
> > Anything that reduces my overhead as a committer is useful as for the
> > 100s of PRs that I have merged, I've only had to rollback a couple so
> > I'm for Charle's suggestion which makes the rollback flow slightly more
> > complicated for a significantly easier PR merge workflow.
> >
> > On Wed, Sep 19, 2018 at 1:13 PM Charles Chen  > > wrote:
> >
> > What I mean is that if you get the first-parent commit using "git
> > log --first-parent", it will incorporate any and all fix up commits
> > so we don't need to worry about missing any.
> >
> > On Wed, Sep 19, 2018, 1:07 PM Maximilian Michels  > > wrote:
> >
> > Generally, +1 for isolated commits which are easy to revert.
> >
> >  > I don't think it's actually harder to roll back a set of
> > commits that are merged together.
> > I think Thomas was mainly concerned about "fixup" commits to
> > land in
> > master (as part of a merge). These indeed make reverting commits
> > more
> > difficult because you have to check whether you missed a "fixup".
> >
> >  > Ideally every commit should compile and pass tests though,
> right?
> >
> > That is definitely what we should strive for when doing a merge
> > against
> > master.
> >
> >  > Perhaps the bigger issue is that we need better documentation
> > and a playbook on how to do this these common tasks in git.
> >
> > We do actually have basic documentation about this but most
> > people don't
> > read it. For example, the commit message of a Merge commit
> > should be:
> >
> > Merge pull request #: [BEAM-] Issue title
> >
> > But most merge commits don't comply with this rule :) See
> > https://beam.apache.org/contribute/committer-guide/#merging-it
> >
> > On 19.09.18 21:34, Reuven Lax wrote:
> >  > Ideally every commit should compile and pass tests though,
> right?
> >  >
> >  > On Wed, Sep 19, 2018 at 12:15 PM Ankur Goenka
> > mailto:goe...@google.com>
> >  > >> wrote:
> >  >
> >  > I agree with the cleanliness of the Commit history.
> >  > "Fixup!", "Address comments", "Address even more
> > comments" type of
> >  > comments does not convey meaningful information and are
> >  

Re: Python PreCommit broken

[Charles Chen]

Do we happen to know the root cause for why this wasn't caught during
review / precommit?

In the future, can we run manually run postcommits for risky changes like
these?  That is, trigger it by commenting "Run Python PostCommit"?

On Fri, Sep 21, 2018 at 10:10 AM Pablo Estrada  wrote:

> Robbe has prepared a better fix on
> https://github.com/apache/beam/pull/6465
> Hopefully that'll be the last of this breakage : )
> -P.
>
> On Fri, Sep 21, 2018 at 9:13 AM Jean-Baptiste Onofré 
> wrote:
>
>> By the way, it fails for me on my machine as well.
>>
>> Regards
>> JB
>>
>> On 21/09/2018 18:10, Pablo Estrada wrote:
>> > I investigated. This failure comes from the activation of
>> > apache_beam.pipeline_test in Python 3 unit tests.
>> >
>> > I have https://github.com/apache/beam/pull/6464 out to fix this.
>> >
>> > In any case, it's very exciting that we have some unit tests running on
>> > Py3 : )
>> > Best
>> > -P.
>> >
>> > On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels > > > wrote:
>> >
>> > Hi,
>> >
>> > The Python PreCommit tests are currently broken. Is anybody looking
>> > into
>> > this?
>> >
>> > Example:
>> >
>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
>> > JIRA: https://issues.apache.org/jira/browse/BEAM-5458
>> >
>> > I'm sure this was just an accident. No big deal but let's make sure
>> > PreCommit passes when merging. A broken PreCommit means that we
>> can't
>> > merge any other changes with confidence.
>> >
>> > Thanks,
>> > Max
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbono...@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>

Re: Removing documentation for old Beam versions

[Scott Wegner]

Re-opening this thread as it came up today in the discussion for PR#6458
[1]. This PR is part of the work for Beam-Site Automation Reliability
improvements; design doc here: https://s.apache.org/beam-site-automation

The current plan is to keep generated javadoc/pydoc sources only on the
asf-site branch, which is necessary for the current githubpubsub publishing
mechanism. This maintains our current approach, the only change being that
we're moving the asf-site branch from the retiring apache/beam-site
repository into a new apache/beam repo branch.

The concern for committing generated content is the extra overhead during
git fetch. I did some analysis to measure the impact [2], and found that
fetching a week of source + generated content history from apache/beam-site
took 0.39 seconds.

I like the idea of publishing javadoc/pydoc snapshots to an external
location like Flink does with buildbot, but that work is separable and
shouldn't be a prerequisite for this effort. The goal of this work is to
improve the reliability of automation for contributing website changes. At
last measure, only about half of beam-site PR merges use Mergebot without
experiencing some reliability issue [3].

I've opened BEAM-5459 [4] to track moving our generated docs out of git.
Thomas, would you have bandwidth to look into this?

[1] https://github.com/apache/beam/pull/6458#issuecomment-423406643
[2]
https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit#heading=h.uqzivheohd7j
[3]
https://docs.google.com/document/d/1lfbMhdIyDzIaBTgc9OUByhSwR94kfOzS_ozwKWTVl5U/edit#heading=h.a208cwi78xmu
[4] https://issues.apache.org/jira/browse/BEAM-5459

On Fri, Aug 24, 2018 at 11:48 AM Thomas Weise  wrote:

> Hi Udi,
>
> Good to know you will continue this work.
>
> Let me know if you want to try the buildbot route (which does not require
> generated documentation to be checked into the repo). Happy to help with
> that.
>
> Thomas
>
> On Fri, Aug 24, 2018 at 11:36 AM Udi Meiri  wrote:
>
>> I'm picking up the website migration. The plan is to not include
>> generated files in the master branch.
>>
>> However, I've been told that even putting generated files a separate
>> branch could blow up the git repository for all (e.g. make git pulls a lot
>> longer?).
>> Not sure if this is a real issue or not.
>>
>> On Mon, Aug 20, 2018 at 2:53 AM Robert Bradshaw 
>> wrote:
>>
>>> On Sun, Aug 5, 2018 at 5:28 AM Thomas Weise  wrote:
>>> >
>>> > Yes, I think the separation of generated code will need to occur prior
>>> to completing the merge and switching the web site to the main repo.
>>> >
>>> > There should be no reason to check generated documentation into either
>>> of the repos/branches.
>>>
>>> Huge +1 to this. Thomas, would have time to set something like this up
>>> for Beam? If not, could anyone else pick this up?
>>>
>>> > Please see as an example how this was solved in Flink, using the ASF
>>> buildbot infrastructure.
>>> >
>>> > Documentation per version/release, for example:
>>> >
>>> > https://ci.apache.org/projects/flink/flink-docs-release-1.5/
>>> >
>>> > The buildbot configuration is here (requires committer access):
>>> >
>>> >
>>> https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/flink.conf
>>> >
>>> > Thanks,
>>> > Thomas
>>> >
>>> > On Thu, Aug 2, 2018 at 6:46 PM Mikhail Gryzykhin 
>>> wrote:
>>> >>
>>> >> Last time I talked with Scott I brought this idea in. I believe the
>>> plan was either to publish compiled site to website directly, or keep it in
>>> separate storage from apache/beam repo.
>>> >>
>>> >> One of the main reasons not to check in compiled version of website
>>> is that every developer will have to pull all the versions of website every
>>> time they clone repo, which is not that good of an idea to do.
>>> >>
>>> >> Regards,
>>> >> --Mikhail
>>> >>
>>> >> Have feedback?
>>> >>
>>> >>
>>> >> On Thu, Aug 2, 2018 at 6:42 PM Udi Meiri  wrote:
>>> >>>
>>> >>> Pablo, the docs are generated into versioned paths, e.g.,
>>> https://beam.apache.org/documentation/sdks/javadoc/2.5.0/ so tags are
>>> not necessary?
>>> >>> Also, once apache/beam-site is merged with apache/beam the release
>>> branch should have the relevant docs (although perhaps it's better to put
>>> them in a different repo or storage system).
>>> >>>
>>> >>> Thomas, I would very much like to not have javadoc/pydoc generation
>>> be part of the website review process, as it takes up a lot of time when
>>> changes are staged (10s of thousands of files), especially when a PR is
>>> updated and existing staged files need to be deleted.
>>> >>>
>>> >>>
>>> >>> On Thu, Aug 2, 2018 at 1:15 PM Mikhail Gryzykhin 
>>> wrote:
>>> 
>>>  +1 For removing old documentation.
>>> 
>>>  @Thomas: Migration work is in backlog and will be picked up in near
>>> time.
>>> 
>>>  --Mikhail
>>> 
>>>  Have feedback?
>>> 
>>> 
>>>  On Thu, Aug 2, 2018 at 12:54 PM Thomas 

Re: Python PreCommit broken

[Pablo Estrada]

Robbe has prepared a better fix on https://github.com/apache/beam/pull/6465
Hopefully that'll be the last of this breakage : )
-P.

On Fri, Sep 21, 2018 at 9:13 AM Jean-Baptiste Onofré 
wrote:

> By the way, it fails for me on my machine as well.
>
> Regards
> JB
>
> On 21/09/2018 18:10, Pablo Estrada wrote:
> > I investigated. This failure comes from the activation of
> > apache_beam.pipeline_test in Python 3 unit tests.
> >
> > I have https://github.com/apache/beam/pull/6464 out to fix this.
> >
> > In any case, it's very exciting that we have some unit tests running on
> > Py3 : )
> > Best
> > -P.
> >
> > On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels  > > wrote:
> >
> > Hi,
> >
> > The Python PreCommit tests are currently broken. Is anybody looking
> > into
> > this?
> >
> > Example:
> >
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
> > JIRA: https://issues.apache.org/jira/browse/BEAM-5458
> >
> > I'm sure this was just an accident. No big deal but let's make sure
> > PreCommit passes when merging. A broken PreCommit means that we can't
> > merge any other changes with confidence.
> >
> > Thanks,
> > Max
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Python PreCommit broken

[Thomas Weise]

Pablo, thanks for taking this up.

It is indeed exciting to see things moving forward on the Python side
(state and timer support as well).

Still, we depend on the green CI status. Just to illustrate how breakage
can negatively affect even folks not immediately working on Python,
consider the portability work: For many changes we need the runner build
(such as Flink) as well as the Python SDK to be in good shape. Either of
the two breaks and the big search begins :)

Thanks,
Thomas


On Fri, Sep 21, 2018 at 9:11 AM Pablo Estrada  wrote:

> I investigated. This failure comes from the activation of
> apache_beam.pipeline_test in Python 3 unit tests.
>
> I have https://github.com/apache/beam/pull/6464 out to fix this.
>
> In any case, it's very exciting that we have some unit tests running on
> Py3 : )
> Best
> -P.
>
> On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels  wrote:
>
>> Hi,
>>
>> The Python PreCommit tests are currently broken. Is anybody looking into
>> this?
>>
>> Example:
>>
>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
>> JIRA: https://issues.apache.org/jira/browse/BEAM-5458
>>
>> I'm sure this was just an accident. No big deal but let's make sure
>> PreCommit passes when merging. A broken PreCommit means that we can't
>> merge any other changes with confidence.
>>
>> Thanks,
>> Max
>>
>

Re: Python PreCommit broken

[Jean-Baptiste Onofré]

By the way, it fails for me on my machine as well.

Regards
JB

On 21/09/2018 18:10, Pablo Estrada wrote:
> I investigated. This failure comes from the activation of
> apache_beam.pipeline_test in Python 3 unit tests.
> 
> I have https://github.com/apache/beam/pull/6464 out to fix this.
> 
> In any case, it's very exciting that we have some unit tests running on
> Py3 : )
> Best
> -P.
> 
> On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels  > wrote:
> 
> Hi,
> 
> The Python PreCommit tests are currently broken. Is anybody looking
> into
> this?
> 
> Example:
> 
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
> JIRA: https://issues.apache.org/jira/browse/BEAM-5458
> 
> I'm sure this was just an accident. No big deal but let's make sure
> PreCommit passes when merging. A broken PreCommit means that we can't
> merge any other changes with confidence.
> 
> Thanks,
> Max
> 

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Python PreCommit broken

[Pablo Estrada]

I investigated. This failure comes from the activation of
apache_beam.pipeline_test in Python 3 unit tests.

I have https://github.com/apache/beam/pull/6464 out to fix this.

In any case, it's very exciting that we have some unit tests running on Py3
: )
Best
-P.

On Fri, Sep 21, 2018 at 6:40 AM Maximilian Michels  wrote:

> Hi,
>
> The Python PreCommit tests are currently broken. Is anybody looking into
> this?
>
> Example:
>
> https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink
> JIRA: https://issues.apache.org/jira/browse/BEAM-5458
>
> I'm sure this was just an accident. No big deal but let's make sure
> PreCommit passes when merging. A broken PreCommit means that we can't
> merge any other changes with confidence.
>
> Thanks,
> Max
>

Re: Rethinking Timers as PCollections

[Maximilian Michels]

Very interesting thread.

Having read the original Timer design document, I find it compelling to 
model timers with a loop from producing to consuming PCollections. This 
makes it very explicit how timers are positioned in the dataflow.


What Robert proposes looks less explicit, yet much closer to how Runner 
authors would go about to implement it. I'm not fully aware of any 
limitations of this model. Lukasz mentioned that we would have to hold 
back the Watermark for as long as the Timer is not yet set, as 
potentially it could have already been passed before set. As for output 
time being different from fire time, I suppose we can add a hold for the 
output watermark before the timer is fired.


Whichever model we pursue, we have to solve the same 
problems/requirements for Timers. It does look like this is more a 
problem of how things are represented in the proto? Practically, the 
runtime implementation looks similar.


If I had to choose I'd probably go for timers being represented as part 
of a spec for a DoFn (which seems to be already the case). Timers as 
separate PCollections seems elegant but less practical to me.


-Max

[Disclaimer: I could be wrong since I just thought about this in more 
detail]


On 20.09.18 00:28, Robert Bradshaw wrote:
On Wed, Sep 19, 2018 at 11:54 PM Lukasz Cwik > wrote:



On Wed, Sep 19, 2018 at 2:46 PM Robert Bradshaw mailto:rober...@google.com>> wrote:

On Wed, Sep 19, 2018 at 8:31 PM Lukasz Cwik mailto:lc...@google.com>> wrote:

*How does modelling a timer as a PCollection help the Beam
model?*

The largest concern was about how to model timers within
Apache Beam that:
1) removed the need for the watermark hold that is typically
accompanied with state/timer implementations
2) enabled the ability to set the explicit output time to be
independent of the firing time for all timer specifications [1]

I felt as though treating timers as a self-loop around the
ParDo PTransform allowed us to use the natural definition of
output watermark = min(all input watermarks) as a way to
define how timers hold output and using windowed values that
contained timers as a natural way to represent the output
time to be independent of the firing time. The purpose of
the PCollection right now is to store the representation of
how timers are encoded. I suspect that at some point in time
we will have different timer encodings.


I agree that being able to separate the hold time from firing
time of a timer is important, but in retrospect don't think
timers as PCollections is the only (or most natural) way to
represent that (in the model or in runner implementations). 


Can you go into more detail as to what your suggesting as the
replacement and why you believe it fits the model more naturally
since "state" doesn't have watermarks or produce output but timers
can. I'm not disagreeing that timers as PCollections may not be a
natural fit but I don't see them as state as well since "user state"
doesn't produce output.


Yeah, timers are their own thing. They come in like data, but are 
written out like state.


I guess the high level is that I think the beam graph should represent, 
within reason, the user's model of what their pipeline is, and treating 
timers as PCollections with this self-loop feels like an implementation 
detail, and furthermore an implementation detail that no runner is 
actually going to use to implement things. And (again, this is 
subjective) seems to complicate both the reasoning about a pipeline and 
implementing its execution over viewing the stateful/timely aspects of a 
DoFn as internal details to the ParDo operation.


Having this fit well with how timers are delivered between
the SDK and Runner was an added bonus. Also, a good portion
of the code that I needed to fix up was more related to the
assumption that there was ever only a single input producer
to an executable stage and plumbing of timer specifications
through all the runner library support layers.

--
*There is no "clear" for timers.*

The current Java API for timers only allows you to set them.
Clearing timers is not exposed to users and is only used by
internal implementations to support runners[2] via
TimerInternals. Usage of a timer is like so:
   @TimerId("timer")
   private final TimerSpec timerSpec =
TimerSpecs.timer(TimeDomain.EVENT_TIME);

   @ProcessElement
   public void process(
       ProcessContext context,
       BoundedWindow window,
       

Re: Compatibility Matrix vs Runners in the code base

[Jean-Baptiste Onofré]

Catcha.

Yes, it makes sense to me, and I agree with Tim: we can use something
similar to the IOs.

Regards
JB

On 21/09/2018 16:06, Maximilian Michels wrote:
>> not sure I fully follow you there.
> 
> @JB I simply meant to ask whether it make sense to have Runners in the
> matrix whose code/documentation is not part of Beam. For it to become a
> part of Beam, it could be as easy as adding a link to the external
> Runner page.
> 
>> I don't know that we need to limit the matrix to runners in the Beam
>> codebase
> 
> @Robert I think we used to only allow Runners in the matrix which were
> in the Beam code base. However, you are right, it is not necessary for
> Runners to live in the Beam repo. But IMHO they should be documented and
> linked before entries to the matrix are made.
> 
>> +1 but perhaps we should having a table listing Runners under
>> development like we do for IOs.
> 
> @Tim Yes, I didn't even know the Hadoop Runner was in a branch.
> 
> 
> I don't want to remove any Runners from the matrix but I propose to
> require some form of documentation on the Beam website in addition to
> the compatibility matrix entry.
> 
> The current state:
> 
> Runners documented
> ==
> 
> Direct Runner
> Apache Apex
> Apache Flink
> Apache Gearpump
> Apache Samza
> Apache Spark
> Google Cloud Dataflow
> 
> Runners according to the matrix
> ===
> 
> Apache Apex
> Apache Flink
> Apache Gearpump
> Apache Samza
> Apache Spark
> Google Cloud Dataflow
> Apache Hadoop MapReduce
> JStorm
> IBM Streams
> 
> 
> If we can fix the diff between these two lists, I'd feel more
> comfortable the next time somebody asks about a Runner I haven't used
> yet :)
> 
> Thanks,
> Max
> 
> On 21.09.18 14:51, Thomas Weise wrote:
>> The MapReduce runner IMHO should not be in the matrix.
>>
>> For the external runners, is there any public documentation available
>> that explains how they can be used and how they are supported?
>>
>> Thanks,
>> Thomas
>>
>>
>> On Fri, Sep 21, 2018 at 5:14 AM Jean-Baptiste Onofré > > wrote:
>>
>>     You are right Tim.
>>
>>     M/R runner is on a branch (in stale for now to be honest ;)).
>>
>>     I think I got Max's remark.
>>
>>     So, agree to focus only Beam coverage in the runner compatibility
>>     matrix. However, it's also important for the community to have some
>>     insights about runners generally speaking.
>>
>>     Regards
>>     JB
>>
>>     On 21/09/2018 14:00, Tim Robertson wrote:
>>  > "what do you think about limiting the matrix to Runners in the
>>     Beam code
>>  > base"
>>  >
>>  > +1 but perhaps we should having a table listing Runners under
>>  > development like we do for IOs.
>>  >
>>  > As a concrete example we have MapReduce listed in the matrix [1],
>>     a page
>>  > documenting it [2] stating it is in Beam 2.6.0 but unless I'm
>>     mistaken
>>  > the code exists only on a branch [3] and hasn't been touched for
>>     a while.
>>  >
>>  > Thanks,
>>  > Tim
>>  >
>>  > [1]
>> https://beam.apache.org/documentation/runners/capability-matrix/
>>  > [2] https://beam.apache.org/documentation/runners/mapreduce/
>>  > [3] https://github.com/apache/beam/tree/mr-runner
>>  >
>>  > On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré
>>     mailto:j...@nanthrax.net>
>>  > >> wrote:
>>  >
>>  >     Hi Max,
>>  >
>>  >     not sure I fully follow you there. You mean that we would
>>     have kind of
>>  >     compability matrix on dedicated page of each runner ?
>>  >
>>  >     Regards
>>  >     JB
>>  >
>>  >     On 21/09/2018 10:57, Maximilian Michels wrote:
>>  >     > Hi Beamers,
>>  >     >
>>  >     > There have been occasions where people asked me about
>>     Runner XY and I
>>  >     > had to find out that it only exists in the compatibility
>>     matrix,
>>  >     but not
>>  >     > as part of our code base. More interestingly, I couldn't
>>     even find its
>>  >     > code or documentation via my favorite search engine.
>>  >     >
>>  >     > This seems to be the case for multiple Runners in the
>> matrix.
>>  >     >
>>  >     > The compatibility matrix will need an overhaul anyways
>> with the
>>  >     > portability changes, but what do you think about limiting
>> the
>>  >     matrix to
>>  >     > Runners in the Beam code base?
>>  >     >
>>  >     > Thanks,
>>  >     > Max
>>  >
>>  >     --
>>  >     Jean-Baptiste Onofré
>>  > jbono...@apache.org 
>>     >
>>  > http://blog.nanthrax.net
>>  >     Talend - http://www.talend.com
>>  >
>>
>>     --     Jean-Baptiste Onofré
>>     jbono...@apache.org 
>> 

Re: Compatibility Matrix vs Runners in the code base

[Alexey Romanenko]

I agree that we should keep all runners in Matrix table but it should be clear 
which runners are part of Beam code base, which are not. In case of non-Beam 
code repo runners we need to provide a link to its documentation. 

MapReduce runner is another case since it’s not yet production ready and it 
lives in separate branch. We need to mention this clearly as well, it took me a 
while to find out where actually this runner
 is.

> On 21 Sep 2018, at 16:06, Maximilian Michels  wrote:
> 
> > not sure I fully follow you there.
> 
> @JB I simply meant to ask whether it make sense to have Runners in the matrix 
> whose code/documentation is not part of Beam. For it to become a part of 
> Beam, it could be as easy as adding a link to the external Runner page.
> 
>> I don't know that we need to limit the matrix to runners in the Beam codebase
> 
> @Robert I think we used to only allow Runners in the matrix which were in the 
> Beam code base. However, you are right, it is not necessary for Runners to 
> live in the Beam repo. But IMHO they should be documented and linked before 
> entries to the matrix are made.
> 
>> +1 but perhaps we should having a table listing Runners under development 
>> like we do for IOs.
> 
> @Tim Yes, I didn't even know the Hadoop Runner was in a branch.
> 
> 
> I don't want to remove any Runners from the matrix but I propose to require 
> some form of documentation on the Beam website in addition to the 
> compatibility matrix entry.
> 
> The current state:
> 
> Runners documented
> ==
> 
> Direct Runner
> Apache Apex
> Apache Flink
> Apache Gearpump
> Apache Samza
> Apache Spark
> Google Cloud Dataflow
> 
> Runners according to the matrix
> ===
> 
> Apache Apex   
> Apache Flink
> Apache Gearpump
> Apache Samza
> Apache Spark  
> Google Cloud Dataflow 
> Apache Hadoop MapReduce   
> JStorm
> IBM Streams
> 
> 
> If we can fix the diff between these two lists, I'd feel more comfortable the 
> next time somebody asks about a Runner I haven't used yet :)
> 
> Thanks,
> Max
> 
> On 21.09.18 14:51, Thomas Weise wrote:
>> The MapReduce runner IMHO should not be in the matrix.
>> For the external runners, is there any public documentation available that 
>> explains how they can be used and how they are supported?
>> Thanks,
>> Thomas
>> On Fri, Sep 21, 2018 at 5:14 AM Jean-Baptiste Onofré > > wrote:
>>You are right Tim.
>>M/R runner is on a branch (in stale for now to be honest ;)).
>>I think I got Max's remark.
>>So, agree to focus only Beam coverage in the runner compatibility
>>matrix. However, it's also important for the community to have some
>>insights about runners generally speaking.
>>Regards
>>JB
>>On 21/09/2018 14:00, Tim Robertson wrote:
>> > "what do you think about limiting the matrix to Runners in the
>>Beam code
>> > base"
>> >
>> > +1 but perhaps we should having a table listing Runners under
>> > development like we do for IOs.
>> >
>> > As a concrete example we have MapReduce listed in the matrix [1],
>>a page
>> > documenting it [2] stating it is in Beam 2.6.0 but unless I'm
>>mistaken
>> > the code exists only on a branch [3] and hasn't been touched for
>>a while.
>> >
>> > Thanks,
>> > Tim
>> >
>> > [1] https://beam.apache.org/documentation/runners/capability-matrix/
>> > [2] https://beam.apache.org/documentation/runners/mapreduce/
>> > [3] https://github.com/apache/beam/tree/mr-runner
>> >
>> > On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré
>>mailto:j...@nanthrax.net>
>> > >> wrote:
>> >
>> > Hi Max,
>> >
>> > not sure I fully follow you there. You mean that we would
>>have kind of
>> > compability matrix on dedicated page of each runner ?
>> >
>> > Regards
>> > JB
>> >
>> > On 21/09/2018 10:57, Maximilian Michels wrote:
>> > > Hi Beamers,
>> > >
>> > > There have been occasions where people asked me about
>>Runner XY and I
>> > > had to find out that it only exists in the compatibility
>>matrix,
>> > but not
>> > > as part of our code base. More interestingly, I couldn't
>>even find its
>> > > code or documentation via my favorite search engine.
>> > >
>> > > This seems to be the case for multiple Runners in the matrix.
>> > >
>> > > The compatibility matrix will need an overhaul anyways with the
>> > > portability changes, but what do you think about limiting the
>> > matrix to
>> > > Runners in the Beam code base?
>> > >
>> > > Thanks,
>> > > Max
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbono...@apache.org 
>>

Re: Compatibility Matrix vs Runners in the code base

[Maximilian Michels]

> not sure I fully follow you there.

@JB I simply meant to ask whether it make sense to have Runners in the 
matrix whose code/documentation is not part of Beam. For it to become a 
part of Beam, it could be as easy as adding a link to the external 
Runner page.



I don't know that we need to limit the matrix to runners in the Beam codebase


@Robert I think we used to only allow Runners in the matrix which were 
in the Beam code base. However, you are right, it is not necessary for 
Runners to live in the Beam repo. But IMHO they should be documented and 
linked before entries to the matrix are made.



+1 but perhaps we should having a table listing Runners under development like 
we do for IOs.


@Tim Yes, I didn't even know the Hadoop Runner was in a branch.


I don't want to remove any Runners from the matrix but I propose to 
require some form of documentation on the Beam website in addition to 
the compatibility matrix entry.


The current state:

Runners documented
==

Direct Runner
Apache Apex
Apache Flink
Apache Gearpump
Apache Samza
Apache Spark
Google Cloud Dataflow

Runners according to the matrix
===

Apache Apex 
Apache Flink
Apache Gearpump
Apache Samza
Apache Spark
Google Cloud Dataflow   
Apache Hadoop MapReduce 
JStorm  
IBM Streams


If we can fix the diff between these two lists, I'd feel more 
comfortable the next time somebody asks about a Runner I haven't used yet :)


Thanks,
Max

On 21.09.18 14:51, Thomas Weise wrote:

The MapReduce runner IMHO should not be in the matrix.

For the external runners, is there any public documentation available 
that explains how they can be used and how they are supported?


Thanks,
Thomas


On Fri, Sep 21, 2018 at 5:14 AM Jean-Baptiste Onofré > wrote:


You are right Tim.

M/R runner is on a branch (in stale for now to be honest ;)).

I think I got Max's remark.

So, agree to focus only Beam coverage in the runner compatibility
matrix. However, it's also important for the community to have some
insights about runners generally speaking.

Regards
JB

On 21/09/2018 14:00, Tim Robertson wrote:
 > "what do you think about limiting the matrix to Runners in the
Beam code
 > base"
 >
 > +1 but perhaps we should having a table listing Runners under
 > development like we do for IOs.
 >
 > As a concrete example we have MapReduce listed in the matrix [1],
a page
 > documenting it [2] stating it is in Beam 2.6.0 but unless I'm
mistaken
 > the code exists only on a branch [3] and hasn't been touched for
a while.
 >
 > Thanks,
 > Tim
 >
 > [1] https://beam.apache.org/documentation/runners/capability-matrix/
 > [2] https://beam.apache.org/documentation/runners/mapreduce/
 > [3] https://github.com/apache/beam/tree/mr-runner
 >
 > On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré
mailto:j...@nanthrax.net>
 > >> wrote:
 >
 >     Hi Max,
 >
 >     not sure I fully follow you there. You mean that we would
have kind of
 >     compability matrix on dedicated page of each runner ?
 >
 >     Regards
 >     JB
 >
 >     On 21/09/2018 10:57, Maximilian Michels wrote:
 >     > Hi Beamers,
 >     >
 >     > There have been occasions where people asked me about
Runner XY and I
 >     > had to find out that it only exists in the compatibility
matrix,
 >     but not
 >     > as part of our code base. More interestingly, I couldn't
even find its
 >     > code or documentation via my favorite search engine.
 >     >
 >     > This seems to be the case for multiple Runners in the matrix.
 >     >
 >     > The compatibility matrix will need an overhaul anyways with the
 >     > portability changes, but what do you think about limiting the
 >     matrix to
 >     > Runners in the Beam code base?
 >     >
 >     > Thanks,
 >     > Max
 >
 >     --
 >     Jean-Baptiste Onofré
 > jbono...@apache.org 
>
 > http://blog.nanthrax.net
 >     Talend - http://www.talend.com
 >

-- 
Jean-Baptiste Onofré

jbono...@apache.org 
http://blog.nanthrax.net
Talend - http://www.talend.com

Python PreCommit broken

[Maximilian Michels]

Hi,

The Python PreCommit tests are currently broken. Is anybody looking into 
this?


Example: 
https://builds.apache.org/job/beam_PreCommit_Python_Commit/1308/#showFailuresLink

JIRA: https://issues.apache.org/jira/browse/BEAM-5458

I'm sure this was just an accident. No big deal but let's make sure 
PreCommit passes when merging. A broken PreCommit means that we can't 
merge any other changes with confidence.


Thanks,
Max

Re: Compatibility Matrix vs Runners in the code base

[John MacMillan]

As someone who works on an active runner that's not in the core codebase (the IBM Streams one), we'd like to stay in the matrix and if that means requiring a link to an external page that's easy enough to provide.
 
Thanks,
John
 
- Original message -From: Robert Bradshaw To: dev@beam.apache.orgCc:Subject: Re: Compatibility Matrix vs Runners in the code baseDate: Fri, Sep 21, 2018 5:07 AM 
I don't know that we need to limit the matrix to runners in the Beam codebase (in fact, I could envision a world where most runners live in an upstream codebase), but at the very lease each of these runners should have a link to a page about using that runner with Beam.  

On Fri, Sep 21, 2018 at 10:57 AM Maximilian Michels  wrote:
Hi Beamers,There have been occasions where people asked me about Runner XY and Ihad to find out that it only exists in the compatibility matrix, but notas part of our code base. More interestingly, I couldn't even find itscode or documentation via my favorite search engine.This seems to be the case for multiple Runners in the matrix.The compatibility matrix will need an overhaul anyways with theportability changes, but what do you think about limiting the matrix toRunners in the Beam code base?Thanks,Max
 

Re: Compatibility Matrix vs Runners in the code base

[Thomas Weise]

The MapReduce runner IMHO should not be in the matrix.

For the external runners, is there any public documentation available that
explains how they can be used and how they are supported?

Thanks,
Thomas


On Fri, Sep 21, 2018 at 5:14 AM Jean-Baptiste Onofré 
wrote:

> You are right Tim.
>
> M/R runner is on a branch (in stale for now to be honest ;)).
>
> I think I got Max's remark.
>
> So, agree to focus only Beam coverage in the runner compatibility
> matrix. However, it's also important for the community to have some
> insights about runners generally speaking.
>
> Regards
> JB
>
> On 21/09/2018 14:00, Tim Robertson wrote:
> > "what do you think about limiting the matrix to Runners in the Beam code
> > base"
> >
> > +1 but perhaps we should having a table listing Runners under
> > development like we do for IOs.
> >
> > As a concrete example we have MapReduce listed in the matrix [1], a page
> > documenting it [2] stating it is in Beam 2.6.0 but unless I'm mistaken
> > the code exists only on a branch [3] and hasn't been touched for a
> while.
> >
> > Thanks,
> > Tim
> >
> > [1] https://beam.apache.org/documentation/runners/capability-matrix/
> > [2] https://beam.apache.org/documentation/runners/mapreduce/
> > [3] https://github.com/apache/beam/tree/mr-runner
> >
> > On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré  > > wrote:
> >
> > Hi Max,
> >
> > not sure I fully follow you there. You mean that we would have kind
> of
> > compability matrix on dedicated page of each runner ?
> >
> > Regards
> > JB
> >
> > On 21/09/2018 10:57, Maximilian Michels wrote:
> > > Hi Beamers,
> > >
> > > There have been occasions where people asked me about Runner XY
> and I
> > > had to find out that it only exists in the compatibility matrix,
> > but not
> > > as part of our code base. More interestingly, I couldn't even find
> its
> > > code or documentation via my favorite search engine.
> > >
> > > This seems to be the case for multiple Runners in the matrix.
> > >
> > > The compatibility matrix will need an overhaul anyways with the
> > > portability changes, but what do you think about limiting the
> > matrix to
> > > Runners in the Beam code base?
> > >
> > > Thanks,
> > > Max
> >
> > --
> > Jean-Baptiste Onofré
> > jbono...@apache.org 
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Compatibility Matrix vs Runners in the code base

[Tim Robertson]

"what do you think about limiting the matrix to Runners in the Beam code
base"

+1 but perhaps we should having a table listing Runners under development
like we do for IOs.

As a concrete example we have MapReduce listed in the matrix [1], a page
documenting it [2] stating it is in Beam 2.6.0 but unless I'm mistaken the
code exists only on a branch [3] and hasn't been touched for a while.

Thanks,
Tim

[1] https://beam.apache.org/documentation/runners/capability-matrix/
[2] https://beam.apache.org/documentation/runners/mapreduce/
[3] https://github.com/apache/beam/tree/mr-runner

On Fri, Sep 21, 2018 at 1:37 PM Jean-Baptiste Onofré 
wrote:

> Hi Max,
>
> not sure I fully follow you there. You mean that we would have kind of
> compability matrix on dedicated page of each runner ?
>
> Regards
> JB
>
> On 21/09/2018 10:57, Maximilian Michels wrote:
> > Hi Beamers,
> >
> > There have been occasions where people asked me about Runner XY and I
> > had to find out that it only exists in the compatibility matrix, but not
> > as part of our code base. More interestingly, I couldn't even find its
> > code or documentation via my favorite search engine.
> >
> > This seems to be the case for multiple Runners in the matrix.
> >
> > The compatibility matrix will need an overhaul anyways with the
> > portability changes, but what do you think about limiting the matrix to
> > Runners in the Beam code base?
> >
> > Thanks,
> > Max
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

Re: Compatibility Matrix vs Runners in the code base

[Jean-Baptiste Onofré]

Hi Max,

not sure I fully follow you there. You mean that we would have kind of
compability matrix on dedicated page of each runner ?

Regards
JB

On 21/09/2018 10:57, Maximilian Michels wrote:
> Hi Beamers,
> 
> There have been occasions where people asked me about Runner XY and I
> had to find out that it only exists in the compatibility matrix, but not
> as part of our code base. More interestingly, I couldn't even find its
> code or documentation via my favorite search engine.
> 
> This seems to be the case for multiple Runners in the matrix.
> 
> The compatibility matrix will need an overhaul anyways with the
> portability changes, but what do you think about limiting the matrix to
> Runners in the Beam code base?
> 
> Thanks,
> Max

-- 
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: Compatibility Matrix vs Runners in the code base

[Robert Bradshaw]

I don't know that we need to limit the matrix to runners in the Beam
codebase (in fact, I could envision a world where most runners live in an
upstream codebase), but at the very lease each of these runners should have
a link to a page about using that runner with Beam.

On Fri, Sep 21, 2018 at 10:57 AM Maximilian Michels  wrote:

> Hi Beamers,
>
> There have been occasions where people asked me about Runner XY and I
> had to find out that it only exists in the compatibility matrix, but not
> as part of our code base. More interestingly, I couldn't even find its
> code or documentation via my favorite search engine.
>
> This seems to be the case for multiple Runners in the matrix.
>
> The compatibility matrix will need an overhaul anyways with the
> portability changes, but what do you think about limiting the matrix to
> Runners in the Beam code base?
>
> Thanks,
> Max
>

Compatibility Matrix vs Runners in the code base

[Maximilian Michels]

Hi Beamers,

There have been occasions where people asked me about Runner XY and I 
had to find out that it only exists in the compatibility matrix, but not 
as part of our code base. More interestingly, I couldn't even find its 
code or documentation via my favorite search engine.


This seems to be the case for multiple Runners in the matrix.

The compatibility matrix will need an overhaul anyways with the 
portability changes, but what do you think about limiting the matrix to 
Runners in the Beam code base?


Thanks,
Max

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #179

[Apache Jenkins Server]

See 


Changes:

[herohde] Add Go quickstart link to the footer

[xiliu] [BEAM-3079]: Samza Runner docs and capability matrix

[xiliu] Update the option docs

[xinyuliu.us] Add Samza to the runner list

[klk] Add a pull request template with staged URL

[apilloud] Add SQL Lexical Structure from BigQuery

[mergebot] Add Apache license headers

[apilloud] Update keywords from BeamSqlParserImplConstants

[apilloud] Update CREATE TABLE from doc

[apilloud] Move CREATE TABLE and SELECT up a level

[mergebot] Update contact-us.md

[mergebot] Spelling correction

[apilloud] Add redirect to new team page

[apilloud] Add SQL Set page

[boyuanz] Update release guide for download page changes

[klk] Make Samza icon a reasonable size

[klk] Alphabetize runners

[mergebot] New blog post about 2.5.0 release

[amaliujia] Update doc

[echauchot] [BEAM-4731] update flink submission port because flink 1.5 new 
standard

[mergebot] Adding YouTube channel page

[mergebot] Processing feedback

[mergebot] YouTube url update

[lcwik] [BEAM-3940] Remove reference to updating container version in pom.xml

[lcwik] Fix typos in programming guide

[lcwik] Fix typos in mobile gaming example

[kirpichov] Updates SDF status in capability matrix

[mergebot] Added new design documents

[mergebot] [BEAM-4430] Improve Performance Testing Documentation

[mergebot] Rename RowType to Schema in SQL walkthrough

[mergebot] Minor tweak to Intellij instructions.

[mergebot] fixup

[melissapa] [BEAM-2977] Improve unbounded prose in wordcount example

[mergebot] [BEAM-4188] Fix 2.4.0 release notes link

[mergebot] Update code style section to mention Spotless.

[mergebot] Clarify that this is only for Java/Groovy and not Python/Go/...

[lcwik] [BEAM-4828] Updated documentation, added SQS BEAM-4828 to In-Progress

[github] Update Python quickstart with Windows instructions

[mergebot] Add courtesy unsubscribe links to mailing lists

[melissapa] Fix shell tabs, add new shell tab switcher

[mergebot] Add a page for release cadence

[mergebot] updated the language

[mergebot] fix the space

[mergebot] [BEAM-4827] Add Nexmark dashboards to nexmark page on the website

[mergebot] [BEAM-4827] Fix typos

[mergebot] Add Flink instructions to portability contrib page.

[mergebot] Add instructions for local Flink cluster.

[mergebot] Update Java Quickstart for Windows

[mergebot] Update quickstart-java.md

[github] Update quickstart-java.md

[mergebot] [BEAM-4802] Update the contribution guide (Dependency section)

[migryz] Add post-commit tests policies summary page.

[migryz] Fix links

[github] Remove [TOC]

[migryz] Update postcommits guides

[ehudm] Remove testing matrix and simplify intro.

[mergebot] Moves KuduIO from in progress to available

[timrobertson100] [BEAM-4260] Document HCatalogIO use with Hive 1.1

[mergebot] Remove old pydoc and javadoc for versions < 2.0.0.

[apilloud] [BEAM-4761] Add Dataflow Runner

[mergebot] Update beam-site for release 2.6.0

[mergebot] Update download page

[mergebot] Updating downloads page.

[melissapa] Fix broken 2.5.0 download links

[mergebot] go version

[mergebot] Add section for SQL shell

[mergebot] Adding 'editable mode' flag installation for Python.

[thw] fix job service startup instruction

[mergebot] Fix Apex Runner instructions for execution on YARN

[mergebot] Update SQL walkthrough for 2.6.0

[github] Adding Pablo Estrada as author for blog

[github] Blog post for Beam 2.6.0 release

[github] Added Rafael Fernandez as author.

[rfernand] Adds Jenkinns job pointer to #building--testing

[rfernand] Minor rewrite to help newcomers explore the project's building and

[thw] fixup

[mergebot] Add instructions of using automation scripts

[mergebot] Fix broken tests

[mergebot] Addressed Ahmet's comments

[mergebot] Fixing 2.6.0 blog post

[mergebot] Quick fixes to layout

[mergebot] Address readability reviews

[mergebot] Grammar pass + pretty print

[mergebot] All files have Apache License

[mergebot] Add instructions about how to build python wheels

[mergebot] Add long term support release concept

[mergebot] Edit suggestions

[mergebot] Add precommit policies and triage guide.

[mergebot] Add dashes to pre-commit and post-commit.

[mergebot] Update programming guide to suggest using NewDoFn approach to

[mergebot] Address comments.

[mergebot] Add compatibility warning.

[mergebot] Fix language tab tags

[mergebot] Add blog post "A review of input streaming connectors"

[mergebot] Add authors for blog post in

[mergebot] Fix typo for author's name in blog post

[mergebot] Fix other typo in author's name for blog post

[mergebot] Blog post updates based on @iemejia's feedback

[mergebot] Updates to streaming connectors blog post

[mergebot] Set publication date for streaming connectors blog post

[mergebot] Update doc links in blog post to point to latest release

[mergebot] Fix extraneous p tag and 

Re: Retiring runners?

[Robert Bradshaw]

Glad to hear Gearpump is still alive.

It is hard to measure how much of a burden these additional runners are at
the moment. I suggest that if it comes to a point that non-trivial changes
are needed, we reach out to the list. If no one agrees to support it, we
could disable the tests and, after some amount of time with no sign of
future support, remove simply it altogether.

I don't think we're to that point yet, but it would like to hear if anyone
else has had to invest a lot of effort here. At some point we'll likely
make portability the only way to execute, in which case (if not before)
there may be a great culling.


On Fri, Sep 21, 2018 at 2:42 AM P. Taylor Goetz  wrote:

> Indeed, retirement is not the death of a project, but rather one way to
> exit the Incubator. There is no shame in it. The ASF isn’t the best fit for
> every project.
>
> -Taylor
>
> On Sep 20, 2018, at 7:49 PM, Lukasz Cwik  wrote:
>
> Thanks for info, I wasn't aware of what retiring meant, I assumed
> (incorrectly) that retiring = death of a project.
>
> On Thu, Sep 20, 2018 at 4:45 PM Manu Zhang 
> wrote:
>
>> Hi Luke,
>>
>> (On behalf of Gearpump runner)
>>
>> Getting retired (from Apache) doesn't mean the end of Gearpump as a
>> project but to make it easier to commit and release for, honestly speaking,
>> a small community.
>> I've been maintaining Gearpump runner and will strive for better
>> integration here.
>> I agree we should revisit this question if the runner is too much of a
>> burden.
>>
>> Thanks,
>> Manu Zhang
>>
>> On Fri, Sep 21, 2018 at 6:34 AM Lukasz Cwik  wrote:
>>
>>> Datatorrent the largest contributor to the Apex project has closed
>>> shop[1], the project hasn't retired but development has significantly
>>> slowed down. The Gearpump project has voted to retire[2].
>>>
>>> Do people know if these runners are being used?
>>> Do people spend enough time on supporting/updating code in these areas
>>> or looking at test flakes that warrant supporting them still?
>>> Are there other runners in a similar situation as to Apex or Gearpump?
>>> Should we revisit this question once we have a larger integration that
>>> we want to do like SplittableDoFn or portability?
>>>
>>> 1:
>>> https://www.datanami.com/2018/05/08/datatorrent-stream-processing-startup-folds/
>>> 2:
>>> https://lists.apache.org/thread.html/03fc0421bb425f79d623863335bc2b81e4b6c77a169f508aadce7785@%3Cdev.gearpump.apache.org%3E
>>>
>>

Re: [VOTE] Release 2.7.0, release candidate #1

[Romain Manni-Bucau]

Hi Charles,

I didnt get a chance to work more on it but the sample shows that just
changing beam version breaks existing code.

Since Beam does not manage its dependency compatibility with runners -
understand this as "it never managed this issue" - I guess you can proceed
with 2.7 ignoring this breaking change. For > 2.7.0 versions, testing with
the officially supported engine versions (kind of matrix compatibility) can
be required with some advanced apps (with sides, unions, etc... maybe
nextmark can be a start?).

My blind guess is that 2.6 was compiled with spark 2.2.1 and 2.7 with spark
2.3.1 and therefore the method can have changed even if signature didn't
(thanks scala and java method lookup which uses returned types vs
signatures ignoring them).
An interesting test would be to compile Beam 2.7.0 with spark 2.2.1 and run
it with my project, I guess it would work.

Side note: during my tests i realized that if you use avro 1.8 new API it
fails in spark since only spark master was upgraded to avro 1.7 and not
earlier versions so beam providing avro 1.8 is another issue.

Anyway, fine to proceed on my side even if there is a "user regression",
nobody being available to identify it would mean delaying the release of
too much and beam is far to be only spark runner so no reason to block
others ;).

Le ven. 21 sept. 2018 03:32, Ahmet Altay  a écrit :

> Good point. However, we agreed that our release policy would be to patch
> only long term support (LTS) releases. Given that we have not made any LTS
> releases yet, perhaps we should use 2.8.0 as the opportunity to make our
> first LTS release.
>
> On Thu, Sep 20, 2018 at 6:26 PM, Thomas Weise  wrote:
>
>> That's not the same for a user though. 2.7.1 would be a patch compatible
>> release that only fixes bugs. 2.8.0 adds new features and potentially also
>> new issues..
>>
>> On Thu, Sep 20, 2018 at 3:16 PM Ahmet Altay  wrote:
>>
>>> +1 to Thomas's suggestion. Instead of 2.7.1 we can follow up with 2.8.0
>>> though. 2.8.0 has a release branch cut date of 10/10 according to our
>>> release calendar.
>>>
>>> On Thu, Sep 20, 2018 at 2:47 PM, Connell O'Callaghan <
>>> conne...@google.com> wrote:
>>>
 +1 to Thomas's suggestion - if Charles or others cannot reproduce.

 On Thu, Sep 20, 2018 at 2:40 PM Thomas Weise  wrote:

> We can also consider releasing 2.7.0 and then follow up with 2.7.1 if
> the problem can be reproduced and requires a fix. Just food for thought :)
>
>
> On Thu, Sep 20, 2018 at 2:13 PM Charles Chen  wrote:
>
>> My mistake, it looks like the correct beam staging repository (
>> https://repository.apache.org/content/repositories/orgapachebeam-1046/)
>> is specified in your pom file.
>>
>> On Thu, Sep 20, 2018 at 2:10 PM Charles Chen  wrote:
>>
>>> Hey Romain and JB, do you have any progress on this?  One thing I
>>> would like to point out is that 2.7.0 isn't yet pushed to Maven 
>>> Central, so
>>> referring to it by version is not expected to work (and it looks like 
>>> this
>>> is what is done in your repo:
>>> https://github.com/rmannibucau/beam-2.7.0-fails).  Luke indicated
>>> above that he doesn't see any dependency changes.  Can you isolate and
>>> reproduce this problem so that we can develop a fix, if necessary?  I 
>>> would
>>> like to proceed with an RC2 as soon as possible.
>>>
>>> On Wed, Sep 19, 2018 at 6:37 AM Romain Manni-Bucau <
>>> rmannibu...@gmail.com> wrote:
>>>
 Quick update on the spark issue: I didn't get enough time to
 identify it clearly but managed to have a passing run of my test 
 changing a
 bunch of versions.
 I suspect my code triggers some class conflicting between spark and
 my shade leading to a serialization issue. I didn't test 
 userClassPathFirst
 option of spark but it can be an interesting thing to enable in beam 
 runner.
 However it is still very confusing to have it not running just
 upgrading beam version and the spark error is very hard to understand.

 Romain Manni-Bucau
 @rmannibucau  |  Blog
  | Old Blog
  | Github
  | LinkedIn
  | Book
 


 Le mar. 18 sept. 2018 à 20:17, Lukasz Cwik  a
 écrit :

> Romain hinted that this was a dependency issue but when comparing
> the two dependency trees I don't get much of a difference:
>
> lcwik@lcwik0: ~$ diff /tmp/260 /tmp/270
> < [INFO] +- org.apache.beam:beam-runners-spark:jar:2.6.0:compile
> < [INFO] |  +-
> 

Re: [ANNOUNCEMENT] New Beam chair: Kenneth Knowles

[Matthias Baetens]

Thank you for all the mentoring and work for the Apache Beam project, Davor.

Welcome as a chair in the project Kenn, congratulations!

On Thu, Sep 20, 2018, 22:58 Reuven Lax  wrote:

> Congratulations Kenn!
>
> On Thu, Sep 20, 2018 at 2:31 PM Griselda Cuevas  wrote:
>
>> Thank you so much Davor for all the work you put into the project since
>> its inception.
>> Congrats Kenn on becoming the new Chair!
>>
>> On Thu, 20 Sep 2018 at 12:54, Neelesh Salian 
>> wrote:
>>
>>> Congratulation Kenn!
>>> Thank you for your continued leadership , Davor.
>>>
>>> On Wed, Sep 19, 2018 at 12:54 PM Davor Bonaci  wrote:
>>>
 Hi everyone --
 It is with great pleasure that I announce that at today's meeting of
 the Foundation's Board of Directors, the Board has appointed Kenneth
 Knowles as the second chair of the Apache Beam project.

 Kenn has served on the PMC since its inception, and is very active and
 effective in growing the community. His exemplary posts have been cited in
 other projects. I'm super happy to have Kenn accepted the nomination, and
 I'm confident that he'll serve with distinction.

 As for myself, I'm not going anywhere. I'm still around and will be as
 active as I have recently been. Thrilled to be able to pass the baton to
 such a key member of this community and to have less administrative work to
 do ;-).

 Please join me in welcoming Kenn to his new role, and I ask that you
 support him as much as possible. As always, please let me know if you have
 any questions.

 Davor

>>>
>>>
>>> --
>>> Regards,
>>> Neelesh S. Salian
>>>
>>> --