Re: JB's back

Good to have you back JB! On Wed, Nov 21, 2018, 06:06 Kenneth Knowles wrote: > Yes, welcome back! > > On Tue, Nov 20, 2018 at 9:51 PM Ahmet Altay wrote: > >> Welcome back! >> >> On Tue, Nov 20, 2018 at 9:11 PM, Jean-Baptiste Onofré >> wrote: >> >>> Hi guys, >>> >>> Sorry to have been quiet rec

Re: JB's back

Yes, welcome back! On Tue, Nov 20, 2018 at 9:51 PM Ahmet Altay wrote: > Welcome back! > > On Tue, Nov 20, 2018 at 9:11 PM, Jean-Baptiste Onofré > wrote: > >> Hi guys, >> >> Sorry to have been quiet recently. >> >> After some rushy things and having been sick last week, I'm back on Beam. >> >> W

Re: JB's back

Welcome back! On Tue, Nov 20, 2018 at 9:11 PM, Jean-Baptiste Onofré wrote: > Hi guys, > > Sorry to have been quiet recently. > > After some rushy things and having been sick last week, I'm back on Beam. > > With Alexey, Etienne and Ismaël, we are working on the Spark runner and > I'm also resumi

JB's back

Hi guys, Sorry to have been quiet recently. After some rushy things and having been sick last week, I'm back on Beam. With Alexey, Etienne and Ismaël, we are working on the Spark runner and I'm also resuming several works I was holding on the IOs. Regards JB -- Jean-Baptiste Onofré jbono...@ap

Re: Bay Area Apache Beam Kickoff!

Nice !! Unfortunately I won't be able to be there. But good luck and I'm sure it will be a great meetup ! Regards JB On 20/11/2018 02:36, Austin Bennett wrote: > We have our first meetup scheduled for December 12th in San Francisco.   > > Andrew Pilloud, a software engineer at Google and Beam c

Re: [PROPOSAL] Prepare Beam 2.9.0 release

Hi Cham, it sounds good to me. I'm resuming some works on IOs but nothing blocker. Regards JB On 21/11/2018 03:59, Chamikara Jayalath wrote: > Hi All, > > Looks like there are three blockers in the burndown list but they are > actively being worked on. > > If there's no objection I'll create

Re: MetricResult querying design questiosn

(1)-(3) make sense to me; perhaps (2) can be autogenerated by gRPC and wrapped into nicer APIs as desired. I think if you transliterate from Java to proto3 then the sketches in the "Querying Metrics" section of http://s.apache.org/beam-metrics-api have some of the same ideas - what is left blank is

Re: [PROPOSAL] Prepare Beam 2.9.0 release

Hi All, Looks like there are three blockers in the burndown list but they are actively being worked on. If there's no objection I'll create the release branch tomorrow morning. We can cherry-pick fixes to the blockers before building the first RC hopefully on Monday. Thanks, Cham On Sat, Nov 1

Re: [DISCUSS] Reverting commits on green post-commit status

Two hours is too quick for people around the world to respond. If something is obviously wrong, that may be fine, but otherwise we should give others time to respond. I think there's another important distinction that impacts urgency and impact: by definition, Beam's on Precommit/Postcommit tests

MetricResult querying design questiosn

I was wondering if we have some design about MetricResult querying, which is a queryable object that exists on the PipelineResult. IMO, the way this should ideally work, is: (1) The runner would be responsible for querying the metrics, since a Runner will have its own metrics aggregation system,

Re: [DISCUSS] Reverting commits on green post-commit status

Following the discussion, we are discussing as how to address failures on external projects caused by changes to Beam project. I believe that that rollback first in case of red pre/postcommit tests is a valid option, since it blocks Beam development process. In case if downstream project has test

Re: [VOTE] Release Vendored gRPC 1.13.1 and Guava 20.0, release candidate #1

+1 then. Thanks for the detailed explanation and links. It will be great to start using these and gaining experience with the vendored artifacts. Kenn On Tue, Nov 20, 2018 at 11:27 AM Lukasz Cwik wrote: > I also looked for documentation as to how this information is used but > couldn't find an

Re: [DISCUSS] Reverting commits on green post-commit status

The instructions in the post-commit policies page [1] is helpful, by not clear enough regarding what 'Rollback First' exactly means in the case of breaking a downstream project. The discussions in this thread makes things more well defined. I summarized things as an updating PR

Re: [DISCUSS] SplittableDoFn Java SDK User Facing API

On Tue, Nov 20, 2018 at 7:10 PM Lukasz Cwik wrote: > I'll perform the swap for a fraction because as I try to map more of the > spaces to an arbitrary byte[] I naturally first map the space onto natural > numbers before mapping to a byte[]. > > Any preference between these options: > A: > // Repr

Re: [VOTE] Release Vendored gRPC 1.13.1 and Guava 20.0, release candidate #1

I also looked for documentation as to how this information is used but couldn't find anything beyond configuring the Maven archive plugin[1]. These seem to be benign since we have been publishing them with beam-sdks-java-core since at least the 2.0.0 release[2]. I believe these files appear becau

Re: [DISCUSS] SplittableDoFn Java SDK User Facing API

Ismael, I looked at the API around ByteKeyRangeTracker and OffsetRangeTracker figured out that the classes are named as such because they are trackers for the OffsetRange and ByteKeyRange classes. Some options are to: 1) Copy the ByteKeyRange and call it ByteKeyRestriction and similarly copy Offset

Re: [Testing] Splitting pre-commits from post-commit test targets

Useful and really practical idea! -Rui On Tue, Nov 20, 2018 at 10:37 AM Ruoyun Huang wrote: > +1 Great improvement! Thanks Scott! > > On Tue, Nov 20, 2018 at 10:33 AM Pablo Estrada wrote: > >> I think this is a great idea, and a good improvement. Thanks Scott! >> -P. >> >> On Tue, Nov 20, 201

Re: [Testing] Splitting pre-commits from post-commit test targets

+1 Great improvement! Thanks Scott! On Tue, Nov 20, 2018 at 10:33 AM Pablo Estrada wrote: > I think this is a great idea, and a good improvement. Thanks Scott! > -P. > > On Tue, Nov 20, 2018 at 10:09 AM Scott Wegner wrote: > >> I wanted to give a heads-up to a small optimization that I hope to

Re: [Testing] Splitting pre-commits from post-commit test targets

I think this is a great idea, and a good improvement. Thanks Scott! -P. On Tue, Nov 20, 2018 at 10:09 AM Scott Wegner wrote: > I wanted to give a heads-up to a small optimization that I hope to make to > our Jenkins test targets. Currently our post-commit test jobs also > redundantly run pre-com

Re: [DISCUSS] SplittableDoFn Java SDK User Facing API

I'll perform the swap for a fraction because as I try to map more of the spaces to an arbitrary byte[] I naturally first map the space onto natural numbers before mapping to a byte[]. Any preference between these options: A: // Represents a non-negative decimal number: unscaled_value * 10^(-scale)

[Testing] Splitting pre-commits from post-commit test targets

I wanted to give a heads-up to a small optimization that I hope to make to our Jenkins test targets. Currently our post-commit test jobs also redundantly run pre-commit tests. I'd like to remove redundant execution to get a faster post-commit test signal. See: https://github.com/apache/beam/pull/70

Re: [BEAM-6077] FlinkRunner: Make UnboundedSource state re-scale friendly

Hi Jozef, I responded on JIRA today before I saw your mail here. The splitting of the UnboundedSource is performed during translation of the Beam pipeline. It think it would be feasible to use Flink's maximum parallelism instead of the configured parallelism. That would enable to increase the

Re: E-mail Organization

I was about to suggest tags in subject lines as well. Easier to see in email listings than anything in the body. On Mon, Nov 19, 2018 at 7:22 PM Lukasz Cwik wrote: > Putting the tags in the subject line is inline with the style of what we > currently do using [DISCUSS], [VOTE], [BEAM-YYY] so I l

Re: [DISCUSS] SplittableDoFn Java SDK User Facing API

I'm still trying to wrap my head around what is meant by backlog here, as it's different than what I've seen in previous discussions. Generally, the backlog represented a measure of the known but undone part of a restriction. This is useful for a runner to understand in some manner what progress i

[BEAM-6077] FlinkRunner: Make UnboundedSource state re-scale friendly

I want to reach out for opinions on what would be the best way to proceed with https://issues.apache.org/jira/browse/BEAM-6077 The problem is, that when FlinkRunner job is being restored from checkpoint, it needs to resurrect source and it's readers given the checkpoint state. State element is rep