What to do about issues that track flaky tests?

2022-09-14 Thread Kenneth Knowles
I would like to make this alert email actionable. I went through most of these issues. About half are P1 "flake" issues. I don't think magically expecting them to be deflaked is helpful. So I have a couple ideas: 1. Exclude "flake" P1s from this email. This is what we used to do. But then... are

DRAFT - Apache Beam Board Report - September 2022

2022-09-13 Thread Kenneth Knowles
Hi all, The next Beam board report is due tomorrow. Sorry for the short notice. Please help me to draft it at https://s.apache.org/beam-draft-report-2022-09. I've opened edit access to anyone with the link to minimize friction of drafting. Ideas: - highlights from CHANGES.md - interesting

Re: [Infrastructure] Periodically run Java microbenchmarks on Jenkins

2022-09-13 Thread Kenneth Knowles
Good idea. I'm curious about our current benchmarks. Some of them run on clusters, but I think some of them are running locally and just being noisy. Perhaps this could improve that. (or if they are running on local Spark/Flink then maybe the results are not really meaningful anyhow) On Tue, Sep

Re: Beam Website Feedback

2022-09-12 Thread Kenneth Knowles
+Reza Rokni do you know about the slowly updating side input window documentation? Do you know who might? On Mon, Sep 12, 2022 at 8:19 AM Peter McArthur wrote: > There are a couple issues with the python SDK and side input patterns page > here >

Re: SingleStore IO design doc

2022-09-07 Thread Kenneth Knowles
Thanks for sharing the doc! Two big picture thoughts I had were: 1. Portability first 2. SDF first Both of these are just ideas to take advantage of Beam's unique features. Happy for the contribution no matter what! Kenn On Wed, Sep 7, 2022 at 7:21 AM John Casey via dev wrote: > Hi Adalbert,

Re: Java compatibility checking library

2022-09-06 Thread Kenneth Knowles
/jkk87ls2lc17q6bqz5b1xzh67pk4x663 / https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152857) Kenn On Mon, Aug 29, 2022 at 4:10 PM Kenneth Knowles wrote: > I think this is still worth considering. We would need to see how well it > does at avoiding false alarms. It was brought up by Andrew a whil

Re: [ANNOUNCE] Apache Beam 2.41.0 Released

2022-09-06 Thread Kenneth Knowles
Thanks! On Fri, Aug 26, 2022 at 11:47 AM Byron Ellis via dev wrote: > Thanks Kiley! > > On Fri, Aug 26, 2022 at 9:37 AM Ahmet Altay via dev > wrote: > >> Thank you Kiley! >> >> On Fri, Aug 26, 2022 at 6:38 AM P Singh >> wrote: >> >>> Hi Team, >>> >>> Looking forward to trying and testing the

Re: [PROPOSAL] Preparing for 2.42.0 Release

2022-09-06 Thread Kenneth Knowles
Thanks for taking this on! Kenn On Wed, Aug 24, 2022 at 3:05 PM Robert Burke wrote: > Hi everyone! > > The next (2.42.0) release branch cut is scheduled for Sept 7th, according > to > the release calendar [1]. > > I would like to volunteer myself to do this release. My plan is to cut > the

Re: Java compatibility checking library

2022-08-29 Thread Kenneth Knowles
I think this is still worth considering. We would need to see how well it does at avoiding false alarms. It was brought up by Andrew a while back at https://lists.apache.org/thread/kq2w6rw0oorcj037ygjvfsqw1l9l55jj and I know we had another discussion of it even further back. I thought also that

Re: Checkpointing on Google Cloud Dataflow Runner

2022-08-29 Thread Kenneth Knowles
Hi Will, David, I think you'll find the best source of answer for this sort of question on the user@beam list. I've put that in the To: line with a BCC: to the dev@beam list so everyone knows they can find the thread there. If I have misunderstood, and your question has to do with building Beam

Re: Java & Python Project Starter / Example Using Terraform to build Dataflow Custom Templates

2022-08-23 Thread Kenneth Knowles
Nice! On Tue, Aug 23, 2022 at 11:00 AM Ahmet Altay via dev wrote: > Thank you for sharing Damon. > > On Tue, Aug 23, 2022 at 9:34 AM Damon Douglas via dev > wrote: > >> Hello Everyone, >> >> You can ignore this email if you do not use the Dataflow runner. >> >> We just published an example /

Fwd: New mailing list queued for creation: stackoverf...@beam.apache.org

2022-08-23 Thread Kenneth Knowles
FYI -- Forwarded message - From: ASF Self-Service Platform Date: Tue, Aug 23, 2022 at 11:40 AM Subject: New mailing list queued for creation: stackoverf...@beam.apache.org To: , ASF Infrastructure Hi there, As requested by k...@apache.org, a new mailing list has been

Re: [VOTE] Release 2.41.0, release candidate #1

2022-08-17 Thread Kenneth Knowles
Seems like there has been a lot of progress on https://github.com/raphw/byte-buddy/issues/1301. Since it has been identified, I think we can be pretty confident that the version downgrade is the necessary part. So we can revert the PR for the release, then on main branch we can proceed with

Re: [VOTE] Release 2.41.0, release candidate #1

2022-08-16 Thread Kenneth Knowles
Kiley Sok wrote: > If we roll back, do we need to roll back +Lukasz Cwik 's > change > [1] on master as well? > > Liam, are we okay to roll back this change for this release? > > [1] https://github.com/apache/beam/pull/22594 > > On Tue, Aug 16, 2022 at 2:25 PM Kenne

Re: [VOTE] Release 2.41.0, release candidate #1

2022-08-16 Thread Kenneth Knowles
Two options to unblock the release are: 1. Roll back https://github.com/apache/beam/pull/17317/files 2. Downgrade unvendored bytebuddy to 1.11.0 (if the above doesn't work or is too high risk) And as a follow up we should make sure there is some test that would exercise this, since that PR was

[ANNOUNCE] New committer: John Casey

2022-07-29 Thread Kenneth Knowles
Hi all, Please join me and the rest of the Beam PMC in welcoming a new committer: John Casey (johnca...@apache.org) John started contributing to Beam in late 2021. John has quickly become our resident expert on KafkaIO - identifying bugs, making enhancements, helping users - in addition to a

Re: Runner benchmarks in portable mode

2022-07-28 Thread Kenneth Knowles
Your question isn't for me, but I just want to say that I am really happy to hear you are doing this. I would like to get more continuous benchmarking so we can reduce any overheads Beam might introduce, for example on Samza in your case. And I would like to basically focus entirely on portable

Re: [RFC] State & Timers API Design for Go SDK

2022-07-28 Thread Kenneth Knowles
Really thorough. Love it! On Thu, Jul 28, 2022 at 9:02 AM Ritesh Ghorse via dev wrote: > Hey everyone, > > Danny and I have been working on > designing the state and timers for Go SDK. We wrote a design doc with > user-facing API, execution details, and different alternatives considered. > It

Re: Output after Pipeline replaceAll

2022-07-28 Thread Kenneth Knowles
Yes, this is expected. The goal of replaceAll is to replace a transform with a different subgraph that implements precisely the same semantics. And since the rest of the graph depends on the PCollections, the new expanded transform is wired directly to the old outputs. First point: Certainly the

Re: Beam gRPC depedency tracing

2022-07-27 Thread Kenneth Knowles
The vendored gRPC is built by transforming the released gRPC jar. Here is where in the Beam git history you can find the source for the transformation: https://github.com/apache/beam/tree/40293eb52ca914acbbbae51e4b24fa280f2b44f0/vendor/grpc-1_26_0 Kenn On Wed, Jul 27, 2022 at 9:24 AM JDW J

Re: Checkpoints timing out upgrading from Beam version 2.29 with Flink 1.12 to Beam 2.38 and Flink 1.14

2022-07-26 Thread Kenneth Knowles
Bumping this and adding +John Casey who knows about KafkaIO and unbounded sources, though probably less about the FlinkRunner. It seems you have isolated it to the Flink translation logic. I'm not sure who would be the best expert to evaluate if that logic is still OK. Kenn On Wed, Jun 29, 2022

Re: Extending 2.41.0 Java snapshot TTL

2022-07-25 Thread Kenneth Knowles
shared >>>>> across projects). As best I can tell, there isn't a way to configure >>>>> retention from the repo side, and it is likely managed by a repo-wide >>>>> cleanup policy [3]. I don't have the right permission set to actually view >>>>

Re: Trigger phrases in Github Actions

2022-07-14 Thread Kenneth Knowles
Is this an idiomatic way to trigger GHA? On Thu, Jul 14, 2022 at 1:36 PM Danny McCormick via dev wrote: > Hey Fer, > > I'm not 100% sure I follow what you're trying to do, but one approach you > could take is to gate everything off of an if like: > > ` if {{ (github.event.issue.pull_request &&

Re: Fun with WebAssembly transforms

2022-07-13 Thread Kenneth Knowles
I agree with Luke. Targeting little helper UDFs that go along with IOs are actually a major feature gap for xlang - like timestamp extractors that have to parse particular data formats. This could be a very useful place to try out the design options. I think we can simplify the problem by

DRAFT - Apache Beam Board Report - July 2022

2022-07-08 Thread Kenneth Knowles
Hi all, The next Beam board report is due July 13. Please help me to write it here: https://s.apache.org/beam-draft-report-2022-07 Ideas: - highlights from CHANGES.md - interesting technical discussions - integrations with other projects - community events - major user facing

Re: Beam Java DoFn Performance (and Auto-Vectorization)

2022-07-02 Thread Kenneth Knowles
Love it. Seems like there are some serious barriers to getting likely benefit for a random pipeline, but for a user who knew and wanted to be sure they got the benefit it could be really good (basically by programming Java like it is garbage collected C :-) Really useful to see how counters are a

Re: Update PR description template

2022-06-28 Thread Kenneth Knowles
we make a check required is by adding it in the .asf.yml file >> <https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Branchprotection>. >> So we could always update that if needed to push an emergency fix as long >> as we don't

Re: Update PR description template

2022-06-27 Thread Kenneth Knowles
I am in favor of requiring precommit green before merge. To make this better, we should: (1) run fewer irrelevant tests, for example almost every language-based presubmit tests vastly more code than could possibly be affected. (2) run more relevant tests, for example when an IO is touched running

Re: [VOTE] Release 2.40.0, candidate #2

2022-06-27 Thread Kenneth Knowles
+1 (binding) I tried out the new starter repositories, pointing them at this RC Check that I have things set up: git clone https://github.com/apache/beam-starter-java && cd beam-starter-java gradle run (actually I have things set up for Java 8 so I modified the starter to be Java 8

Re: Jira -> GitHub Issues Migration (This Friday)

2022-06-27 Thread Kenneth Knowles
I do not have a strong feeling about whether it should be in the title or in the description. I kind of like all titles looking "the same" versus some PRs missing an issue that could cause the PR to get ignored like it is less important. I agree with Alexey's concern. I do expect that issue

Re: [DISCUSS] What to do about P0/P1/flake automation Was: P1 issues report (70)

2022-06-27 Thread Kenneth Knowles
f not triaged or stagnate issues >>>>> and include them? >>>>> 2. The links in this report start with api.github.* and don’t take us >>>>> directly to the issues. >>>>> >>>>> >>>>> Danny McCormick 于2022年6月

Re: [DISCUSS] What to do about P0/P1/flake automation Was: P1 issues report (70)

2022-06-23 Thread Kenneth Knowles
ated >7 days ago, and those with no assignee) fits well with > the two goals you describe, if we include "triage needed" issues in the > latter category. Maybe we also explicitly separate these two concerns in > the report? > > > On Thu, Jun 23, 2022 at 1:14 PM Kenneth

[DISCUSS] What to do about P0/P1/flake automation Was: P1 issues report (70)

2022-06-23 Thread Kenneth Knowles
Forking thread because lots of people may just ignore this topic, per the discussion :-) (sometimes gmail doesn't fork thread properly, but here's hoping...) I'll add some other outcomes of these emails: - people file P0s that are not outages and P1s that are not data loss and I downgrade them

Re: [CdapIO] Dereference of possibly-null reference warning

2022-06-23 Thread Kenneth Knowles
Hi Elizaveta, Very good question. For this, I created org.apache.beam.sdk.util.Preconditions.checkStateNotNull ( https://github.com/apache/beam/blob/1851eef15c5f455c95402ced50d757ea167d33d9/sdks/java/core/src/main/java/org/apache/beam/sdk/util/Preconditions.java#L450 ) Multiple reasons: -

Re: Null PCollection errors in v2.40 unit tests

2022-06-20 Thread Kenneth Knowles
Just to bring this back to the thread, it turns out to be a JDK bug https://bugs.openjdk.org/browse/JDK-8225377 reported in 2019 by +Liam Miller-Cushon who is now also a contributor to Beam :-) We may be able to work around it somehow. On Thu, Jun 16, 2022 at 3:18 PM Kenneth Knowles wrote

Re: Null PCollection errors in v2.40 unit tests

2022-06-16 Thread Kenneth Knowles
mistake, please don't forward > it to anyone else (it may contain confidential or privileged information), > please erase all copies of it, including all attachments, and please let > the sender know it went to the wrong person. Thanks > > > On Wed, 15 Jun 2022 at 15:42, Kenneth Know

Re: Questions regarding contribution: Support for reading Kafka topics from any startReadTime in Java

2022-06-16 Thread Kenneth Knowles
These last two emails make sense to me, from both of you. Does startReadTime impact the watermark in the right way so that late data is dropped, or whatnot? Because even if you have a partition and you _do_ find an offset that works, late data could come in. So if you restrict the time to be in

Re: Null PCollection errors in v2.40 unit tests

2022-06-15 Thread Kenneth Knowles
This looks to be clearly a bug in autovalue. The PCollection field is clearly labeled as @Nullable. Autovalue generates the `checkArgument(pcollection != null)` for things that are not marked @Nullable. It should not do so here. The bugs sounds like it is stateful, where some global state in the

Re: Changing the interface in CassandraIO Mapper

2022-06-15 Thread Kenneth Knowles
Minor but important correction: Beam does *not* "shade" Guava. That tends to refer to build-time re-namespacing and/or bundling. Beam does neither of those things. What Beam has done is to create the equivalent of a totally independent fork. It has no impact on whether various libraries or IOs use

Re: Clean Up GitHub Labels

2022-06-14 Thread Kenneth Knowles
+1 sounds good to me One thing I did a lot of when triaging Jiras was moving them from one component to another, after which people who cared about those components would go through them. Making the labels more straightforward for users would streamline that. Kenn On Sun, Jun 12, 2022 at 9:04

Re: Jenkins CI currently unavailable

2022-06-14 Thread Kenneth Knowles
;>>>>> There are a couple of Go SDK PRs that are basically blocked on final >>>>>> manual runs of the post commits, that we'd like to get in for the 2.40 >>>>>> cut. >>>>>> >>>>>> Are we intending on delaying the 2.40 cut a little bit so

Re: [PROPOSAL] Preparing for Beam release 2.40.0

2022-06-14 Thread Kenneth Knowles
I did a pass this morning. I believe there is only one release blocker that doesn't already have a fix. If I closed your issue or moved it off the milestone, feel free to have a different opinion and revert my action. Kenn On Mon, Jun 13, 2022 at 5:04 PM Ahmet Altay wrote: > > > On Tue, Jun 7,

Re: Jenkins CI currently unavailable

2022-06-13 Thread Kenneth Knowles
ins UI to unblock? We've >>>>>> turned off the triggers for all post commits, but could turn it on for a >>>>>> select few as well. >>>>>> >>>>>> On Mon, Jun 13, 2022 at 3:31 PM Robert Burke >>>>>> wrote: >>

Re: Jenkins CI currently unavailable

2022-06-13 Thread Kenneth Knowles
Yes, the ghprb plugin was disabled. That was the entire action. I believe my PR will reduce the load caused by the ghprb plugin; we are currently restarting Jenkins to re-enable it. So we can unfreeze master as soon as Jenkins reboots. Basically, if your PR has a precommit status great, otherwise

Re: Jenkins CI currently unavailable

2022-06-13 Thread Kenneth Knowles
I definitely think we should freeze until we have precommits re-enabled. We have a hypothesis that https://github.com/apache/beam/pull/21821 may reduce the impact on the Jenkins master. We would lose phrase triggering on postcommits, but we would still have it on precommits for retriggering after

Re: Join a meeting to help coordinate implementing a Dask Runner for Beam

2022-06-13 Thread Kenneth Knowles
I would love to try to join any meetings if you add me. My calendar is too chaotic to be useful on the when2meet :-) but I can often move things around. Kenn On Wed, Jun 8, 2022 at 2:50 PM Brian Hulette wrote: > Thanks for reaching out, Ryan, this sounds really cool. I added my > availability

Re: Bazel based build

2022-06-09 Thread Kenneth Knowles
A big reason we chose Gradle over Bazel was that we had a hackathon to implement both, and only one person* chose to work on Bazel while the Gradle team attracted many contributors. Having a build system with more widespread knowledge and interest is an important consideration. Basically I suggest

Re: Failing beam_SeedJob

2022-06-02 Thread Kenneth Knowles
configuration of the instance, and the worker > nodes. > > >> >> On Thu, Jun 2, 2022 at 11:47 AM Ahmet Altay wrote: >> >>> /cc @Kenneth Knowles >>> >>> On Thu, Jun 2, 2022 at 8:44 AM Ahmet Altay wrote: >>> >>>> I do not

Re: [ANNOUNCE] Apache Beam 2.38.0 Released

2022-04-20 Thread Kenneth Knowles
Awesome. Thanks! On Wed, Apr 20, 2022 at 4:44 PM Ahmet Altay wrote: > Thank you Daniel! > > On Wed, Apr 20, 2022 at 4:41 PM Daniel Oliveira > wrote: > >> The Apache Beam team is pleased to announce the release of version 2.38.0. >> >> Apache Beam is an open source unified programming model to

Re: BQ vs serialVersionUID

2022-04-20 Thread Kenneth Knowles
Yes, there is something bigger here. There was a thread on this long ago, but I can't find it. It is never enough to add a serialVersionUID. If you add that, you also must define an explicit encoding (via writeReplace / readReplace, etc) that corresponds to the serialVersionUID. This is generally

Re: Metrics-staging Deletion

2022-04-19 Thread Kenneth Knowles
I assume it is a staging deployment of metrics.beam.apache.org. Is that not the case? I don't know what kinds of safety checks might be happening in staging before pushing to prod, though. Kenn On Mon, Apr 18, 2022 at 4:06 PM Elias Segundo Antonio < elias.segu...@wizeline.com> wrote: > Hi

Re: Re: Re: RE: Re: unvendoring bytebuddy

2022-04-12 Thread Kenneth Knowles
en raised so far? Is there additional testing or > dependency analysis that might be helpful? > > On 2022/04/05 18:59:06 Kenneth Knowles wrote: > > Hmm. Too bad the information on the jira is inadequate to explain or > > justify the change. TBH if faced with a conflict between bytebuddy and

Re: [PROPOSAL] Preparing for 2.38.0 Release

2022-04-12 Thread Kenneth Knowles
+1 to cherry-pick. It will make the release broken for enough users that we shouldn't produce a release without the fix. Kenn On Sat, Apr 9, 2022 at 4:05 PM Ahmet Altay wrote: > +1 to cherry picking https://github.com/apache/beam/pull/17329 to the > release branch. > > For completeness,

Re: releasing a new vendored bytebuddy version

2022-04-07 Thread Kenneth Knowles
If I have my timeline right, we might consider unvendoring rather than this release. Or are we going for both in parallel? Or have we given up on unvendoring? Kenn On Thu, Apr 7, 2022 at 2:18 PM Andrew Pilloud wrote: > It looks like Kenn was helping you with the PR >

Re: Updating output watermark on bundle boundaries

2022-04-05 Thread Kenneth Knowles
Coming to this a bit late, but to remind that we switched to a whole-window definition of lateness/droppability. Elements in front of / behind a watermark or other elements should hardly matter for lateness. Anything output within a window should be fine as long as the window itself is treated

Re: Re: RE: Re: unvendoring bytebuddy

2022-04-05 Thread Kenneth Knowles
Hmm. Too bad the information on the jira is inadequate to explain or justify the change. TBH if faced with a conflict between bytebuddy and mockito, working to use mocks less, or in more straightforward ways, would have been my preference. This isn't actually a diamond dep problem that impacts

Re: [ANNOUNCE] New committer: Evan Galpin

2022-03-23 Thread Kenneth Knowles
Thanks for all your contributions! On Mon, Mar 14, 2022 at 1:56 PM Aizhamal Nurmamat kyzy wrote: > Congratulations, Evan! And thank you for all your contributions! > > On Fri, Mar 11, 2022 at 10:14 AM Alexey Romanenko < > aromanenko@gmail.com> wrote: > >> Congratulations, Evan! Well

Re: Deprecating ProcessContext in Java

2022-03-23 Thread Kenneth Knowles
>>> just accept all of them to my @ProcessElement method. And, I could >>> presumably just pass unconstrained generics or Object for any types I >>> couldn't get the right type for in my generic framework without issue. >>> >>> -Daniel >>> >&

Re: Gradle builds: Kotlin or Groovy?

2022-03-23 Thread Kenneth Knowles
itten in Groovy, do we want to change >>> that to Kotlin? If so, we can do that in a follow up PR so we can get this >>> merged sooner rather than later. >>> >>> On Thu, Mar 3, 2022 at 7:17 PM David Huntsperger < >>> dhuntsper...@google.com>

What should be the priority of disabled flaky tests?

2022-03-23 Thread Kenneth Knowles
Hi all, We have a growing backlog of P1 issues and flaky test issues. The emails are not very actionable. If something is "P1" but is not addressed, it is not really P1. This is why we have a bot automatically move things from P2 to P3. I think a test that is currently failing is P0 because in

Re: MapState API reloaded

2022-03-23 Thread Kenneth Knowles
gt; can we rely on that?). I'd therefore suggest to: >>>> >>>> a) deprecate computeIfAbsent() >>>> >>>> b) create compute() as a new method and temporary replacement for >>>> computeIfAbsent() >>>> >>>> c) after s

Re: Deprecating ProcessContext in Java

2022-03-22 Thread Kenneth Knowles
Another aspect to what Reuven brought up is that it is quite difficult to dynamically create a DoFn now that it requires annotations or magic parameters for everything. For example if I have my own DSL and I want to compile it to a DoFn where the state / parameters of the DoFn depend on the input

Re: Possible bug in ElasticsearchIO

2022-03-10 Thread Kenneth Knowles
On Wed, Mar 9, 2022 at 10:02 AM Evan Galpin wrote: > But I'm not sure if an IO throwing away windows (in favour of Global) >> would be acceptable either. BigQueryIO#write (streaming) also has a notion >> of "getFailedWrites" with the same intent as what's in ElasticsearchIO; I >> wonder how

Re: DRAFT - Beam board report March 2022

2022-03-08 Thread Kenneth Knowles
Typo: March 9, aka Wednesday, aka tomorrow On Mon, Mar 7, 2022 at 5:36 AM Kenneth Knowles wrote: > Hi all, > > The March board report is due March 7. Please help me to write it here: > https://s.apache.org/beam-draft-report-2022-03 > > Ideas: > > - highlights from CHA

Re: Beam Java starter project template

2022-03-07 Thread Kenneth Knowles
;>> can't fork the repo because it's empty. Can I either get permissions to >>> directly push or bother you with creating an empty README or some other >>> file so I can fork it and open a PR? Thanks! >>> >>> [image: image.png] >>> >>> On Fr

DRAFT - Beam board report March 2022

2022-03-07 Thread Kenneth Knowles
Hi all, The March board report is due March 7. Please help me to write it here: https://s.apache.org/beam-draft-report-2022-03 Ideas: - highlights from CHANGES.md - interesting technical discussions - integrations with other projects - community events - major user facing

Re: Gradle builds: Kotlin or Groovy?

2022-03-03 Thread Kenneth Knowles
I presume this whole page will go away and be replaced by a use of the new starter repos, no? On Thu, Mar 3, 2022 at 11:02 AM David Cavazos wrote: > Sounds good to me too. > > On Thu, Mar 3, 2022 at 9:00 AM Luke Cwik wrote: > >> Sounds good. >> >> On Thu, Mar 3, 2022 at 6:46 AM David

Re: Beam Java starter project template

2022-02-18 Thread Kenneth Knowles
ing "clone and go", but I'd like >>> to keep them as minimal as possible. We could have another repo like >>> `beam-working-examples` for more complete examples where each subdirectory >>> is a self-contained example with all its build files and everything. >>

Re: MapState API reloaded

2022-02-17 Thread Kenneth Knowles
I think this is a design & documentation bug. The behavior does not make sense. Deviating from Map#computeIfAbsent cannot really be fixed by documentation in bold, even though that is definitely better than nothing. Fixing this should not be considered a breaking change. It should be identical to

Re: Apache Trift vs GRPC summary

2022-02-17 Thread Kenneth Knowles
per - but having just those pointers is a good > start (I likely mixed up gRPC - Thrift bridges with replacing of Thrift > Luke!) > > On Thu, Feb 17, 2022 at 5:28 AM Kenneth Knowles wrote: > >> I can find you that fun mailing list pointer, if you like. Here's a >> starting po

Re: Apache Trift vs GRPC summary

2022-02-16 Thread Kenneth Knowles
I can find you that fun mailing list pointer, if you like. Here's a starting point with the subject "[DISCUSS] Beam data plane serialization tech" https://lists.apache.org/thread/dz24chmm18skzgcmxl2jxookd3yn79r1 Kenn On Wed, Feb 16, 2022 at 10:23 AM Luke Cwik wrote: > Apache Beam never had an

Re: Beam Java starter project template

2022-02-16 Thread Kenneth Knowles
g. > > Regards > Reza > > On Wed, 9 Feb 2022 at 13:56, Robert Burke wrote: > >> SGTM. >> >> On Wed, Feb 9, 2022 at 1:09 PM Kenneth Knowles wrote: >> >>> Based on discussion on https://issues.apache.org/jira/browse/LEGAL-601 >>> I think it

Re: [RFC][Design] Automate Reviewer Assignment

2022-02-14 Thread Kenneth Knowles
Yea, great proposal. I expect we'll discover further refinements through experience more than deliberation, so I don't have any more comments on the doc. Kenn On Mon, Feb 14, 2022 at 9:04 AM Kerry Donny-Clark wrote: > Thanks Danny, we can try this out and update as well. Everyone, please let >

Re: Beam Java starter project template

2022-02-09 Thread Kenneth Knowles
olicy. We can easily set it up to be a noop for the user. So my overall take is that we should go ahead with ASL2 and a simple NOTICE file. Check the Jira for details. Kenn On Mon, Feb 7, 2022 at 10:47 AM Kenneth Knowles wrote: > And I've created the repos just now. > > Kenn >

Re: Inconsistent behavior when parsing boolean flags across different APIs in Python SDK

2022-02-09 Thread Kenneth Knowles
Thanks for writing this up. This is an instant classic sort of bug/inconsistency. Kenn On Tue, Feb 8, 2022 at 11:46 AM Anand Inguva wrote: > Hi all, > > For the Python SDK, there is some inconsistency across different APIs when > it comes to parsing the boolean flags. The issue is briefly

Re: KafkaIO.write and Avro

2022-02-09 Thread Kenneth Knowles
> > Just an idea. > Matt > > On Wed, Feb 9, 2022 at 3:40 PM Kenneth Knowles wrote: > >> Just a quick type check: is it the case that a Serializer is >> expected to be able to properly serde any subclass of object? More >> generally that any Serializer should be ab

Re: KafkaIO.write and Avro

2022-02-09 Thread Kenneth Knowles
Just a quick type check: is it the case that a Serializer is expected to be able to properly serde any subclass of object? More generally that any Serializer should be able to properly serde V? Typically this isn't the case. Not saying we shouldn't make the proposed change, but it could result in

Re: Beam Java starter project template

2022-02-07 Thread Kenneth Knowles
And I've created the repos just now. Kenn On Mon, Feb 7, 2022 at 10:39 AM Kenneth Knowles wrote: > Legal question asked at https://issues.apache.org/jira/browse/LEGAL-601 > > Kenn > > On Fri, Feb 4, 2022 at 7:58 AM Danny McCormick > wrote: > >> Sure - I'm happy t

Re: Beam Java starter project template

2022-02-07 Thread Kenneth Knowles
tests > > Always happy to help with or consult on any actions issues  > > Thanks, > Danny > > On Fri, Feb 4, 2022 at 10:21 AM Kerry Donny-Clark > wrote: > >> Danny has extensive experience with GitHub actions, and may be able to >> help out. >> Kerry >> >

Re: [VOTE] Release 2.36.0, release candidate #3

2022-02-07 Thread Kenneth Knowles
+1 (binding) On Fri, Feb 4, 2022 at 8:00 PM Reuven Lax wrote: > +1 (binding) > > On Fri, Feb 4, 2022 at 6:21 PM Ahmet Altay wrote: > >> +1 (binding) >> >> I validated python quick start examples with the direct runner. Thank you >> Emily! >> >> On Fri, Feb 4, 2022 at 10:11 AM Alexey Romanenko

Are runners-spark2 and beam-site branches obsolete?

2022-02-04 Thread Kenneth Knowles
Hi all, I was cleaning up branches on the main repo like a lot of "patch" and "revert" branches that GitHub creates through the UI. I noticed the following branches that represent real work and I was thinking about adding them to the list of "protected" branches so they cannot be accidentally

Re: Wiki Edit Access

2022-02-04 Thread Kenneth Knowles
Done! On Fri, Feb 4, 2022 at 5:01 AM Danny McCormick wrote: > Hey folks, would someone mind granting me access to the Confluence wiki? > I'd like the ability to update the Go design docs and tips pages. My id > is dannymccormick. > > Thanks, > Danny >

Re: Javascript SDK

2022-02-04 Thread Kenneth Knowles
usybody > > On Thu, Feb 3, 2022, 3:29 PM Kenneth Knowles wrote: > >> We did the same for the Go SDK for some time. I imagine just "not doing >> the work to release it" suffices? Maybe +Robert Burke has >> some other memories of how to not release. >> >> Ke

Re: Beam Summit Status Report - 2/2

2022-02-03 Thread Kenneth Knowles
Hi Danielle, Would it be possible in the future to inline the content of the status report to the email? This way it will be archived with the mailing list. Also it will make it more likely that casual readers of the list may skim it. Kenn On Wed, Feb 2, 2022 at 3:39 PM Danielle Syse wrote: >

Re: Beam Java starter project template

2022-02-03 Thread Kenneth Knowles
> If we all agree on the repo names, who can help us create them? > > > > On Thu, Jan 27, 2022 at 12:58 PM Robert Bradshaw > wrote: > >> > >> On Tue, Jan 18, 2022 at 6:17 AM Kenneth Knowles > wrote: > >> > > >> > Agree with Luke here

Re: Timestamp Verification when Outputting from FinishBundleContext Vs. ProcessContext

2022-02-03 Thread Kenneth Knowles
One reason is the lack of `elem` used here: https://github.com/apache/beam/blob/15048929495ad66963b528d5bd71eb7b4a844c96/runners/core-java/src/main/java/org/apache/beam/runners/core/SimpleDoFnRunner.java#L440 That might be the whole reason. Still, your point is a good one that the sort of

Re: @Timer and Kafka SDF not interacting properly

2022-02-03 Thread Kenneth Knowles
I know we chatted about this off-list, but I wanted to just follow up and see if you figured it out. Sounds like it could be an important bug in the DirectRunner. I don't recall whether it reproduces on e.g. local Flink/Spark Runner or the Python local portable runner. Kenn On Tue, Feb 1, 2022

Re: Javascript SDK

2022-02-03 Thread Kenneth Knowles
ngratulations to all who worked on this project. >>> >>> On Fri, Jan 14, 2022 at 4:41 PM Kenneth Knowles wrote: >>> >>>> This was super fun, and I really hope it can be an inspiration to >>>> others that you can build a working Beam SD

Re: Thoughts from a first time contributor

2022-02-03 Thread Kenneth Knowles
Thanks for writing all this up and for putting up PRs to improve things! Kenn On Mon, Jan 31, 2022 at 12:17 PM Danny McCormick wrote: >  Hey folks, my name is Danny - I recently completed my first Beam > PR[1] (a small extension to the Go Dataflow runner) and am planning on > becoming a more

Re: [DISCUSS] Migrate Jira to GitHub Issues?

2022-01-31 Thread Kenneth Knowles
a label added as commentary or is it a resolution status?) and issue type (something could be marked "bug" and "feature request"). Kenn On Mon, Jan 31, 2022 at 8:38 AM Kenneth Knowles wrote: > Great. I added a few items to the "summary of discussion" even though th

Re: [DISCUSS] Migrate Jira to GitHub Issues?

2022-01-31 Thread Kenneth Knowles
tomation for porting tools. >> Manual porting cons: >> - Unambiguous situations get (unnecessary) human attention. >> >> A compromise might be to build a simple tool for porting jiras, but don't >> automatically run it on everything. >> >> On Tue, Jan 18, 202

Re: [PROPOSAL] Preparing for 2.37.0 Release

2022-01-31 Thread Kenneth Knowles
+1 and don't hesitate to ask for help Kenn On Fri, Jan 28, 2022 at 3:41 PM Ahmet Altay wrote: > +1 and thank you Brian! > > On Fri, Jan 28, 2022 at 2:47 PM Brian Hulette wrote: > >> Hi all, >> >> The next (2.37.0) release branch cut is scheduled for 9 February, >> according to >> the release

Re: beam/jackson NPE during test

2022-01-31 Thread Kenneth Knowles
erialization roundtrip. > > Cheers, > Kellen > > > > > On Fri, Jan 28, 2022 at 11:46 AM Kenneth Knowles wrote: > >> If I understand correctly, the documentation does imply that such >> synchronization at [2] should not be necessary and that [4] and [

Re: [VOTE] Release 2.36.0, release candidate #1

2022-01-29 Thread Kenneth Knowles
+1 (binding) I tried a couple more pipelines and flag combinations I particularly care about :-) Kenn On Fri, Jan 28, 2022 at 10:03 PM Valentyn Tymofieiev wrote: > +1 (non-binding). Verified that Dataflow containers have necessary Beam > requirements. > > On Fri, Jan 28, 2022 at 3:16 PM

Re: beam/jackson NPE during test

2022-01-28 Thread Kenneth Knowles
If I understand correctly, the documentation does imply that such synchronization at [2] should not be necessary and that [4] and [5] are correct. Of course it would be better if the docs said _how_ this guarantee is provided, or perhaps more than that it is a matter of restating it as a

Re: Swap gradle task.create() with task.register() if applicable?

2022-01-28 Thread Kenneth Knowles
+1 very good initiative. Kenn On Fri, Jan 28, 2022 at 7:27 AM Kerry Donny-Clark wrote: > Sounds great. Is there a Jira issue? Also, are you planning on > implementing this? > > On Thu, Jan 27, 2022 at 9:00 PM Brian Hulette wrote: > >> I'm +1 for this, it's frustrating to run into issues with

Re: Beam Java starter project template

2022-01-27 Thread Kenneth Knowles
. >>> >>> Most Go tooling is pretty good at doing package lookups and similar, but >>> only after the a module has been loaded into your cache. >>> >>> On Tue, Jan 18, 2022 at 6:20 AM Kenneth Knowles wrote: >>> >>>> I want to clarify

IntelliJ import treating sdks/java/core/src/test/java not as source

2022-01-26 Thread Kenneth Knowles
Hi all, I don't remember if I asked this before or figured it out. My most recent refresh of the IntelliJ project does not index sdks/java/core/src/test/java as source files. I looked at the history of the `idea { }` blocks in the gradle configs and I did not see anything obvious. Has anyone else

Re: Hello Beam Community!

2022-01-26 Thread Kenneth Knowles
Welcome to Beam! On Wed, Jan 26, 2022 at 8:29 AM Ahmet Altay wrote: > Welcome! > > On Tue, Jan 25, 2022 at 10:58 PM Pablo Estrada wrote: > >> Welcome Danielle! >> >> On Tue, Jan 25, 2022 at 2:48 PM Robert Bradshaw >> wrote: >> >>> Welcome! >>> >>> On Tue, Jan 25, 2022 at 1:33 PM Danielle Syse

Re: PMC help - moving vendored artifact source from dev to release

2022-01-20 Thread Kenneth Knowles
Just to close the loop here: Looks like Cham did this a few days ago. (thanks!) Kenn On Tue, Jan 18, 2022 at 9:09 AM Emily Ye wrote: > Hello dev@/PMCs! > > I just newly released the vendored gRPC and calcite artifacts. Per the > "Deploy source release to dist.apache.org" step of the release

<    1   2   3   4   5   6   7   8   9   10   >