Re: Java Direct Runner technical documentation is coming soon!

Very nice ! Etienne Le mercredi 23 mai 2018 à 11:20 -0700, Huygaa Batsaikhan a écrit : > Hi devs, > Robin Qu and I, both new Beam contributors, have been working on adding new > features in Java Direct Runner. However, > our experience was not that smooth because there were no technical

Re: Missing copyright notices for shaded packages

I merged Boyuan's PR since it seems like a clear improvement. But it looks like the suggested approach of overapproximating what is bundled in a particular artifact is contrary to http://www.apache.org/dev/licensing-howto.html#bundled-vs-non-bundled Kenn On Wed, May 23, 2018 at 1:52 PM Scott

Re: The full list of proposals / prototype documents

Thank you everybody for positive feedback and sending me links to design docs. I’m going to complete the full list and create PR for review. Griselda Perhaps, it was a bit of misunderstanding here, let me explain what I assume under creating of such page. Actually, for now I’m going to

Re: Missing copyright notices for shaded packages

For the benefit of the thread, since I know the ASF site can take some time to navigate: http://www.apache.org/dev/licensing-howto.html http://www.apache.org/legal/ On Wed, May 23, 2018 at 1:52 PM Scott Wegner wrote: > FYI, I've opened

Re: The full list of proposals / prototype documents

OK, I will also put a list here of those I know off the top of my head. Some are redundant with Etienne's but short links that I can think of: https://s.apache.org/a-new-dofn https://s.apache.org/beam-triggers https://s.apache.org/beam-sink-triggers https://s.apache.org/beam-runner-composites

[Proposal] Automation For Beam Dependency Check

Hello, I have a proposal to automate Beam dependency check. Since some Beam dependent packages are out-of-date, we want to identify them and check for dependency updates regularly in the future. Generally, we have couple options to do it: 1. Implementing a Jenkins job that check dependency

Re: Missing copyright notices for shaded packages

I'm not sure what the right approach is, but the jars generated off of current master contain no LICENSE, NOTICE, or other metadata about dependencies. (The ones checked in at the root of the beam repo are not bundled.) Andrew On Thu, May 24, 2018 at 7:05 AM Kenneth Knowles

Re: Existing transactionality inconsistency in the Beam Java State API

I think Kenn's second option accurately reflects my memory of the original intentions: 1. I remember we we considered either using the Future interface or calling the ReadableState interface a future, and explicitly said "no, future implies asynchrony and that the value returned by `get` won't

Re: SQL shaded jars don't work. How to test?

Were you able to figure out how to fix them or still having problems? On Thu, May 24, 2018 at 9:27 AM Andrew Pilloud wrote: > I spent all of yesterday investigating and fixing dependency issues > outside of SQL. I really regret the decision to write a test for this. > Would

Re: SQL shaded jars don't work. How to test?

I spent all of yesterday investigating and fixing dependency issues outside of SQL. I really regret the decision to write a test for this. Would it be acceptable for us to put testing with the output jar behind a flag like we do for failOnWarning? On Wed, May 23, 2018 at 5:21 PM Kenneth Knowles

Re: Missing copyright notices for shaded packages

> > jars generated off of current master contain no LICENSE, NOTICE, or other > metadata > We should fix this. This may be an issue. suggested approach of overapproximating >> > The purpose of quoted sentence above ("A part of...") is to qualify when a particular portion applies. It contains

Re: Gradle closure hack

This is a good idea because it is in tune with how people familiar with Gradle think. When I started applyJavaNature, I couldn't think of a good way to get certain closures to apply to more then one task. For example, we have a default shading configuration which we want to apply to both the main

Re: Kubernetes cluster of apache-beam-testing project

I can update the version of the io-datastores cluster master and node pools. I don't have access to the kubectl version on the executors ... I can ask around for that. On Wed, May 23, 2018 at 4:59 AM Kamil Szewczyk wrote: > Dear Beam Devs, > > we are using kubernetes to on

Re: SQL shaded jars don't work. How to test?

I think I'm down to 11 packages with failures, some of which might be precommits that don't run outside of Jenkins. Its not an issue of figuring them out, it is an issue of time spent doing so. On Thu, May 24, 2018 at 9:31 AM Lukasz Cwik wrote: > Were you able to figure out

Re: Existing transactionality inconsistency in the Beam Java State API

I see what you mean but I don't agree that futures imply anything other than "it is a value that you have to force", with deliberately many possible implementations. When at the point in 1 and you've got interface ReadableState { T read() } and you want to improve performance,

Re: Kubernetes cluster of apache-beam-testing project

Seems like kubectl v1.9.0 was installed on those nodes. Here is the puppet script I found that used to download and install kubectl on beam-jenkins. https://github.com/apache/infrastructure-puppet/blob/95aed4f4f7cdce79a1f0fc8b1e581f13fedf63ff/modules/build_slaves/manifests/kube.pp On Thu, May 24,

Re: Kubernetes cluster of apache-beam-testing project

Updated the io-datastores kubernetes cluster in apache-beam-testing to version 1.9.7-gke.1 for the master and version 1.9.7-gke.1 for the node pool. On Thu, May 24, 2018 at 10:31 AM Yifan Zou wrote: > Seems like kubectl v1.9.0 was installed on those nodes. Here is the

Re: SQL shaded jars don't work. How to test?

If it is taking days of work we should definitely put it behind a flag and do it incrementally, find a way to share the work. If our tests aren't running on the actual artifacts, then we don't really have assurance that they work. That should interest just about everyone. (though it is also

Re: [Proposal] Automation For Beam Dependency Check

Thanks Yifan. Added some comments. I think having regularly generated human reports on outdated decencies of Beam SDKs will be extremely helpful in keeping Beam in a healthy state. - Cham On Thu, May 24, 2018 at 7:08 AM Yifan Zou wrote: > Hello, > > I have a proposal to

Re: Missing copyright notices for shaded packages

Thanks Boyuan for the prompt resolution! : D -P/ On Thu, May 24, 2018 at 4:33 PM Boyuan Zhang wrote: > Hey all, > > Release blocker https://issues.apache.org/jira/browse/BEAM-4393 marked as > Resolved. > > > Boyuan > > On Wed, May 23, 2018 at 1:52 PM Scott Wegner

Re: Kubernetes cluster of apache-beam-testing project

We had to file a JIRA against Apache infra for getting kubectl installed: https://issues.apache.org/jira/browse/INFRA-14819 Not sure if we need to get infra involved for updating the version as well. - Cham On Thu, May 24, 2018 at 10:42 AM Alan Myrvold wrote: > Updated

Re: Closing (automatically?) inactive pull requests

+1 for automatically closing. Currently, contribution guide mentions that pull requests without responses to actionable comments become stale (and closed) after two months but last I checked there were many pull requests that met this criteria but had not been closed after many months. So seems

Re: The full list of proposals / prototype documents

Great that you take this action Alexey !Here are the links I have, there is duplicates with the ones you already received and maybe old docs as well: https://docs.google.com/document/d/1MtBZYV7NAcfbwyy9Op8STeFNBxtljxgy69FkHMvhTMA/edithttps://docs.google.com/document/d/1

Re: SQL shaded jars don't work. How to test?

I've make the SQL PR able to be merged standalone so I can get back to work. I've also opened issues to track the things I found and dumped my work in progress into a PR. https://issues.apache.org/jira/browse/BEAM-4402 https://issues.apache.org/jira/browse/BEAM-4403

Re: The full list of proposals / prototype documents

Thanks for sharing these. I also put together a design doc template based on common styling / sections I saw in the docs listed above. Others are free to use it as they'd like. https://docs.google.com/document/d/1kVePqjt2daZd0bQHGUwghlcLbhvrny7VpflAzk9sjUg/edit?usp=sharing On Thu, May 24, 2018

Re: [PROPOSAL] Preparing 2.5.0 release next week

The license inclusion issue that was brought up on the thread has been resolved https://issues.apache.org/jira/browse/BEAM-4393. JB, you find any other release related issues? On Fri, May 18, 2018 at 10:33 AM Lukasz Cwik wrote: > I believe JB is referring to >

Re: Missing copyright notices for shaded packages

Hey all, Release blocker https://issues.apache.org/jira/browse/BEAM-4393 marked as Resolved. Boyuan On Wed, May 23, 2018 at 1:52 PM Scott Wegner wrote: > FYI, I've opened https://issues.apache.org/jira/browse/BEAM-4393 to track > this work and marked it as a 2.5.0 release

Re: [PROPOSAL] Preparing 2.5.0 release next week

Do we have adequate integration test coverage of the actual released artifacts that https://issues.apache.org/jira/browse/BEAM-4402 is not a blocker? On Thu, May 24, 2018 at 4:35 PM Lukasz Cwik wrote: > The license inclusion issue that was brought up on the thread has been >