Re: Configuring Jenkins
This is very cool….. I definitely need to look at this a bit as there a few there projects that would love to have something similar. :) Dan > On Nov 15, 2016, at 4:01 AM, Davor Bonaci <da...@google.com.INVALID> wrote: > > Hi everybody, > As I'm sure everybody knows, we use Apache's Jenkins instance for all our > testing, including pre-commit, post-commit, nightly snapshot, etc. (Travis > CI is a backup system and recommended for individual forks only.) > > Managing Jenkins projects has been a big pain point so far. Among other > reasons, only a few of us have access to configure it, way too few of us > have visibility into what those jobs do, and nobody has any visibility into > changes being made or an opportunity to comment on them. > > Well, not any more! I was playing a little bit with Jenkins DSL plugin and > was able to move our configuration out of Jenkins and into the git > repository. I've done it as a proof of concept for the website repository > only [1], but Jason is planning on extending that work to the main > repository. Look for a PR shortly! > > Going forward, anyone can see what our Jenkins jobs are doing, and anyone > can add new jobs or improve existing ones by simply proposing a pull > request to change the configuration. Finally, the project maintains a > history in source repository, instead of direct changes without much > accountability. > > How this works? There's a "seed" job that periodically applies > configuration specified in the source repository into Jenkins. Currently, > this happens once per day. If you modify the configuration in the source > repository, it will be applied within 24 hours. If you, however, modify the > configuration in Jenkins directly, it will revert back to whatever is > specified in the code repository also within 24 hours. > > How to understand Jenkins DSL? There are many resources available; I've > found Jenkins Job DSL API [2] particularly helpful. > > I hope you are excited to have this feature available to us! If you have > any thoughts on improving this further, please comment. Thanks! > > Davor > > [1] https://github.com/apache/incubator-beam-site/pull/80 > [2] https://jenkinsci.github.io/job-dsl-plugin/ -- Daniel Kulp dk...@apache.org - http://dankulp.com/blog Talend Community Coder - http://coders.talend.com
Re: Documentation for IDE setup
Just a follow up based on some discoveries while trying to rebase my branch on master this morning. Eclipse JDT outputs methods/fields into class files in a different order than Oracle compiler. That’s perfectly acceptable from a “binary compatibility” standpoint, but it has a side effect of causing potential problems with AutoValue. If the AutoValue class tries to get it’s values from an interface, the methods on the interface will come in a different order than with Oracle and the resulting constructor/fields/etc… will be different. Based on some experiments and back and forth with Dan H., I believe the best fix is to explicitly define the properties on the AutoValue class as if it didn’t pull those via the interface. Thus, the APT processing gets the attributes in the order intended and generates the right code. The alternative would be to use the Builder pattern instead of the constructor, but that requires more code to be written than just defining the attributes in the right order. However, if you are already defining a Builder, that might be the best option. Anyway, something to be aware of when using the AutoValue things. Once we get the branch merged, travis should automatically pick this up. Dan > On Oct 14, 2016, at 11:37 AM, Daniel Kulp <dk...@apache.org> wrote: > > >> On Oct 14, 2016, at 10:06 AM, Jesse Anderson <je...@smokinghand.com> wrote: >> >> Last week I imported Beam with IntelliJ and everything worked. >> >> That said, I tried to import the Eclipse project and that doesn't compile >> anymore. I didn't have time to figure out what happened though. >> > > I have a pull request https://github.com/apache/incubator-beam/pull/1094 that > fixes the compile issues. It has two LGTM’s, just needs someone to merge it. > > With eclipse, you need to have all the needed m2e connectors. Some of them > (find bugs, check style) can be auto-detected and installed when beam is > first imported. The apt one doesn’t. You need to go to the eclipse > marketplace, install it, then configure it in the Eclipse properties to turn > on the “experimental” m2e-apt processing. Once you do that, a refresh of > the maven projects should result in it building/compiling. > > Running tests is another matter. Since eclipse compiles everything in a > module in one pass (instead of two like maven), one of the apt processors > doesn’t know where to output files and always dumps the files in /classes > instead of /test-classes. Thus, any test that relies on a runner will > likely fail as it results in the “test” versions of various services from > core being picked up. A simple: > > rm sdks/java/core/target/classes/META-INF/services/* > > From the command line will fix that. That should also be documented on the > IDE page until someone can figure out how to work around it. > > Dan > > > >> On Fri, Oct 14, 2016 at 1:21 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> >>> Hi Christian, >>> >>> IntelliJ doesn't need any special config (maybe the code style can be >>> documented or imported). >>> >>> Anyway, agree to add such on website in the contribute directory. I >>> think it could be part of the contribution-guide as it's first setup step. >>> >>> Regards >>> JB >>> >>> On 10/14/2016 10:17 AM, Christian Schneider wrote: >>>> Hello all, >>>> >>>> I am new to the beam community and currently start making myself >>>> familiar with the code. I quickly found the contribution guide and was >>>> able to clone the code and build beam using maven. >>>> >>>> The first obstacle I faced was getting the code build in eclipse. I >>>> naively imported as existing maven projects but got lots of compile >>>> errors. After talking to Dan Kulp we found that this is due to the apt >>>> annotation processing for auto value types. Dan explained me how I need >>>> to setup eclipse to make it work. >>>> >>>> I still got 5 compile errors (Some bound mismatch at Read.bounded, and >>>> one ambiguous method empty). These errors seem to be present for >>>> everyone using eclipse and Dan works on it. So I think this is not a >>>> permanent problem. >>>> >>>> To make it easier for new people I would like to write a documentation >>>> about the IDE setup. I can cover the eclipse part but I think intellij >>>> should also be described. >>>> >>>> I already started with it and placed it in /contribute/ide-setup
Re: [PROPOSAL] Introduce review mailing list and provide update on open discussion
> On Oct 14, 2016, at 7:46 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > I think we agreed on most of the points. We also agreed that points 4 & 5 > should be a best effort and not "enforced”. 4 and 5 are really just needed when any “significant change” are part of the discussion. Things like whitespace and typos and such don’t need to be pushed back to the list. However, if you run into something that “just doesn’t work”, that should be reflected back.As an example, my first GridFS PR, when Eugene basically stated that the entire structure needed to change, (from doing everything in the Source to splitting into a simpler Source and a ParDo) that’s something that should have been brought back to the list in a summary. BTW: this ALSO goes for anything in google docs. Some of the proposals are in google doc form so any changes/discussions at google are also not “at Apache”. In looking at the proposed workflow, I also think when the proposal is accepted/complete/whatever, the google doc needs to be exported to something and attached to the JIRA so it is properly archived at Apache. Personally, I’d prefer limiting the use of google docs, but I do understand that having the diagrams can be important. Not sure how to deal with that in email. One thing the incubator does with proposals from their wiki is require the full TEXT of proposals be copied into the email (so searchable and also clear what version is be being looked at) with the link to the full doc. Dan > > If there's no objection, I will create the review mailing list and update the > github integration configuration. > > Thanks all for your comments and feebacks ! > Regards > JB > > On 10/06/2016 01:53 PM, Jean-Baptiste Onofré wrote: >> Hi team, >> >> following the discussion we had about technical discussion that should >> happen on the mailing list, I would like to propose the following: >> >> 1. We create a new mailing list: rev...@beam.incubator.apache.org. >> 2. We configure github integration to send all pull request comments on >> review mailing list. It would allow to track and simplify the way to >> read the comments and to keep up to date. >> 3. A technical discussion should be send on dev mailing list with the >> [DISCUSS] keyword in the subject. >> 4. Once a discussion is open, the author should periodically send an >> update on the discussion (once a week) containing a summary of the last >> exchanges happened on the Jira or github (quick and direct summary). >> 5. Once we consider the discussion close (no update in the last two >> weeks), the author send a [CLOSE] e-mail on the thread. >> >> WDYT ? >> >> Regards >> JB > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com -- Daniel Kulp dk...@apache.org - http://dankulp.com/blog Talend Community Coder - http://coders.talend.com
Re: Documentation for IDE setup
> On Oct 14, 2016, at 10:06 AM, Jesse Anderson <je...@smokinghand.com> wrote: > > Last week I imported Beam with IntelliJ and everything worked. > > That said, I tried to import the Eclipse project and that doesn't compile > anymore. I didn't have time to figure out what happened though. > I have a pull request https://github.com/apache/incubator-beam/pull/1094 that fixes the compile issues. It has two LGTM’s, just needs someone to merge it. With eclipse, you need to have all the needed m2e connectors. Some of them (find bugs, check style) can be auto-detected and installed when beam is first imported. The apt one doesn’t. You need to go to the eclipse marketplace, install it, then configure it in the Eclipse properties to turn on the “experimental” m2e-apt processing. Once you do that, a refresh of the maven projects should result in it building/compiling. Running tests is another matter. Since eclipse compiles everything in a module in one pass (instead of two like maven), one of the apt processors doesn’t know where to output files and always dumps the files in /classes instead of /test-classes. Thus, any test that relies on a runner will likely fail as it results in the “test” versions of various services from core being picked up. A simple: rm sdks/java/core/target/classes/META-INF/services/* From the command line will fix that. That should also be documented on the IDE page until someone can figure out how to work around it. Dan > On Fri, Oct 14, 2016 at 1:21 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi Christian, >> >> IntelliJ doesn't need any special config (maybe the code style can be >> documented or imported). >> >> Anyway, agree to add such on website in the contribute directory. I >> think it could be part of the contribution-guide as it's first setup step. >> >> Regards >> JB >> >> On 10/14/2016 10:17 AM, Christian Schneider wrote: >>> Hello all, >>> >>> I am new to the beam community and currently start making myself >>> familiar with the code. I quickly found the contribution guide and was >>> able to clone the code and build beam using maven. >>> >>> The first obstacle I faced was getting the code build in eclipse. I >>> naively imported as existing maven projects but got lots of compile >>> errors. After talking to Dan Kulp we found that this is due to the apt >>> annotation processing for auto value types. Dan explained me how I need >>> to setup eclipse to make it work. >>> >>> I still got 5 compile errors (Some bound mismatch at Read.bounded, and >>> one ambiguous method empty). These errors seem to be present for >>> everyone using eclipse and Dan works on it. So I think this is not a >>> permanent problem. >>> >>> To make it easier for new people I would like to write a documentation >>> about the IDE setup. I can cover the eclipse part but I think intellij >>> should also be described. >>> >>> I already started with it and placed it in /contribute/ide-setup. Does >>> that make sense? >>> >>> I currently did not link to it from anywhere. I think it should be >>> linked in the contribute/index and in the Contribute menu. >>> >>> Christian >>> >> >> -- >> Jean-Baptiste Onofré >> jbono...@apache.org >> http://blog.nanthrax.net >> Talend - http://www.talend.com >> -- Daniel Kulp dk...@apache.org - http://dankulp.com/blog Talend Community Coder - http://coders.talend.com
Re: [3/4] incubator-beam git commit: Remove KeyedResourcePool
I just submitted a pull request that fixes the code as well as cherry-picks the yaml change from the last branch. Dan > On Oct 13, 2016, at 10:48 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > > Indeed the .travis.yml has not been merged. I gonna fix that. > > Sorry about that. > > Regards > JB > > On 10/13/2016 04:37 PM, Daniel Kulp wrote: >> >> This is in m2e.That said, it looks like the travis.yml file wasn’t >> merged from my “eclipse” branch so Travis wasn’t actually running agains the >> eclipse compiler. That would have caught this. JB and I will investigate >> how that got lost in the merge to master. >> >> A "mvn -Peclipse-jdt clean install” in direct-java would show the same error. >> >> >> Dan >> >> >> >>> On Oct 13, 2016, at 10:05 AM, Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>> Hi Dan, >>> >>> You mean directly building in Eclipse I guess using m2e ? >>> >>> Regards >>> JB >>> >>> On 10/13/2016 03:59 PM, Daniel Kulp wrote: >>>> >>>> Just an FYI: this commit has caused things to not build in Eclipse, but >>>> I’m not exactly sure why. The errors are in place where methods of the >>>> exact signature just moved into an internal class so I’m not yet sure why >>>> it’s causing an issue. >>>> >>>> DescriptionResourcePathLocationType >>>> Bound mismatch: The type Read.Bounded is not a valid substitute >>>> for the bounded parameter >>> InputT,OutputT>> of the type AppliedPTransform<InputT,OutputT,TransformT> >>>> BoundedReadEvaluatorFactory.java >>>> /beam-runners-direct-java/src/main/java/org/apache/beam/runners/direct >>>> line 134Java Problem >>>> >>>> >>>> Dan >>>> >>>> >>>> >>>> >>>> On 2016-10-06 18:31 (-0400), lc...@apache.org wrote: >>>>> Remove KeyedResourcePool >>>>> >>>>> This interface is no longer used. Instead, the runner ensures that >>>>> bundles will be provided containing the appropriate input to the >>>>> TestStreamEvaluatorFactory. >>>>> >>>>> >>>>> Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo >>>>> Commit: >>>>> http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/41fb16f0 >>>>> Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/41fb16f0 >>>>> Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/41fb16f0 >>>>> >>>>> Branch: refs/heads/master >>>>> Commit: 41fb16f014a79d2b9c149c5b369db12b61c4c774 >>>>> Parents: 7306e16 >>>>> Author: Thomas Groh <tg...@google.com> >>>>> Authored: Wed Oct 5 13:12:48 2016 -0700 >>>>> Committer: Luke Cwik <lc...@google.com> >>>>> Committed: Thu Oct 6 15:14:38 2016 -0700 >>>>> >>>>> -- >>>>> .../direct/BoundedReadEvaluatorFactory.java | 40 +++-- >>>>> .../beam/runners/direct/DirectRunner.java | 2 + >>>>> .../beam/runners/direct/EmptyInputProvider.java | 49 ++ >>>>> .../direct/ExecutorServiceParallelExecutor.java | 27 ++- >>>>> .../runners/direct/FlattenEvaluatorFactory.java | 18 +- >>>>> .../beam/runners/direct/KeyedResourcePool.java | 47 -- >>>>> .../runners/direct/LockedKeyedResourcePool.java | 95 --- >>>>> .../beam/runners/direct/RootInputProvider.java | 41 + >>>>> .../runners/direct/RootProviderRegistry.java| 65 >>>>> .../direct/RootTransformEvaluatorFactory.java | 42 - >>>>> .../direct/TestStreamEvaluatorFactory.java | 39 +++-- >>>>> .../direct/TransformEvaluatorRegistry.java | 17 +- >>>>> .../direct/UnboundedReadEvaluatorFactory.java | 56 --- >>>>> .../direct/BoundedReadEvaluatorFactoryTest.java | 3 +- >>>>> .../direct/FlattenEvaluatorFactoryTest.java | 3 +- >>>>> .../direct/LockedKeyedResourcePoolTest.java | 163 --- >>>>> .../direct/TestStreamEvaluatorFactoryTest.java | 3 +- >>>>> .../U
Re: [3/4] incubator-beam git commit: Remove KeyedResourcePool
This is in m2e.That said, it looks like the travis.yml file wasn’t merged from my “eclipse” branch so Travis wasn’t actually running agains the eclipse compiler. That would have caught this. JB and I will investigate how that got lost in the merge to master. A "mvn -Peclipse-jdt clean install” in direct-java would show the same error. Dan > On Oct 13, 2016, at 10:05 AM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > > Hi Dan, > > You mean directly building in Eclipse I guess using m2e ? > > Regards > JB > > On 10/13/2016 03:59 PM, Daniel Kulp wrote: >> >> Just an FYI: this commit has caused things to not build in Eclipse, but >> I’m not exactly sure why. The errors are in place where methods of the >> exact signature just moved into an internal class so I’m not yet sure why >> it’s causing an issue. >> >> Description ResourcePathLocationType >> Bound mismatch: The type Read.Bounded is not a valid substitute for >> the bounded parameter > InputT,OutputT>> of the type AppliedPTransform<InputT,OutputT,TransformT> >> BoundedReadEvaluatorFactory.java >> /beam-runners-direct-java/src/main/java/org/apache/beam/runners/direct line >> 134Java Problem >> >> >> Dan >> >> >> >> >> On 2016-10-06 18:31 (-0400), lc...@apache.org wrote: >>> Remove KeyedResourcePool >>> >>> This interface is no longer used. Instead, the runner ensures that >>> bundles will be provided containing the appropriate input to the >>> TestStreamEvaluatorFactory. >>> >>> >>> Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo >>> Commit: >>> http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/41fb16f0 >>> Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/41fb16f0 >>> Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/41fb16f0 >>> >>> Branch: refs/heads/master >>> Commit: 41fb16f014a79d2b9c149c5b369db12b61c4c774 >>> Parents: 7306e16 >>> Author: Thomas Groh <tg...@google.com> >>> Authored: Wed Oct 5 13:12:48 2016 -0700 >>> Committer: Luke Cwik <lc...@google.com> >>> Committed: Thu Oct 6 15:14:38 2016 -0700 >>> >>> -- >>> .../direct/BoundedReadEvaluatorFactory.java | 40 +++-- >>> .../beam/runners/direct/DirectRunner.java | 2 + >>> .../beam/runners/direct/EmptyInputProvider.java | 49 ++ >>> .../direct/ExecutorServiceParallelExecutor.java | 27 ++- >>> .../runners/direct/FlattenEvaluatorFactory.java | 18 +- >>> .../beam/runners/direct/KeyedResourcePool.java | 47 -- >>> .../runners/direct/LockedKeyedResourcePool.java | 95 --- >>> .../beam/runners/direct/RootInputProvider.java | 41 + >>> .../runners/direct/RootProviderRegistry.java| 65 >>> .../direct/RootTransformEvaluatorFactory.java | 42 - >>> .../direct/TestStreamEvaluatorFactory.java | 39 +++-- >>> .../direct/TransformEvaluatorRegistry.java | 17 +- >>> .../direct/UnboundedReadEvaluatorFactory.java | 56 --- >>> .../direct/BoundedReadEvaluatorFactoryTest.java | 3 +- >>> .../direct/FlattenEvaluatorFactoryTest.java | 3 +- >>> .../direct/LockedKeyedResourcePoolTest.java | 163 --- >>> .../direct/TestStreamEvaluatorFactoryTest.java | 3 +- >>> .../UnboundedReadEvaluatorFactoryTest.java | 8 +- >>> 18 files changed, 269 insertions(+), 449 deletions(-) >>> -- >>> >>> >>> http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/41fb16f0/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java >>> -- >>> diff --git >>> a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java >>> >>> b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java >>> index 4936ad9..326a535 100644 >>> --- >>> a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java >>> +++ >>> b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java >>> @@ -39,28 +39,13 @@ import org.apache.beam.sdk
Re: [3/4] incubator-beam git commit: Remove KeyedResourcePool
e());
> + pending.addAll(initialInputs);
>pendingRootBundles.put(root, pending);
> }
> evaluationContext.initialize(pendingRootBundles);
> @@ -385,7 +403,8 @@ final class ExecutorServiceParallelExecutor implements
> PipelineExecutor {
>LOG.debug("Executor Update: {}", update);
>if (update.getBundle().isPresent()) {
> if (ExecutorState.ACTIVE == startingState
> -|| (ExecutorState.PROCESSING == startingState &&
> noWorkOutstanding)) {
> +|| (ExecutorState.PROCESSING == startingState
> +&& noWorkOutstanding)) {
>scheduleConsumers(update);
> } else {
>allUpdates.offer(update);
>
> http://git-wip-us.apache.org/repos/asf/incubator-beam/blob/41fb16f0/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java
> --
> diff --git
> a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java
>
> b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java
> index 90db040..57d5628 100644
> ---
> a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java
> +++
> b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/FlattenEvaluatorFactory.java
> @@ -17,15 +17,12 @@
> */
> package org.apache.beam.runners.direct;
>
> -import java.util.Collection;
> -import java.util.Collections;
> import org.apache.beam.runners.direct.DirectRunner.CommittedBundle;
> import org.apache.beam.runners.direct.DirectRunner.UncommittedBundle;
> import org.apache.beam.sdk.transforms.AppliedPTransform;
> import org.apache.beam.sdk.transforms.Flatten;
> import org.apache.beam.sdk.transforms.Flatten.FlattenPCollectionList;
> import org.apache.beam.sdk.transforms.PTransform;
> -import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
> import org.apache.beam.sdk.util.WindowedValue;
> import org.apache.beam.sdk.values.PCollection;
> import org.apache.beam.sdk.values.PCollectionList;
> @@ -34,26 +31,13 @@ import org.apache.beam.sdk.values.PCollectionList;
> * The {@link DirectRunner} {@link TransformEvaluatorFactory} for the {@link
> Flatten}
> * {@link PTransform}.
> */
> -class FlattenEvaluatorFactory implements RootTransformEvaluatorFactory {
> +class FlattenEvaluatorFactory implements TransformEvaluatorFactory {
>priva
[message truncated...]
--
Daniel Kulp
dk...@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com
Re: [REMINDER] Technical discussion on the mailing list
too, as > well as almost everybody else. > > My best answer is -- we should realize that we are an extremely large and > complex project, with >100 contributors and >20 people working on it full > time. Nobody can follow every SDK, every runner, every IO connector, every > pull request, every comment that all contributors make each and every day. > > While nobody can follow everything, everything is being followed by > multiple people. And, we need to be accountable to each other to surface > everything relevant to the dev@ list. And I believe that is already > happening the vast majority of time. This is just one example where it > didn't happen. > > -- > > All that said, there are certainly areas for improvement. If anyone has > specific ideas, please reach out! I'd love to discuss them in detail and > propose improvements to the wider community. > > Thanks! > > > On Wed, Oct 5, 2016 at 6:16 PM, Thomas Weise <t...@apache.org> wrote: >> >> How about sending just the notifications for creating new JIRA and opening >> PR to dev@ so that those that are interested can start watching? >> >> Thanks, >> Thomas >> >> On Wed, Oct 5, 2016 at 5:33 PM, Dan Halperin <dhalp...@google.com.invalid> >> wrote: >> >>> On Wed, Oct 5, 2016 at 5:13 PM, Daniel Kulp <dk...@apache.org> wrote: >>> >>>> I just want to give a little more context to this…. I’ve been > lurking on >>>> this list for several months now reading everything that’s going on. >>> From >>>> Apache’s standpoint, that should be a “very good start” for getting to >>> know >>>> what is happening in a project. >>>> >>>> On my last PR, Eugene commented about using the AutoValue pattern for >>> part >>>> of it which caught me off guard. None of the other IO’s in master > were >>>> using it, there wasn’t any discussion on this list about it, I had no >>> idea >>>> what it was about. So I asked JB to make sure I hadn’t missed > anything. >>>> >>> >>>> Anyway, this is one of the main concerns I have with Beam’s PR work > flow, >>>> I feel I’m missing things as there is significant amount of things not >>>> happening on a list. The initial pull request is going to the > commits >>>> list (ok, would prefer the dev list, but at least its on a list). >>> However, >>>> none of the comments or discussions or anything that is occurring as > part >>>> of the review is making it to any list. The only people that “learn” >>> from >>>> the reviews are the reviewers and the person who initiated the PR > unless >>>> they go into each and every PR and read the comments (and find the > news >>>> ones and such).With my Apache hat on, this bothers me. >>> >>> >>> Anyway, I don’t really understand why the full github integration wasn’t >>>> setup for the beam PR’s so that the comments would come back to the > lists >>>> as well (and JIRA, BTW). >>>> >>> >>> This part confuses me. We've been told that discussions on JIRA, even >>> though they are emailed to the mailing lists, don't count as happening > on >>> the mailing list. So why would github integration be helpful vs just > more >>> spam? >>> >>> As another example, the comments on PR1003 are very applicable to anyone >>>> looking into writing IO’s and they could learn about some of the “best >>>> practices” presented there. >>> >>> >>> As Beam grows during its incubation, we are moving a lot of knowledge to >>> documentation, but yes -- right now, most of the I/O related practices > live >>> in Eugene's and my head (and now, JB's!). We're working on it, and hope > to >>> dramatically improve documentation for source authors in the next > quarter. >>> >>> For AutoValue specifically, this is by no means codified and it is >>> DEFINITELY not mandatory. Eugene and JB just experimented with it in the >>> last few days and decided it was useful in a few cases. We do (or did, >>> before this thread) need to have an actual discussion on the mailing > list >>> before moving forward further towards making it policy. >>> >>> Right now Ben Chambers is trying to apply AutoValue in places that need >>> templated types and struggling with multiple ?s, so the discussion may > need >>> to continue! ... >>> >>
Re: [REMINDER] Technical discussion on the mailing list
> On Oct 6, 2016, at 2:33 AM, Dan Halperin <dhalp...@google.com.INVALID> wrote: > > Anyway, I don’t really understand why the full github integration wasn’t >> setup for the beam PR’s so that the comments would come back to the lists >> as well (and JIRA, BTW). >> > > This part confuses me. We've been told that discussions on JIRA, even > though they are emailed to the mailing lists, don't count as happening on > the mailing list. So why would github integration be helpful vs just more > spam? Not sure who told you that… With JIRA, you can actually do a reply-all and respond to comments via email and your comments would make it back to the JIRA. Thus, the discussion can be done “on the list” yet still be in jira. That kind of depends on how the JIRA project is setup though. -- Daniel Kulp dk...@apache.org - http://dankulp.com/blog Talend Community Coder - http://coders.talend.com
Re: [REMINDER] Technical discussion on the mailing list
I just want to give a little more context to this…. I’ve been lurking on this list for several months now reading everything that’s going on. From Apache’s standpoint, that should be a “very good start” for getting to know what is happening in a project. On my last PR, Eugene commented about using the AutoValue pattern for part of it which caught me off guard. None of the other IO’s in master were using it, there wasn’t any discussion on this list about it, I had no idea what it was about. So I asked JB to make sure I hadn’t missed anything. Anyway, this is one of the main concerns I have with Beam’s PR work flow, I feel I’m missing things as there is significant amount of things not happening on a list. The initial pull request is going to the commits list (ok, would prefer the dev list, but at least its on a list). However, none of the comments or discussions or anything that is occurring as part of the review is making it to any list. The only people that “learn” from the reviews are the reviewers and the person who initiated the PR unless they go into each and every PR and read the comments (and find the news ones and such).With my Apache hat on, this bothers me.As another example, the comments on PR1003 are very applicable to anyone looking into writing IO’s and they could learn about some of the “best practices” presented there. Anyway, I don’t really understand why the full github integration wasn’t setup for the beam PR’s so that the comments would come back to the lists as well (and JIRA, BTW). That’s basically the background as to why JB sent this. I was confused and bugged him. :-) Dan > On Oct 5, 2016, at 1:51 PM, Jean-Baptiste Onofré <j...@nanthrax.net> wrote: > > Hi team, > > I would like to excuse myself to have forgotten to discuss and share with you > a technical point and generally speaking do a small reminder. > > When we work with Eugene on the JdbcIO, we experimented AutoValue to deal > with IO configuration. AutoValue provides a nice way to reduce and limit the > boilerplate code required by the IO configuration. > We used AutoValue in JdbcIO and, regarding the good improvements we saw, we > started to refactor the other IOs. > > The use of AutoValue should have been notice and discussed on the mailing > list. > > "If it doesn't exist on the mailing list, it doesn't exist at all." > > So, any comment happening on a GitHub pull request, or discussion on hangouts > which can impact the project (generally speaking) has to happen on the > mailing list. > > It provides project transparency and facilitates the new contribution > onboarding. > > Thanks ! > > Regards > JB > -- Daniel Kulp dk...@apache.org <mailto:dk...@apache.org> - http://dankulp.com/blog <http://dankulp.com/blog> Talend Community Coder - http://coders.talend.com <http://coders.talend.com/>
