I'm generally in favor of greater decoupling / decentralization where possible. It is easy to imagine a world in which Beam consists of a few fairly autonomous projects, some centralized policy, opt-in infrastructure, just like ~all non-tiny software projects/foundations/companies.
The granularity matters quite a lot. But in my experience if you choose subprojects to match clear self-sustaining subcommunities then each one moves faster and has a stronger community than when you try to erase those distinctions in a giant monoproject. Starting with an active project that already exists and has users and maintainers works perfectly for getting the granularity right. I'm for trying it if/when the opportunity arises. There are technical challenges around shared infrastructure and global quality assurance. It would be an opportunity to make them concrete and address them. Kenn On Fri, Jun 22, 2018 at 10:24 AM Rafael Fernandez <[email protected]> wrote: > I like the idea of per-module releases for Beam. I know Henning and others > have thought about that space as well. > > BTW, I'm a big fan of scio, and happy to help in any way possible if they > are interested in turning "de facto" into "de jure" :D > > In such a world, Scio could be a very good first use case to drive the > mechanisms to enable per-module releases. I think it allows us to scale > better and sets a healthy path for special interest groups to naturally > emerge and collaborate with their own scope in mind. > > > On Thu, Jun 21, 2018 at 11:48 PM [email protected] < > [email protected]> wrote: > >> >> >> On 2018/06/21 17:17:36, Reuven Lax <[email protected]> wrote: >> > In that case things have changed since I talked to Neville about it last >> > November. >> > >> > On Thu, Jun 21, 2018 at 10:16 AM Rafal Wojdyla <[email protected]> wrote: >> > >> > > Nope - it uses standard runners and is fully Beam compliant. >> > > >> > > On Thu, Jun 21, 2018 at 1:12 PM, Reuven Lax <[email protected]> wrote: >> > > >> > >> My understanding was that under the covers it used the low-level >> Dataflow >> > >> service API to run the evaluations. >> > >> >> > >> On Thu, Jun 21, 2018 at 10:10 AM Rafal Wojdyla <[email protected]> >> wrote: >> > >> >> > >>> Hi. >> > >>> Reuven - sorry to hijack the thread - regarding REPL - what do you >> mean >> > >>> by it being very Dataflow specific? >> > >>> >> > >>> On Thu, Jun 21, 2018 at 12:04 PM, Jean-Baptiste Onofré < >> [email protected]> >> > >>> wrote: >> > >>> >> > >>>> As the code is not hosted at Apache as Beam, I would not consider >> SCIO >> > >>>> as the "official" Scala DSL. >> > >>>> >> > >>>> However, I agree that it's "de facto" Scala DSL for Beam ;) >> > >>>> >> > >>>> Just "wording" ;) >> > >>>> >> > >>>> Regards >> > >>>> JB >> > >>>> >> > >>>> On 21/06/2018 18:00, Robert Bradshaw wrote: >> > >>>> > I might go so far as to say Scio *is* the official Scala API for >> Beam. >> > >>>> > We point to it on our website, and have no plans to create >> another. It >> > >>>> > just happens to not be maintained and released by us. >> > >>>> > On Thu, Jun 21, 2018 at 7:37 AM Jean-Baptiste Onofré < >> [email protected]> >> > >>>> wrote: >> > >>>> >> >> > >>>> >> Hi Alistair, >> > >>>> >> >> > >>>> >> we discussed several times in the past with SCIO guys >> (especially >> > >>>> >> Neville), but it seems there's no strong plan right now about a >> > >>>> donation >> > >>>> >> of SCIO in Beam. >> > >>>> >> I think one of the concern is the release cycle, but I think it >> makes >> > >>>> >> sense to think about a release per module in Beam. It would >> allow >> > >>>> use to >> > >>>> >> release DSLs, IOs/extensions independently. But that's another >> story >> > >>>> ;) >> > >>>> >> >> > >>>> >> Regards >> > >>>> >> JB >> > >>>> >> >> > >>>> >> On 21/06/2018 16:34, [email protected] wrote: >> > >>>> >>> Hi, >> > >>>> >>> >> > >>>> >>> Is there any plan to make scio an official scala API for beam? >> If >> > >>>> not, is there any plan to have a scala API? >> > >>>> >>> >> > >>>> >>> Thanks, >> > >>>> >>> Alistair >> > >>>> >>> >> > >>>> >> >> > >>>> >> -- >> > >>>> >> Jean-Baptiste Onofré >> > >>>> >> [email protected] >> > >>>> >> http://blog.nanthrax.net >> > >>>> >> Talend - http://www.talend.com >> > >>>> >> > >>>> -- >> > >>>> Jean-Baptiste Onofré >> > >>>> [email protected] >> > >>>> http://blog.nanthrax.net >> > >>>> Talend - http://www.talend.com >> > >>>> >> > >>> >> > >>> >> > > >> > Thanks for the responses everyone. I'm essentially looking for some >> reassurance scio is still going to be supported in the long term. We'd like >> to use it for a big project. >> >> It states in the scio repo readme that from v0.3.0 it depends on beam and >> not on dataflow. >> >> Thanks, >> Alistair >> >
