Agree with Kenn, granularity would be key for per-module Beam releases. We can easily think today of releasing separately IOs and extensions because they depend only on the public API of the SDK which has not changed much since 2.0.0. So scio will probably fit this category.
A different story would be for runners at least at this moment with all the ongoing work on portability, there could be important benefits on having fixes and refactors applied globally when multiple internal changes are happening. In the end separating modules for releases has its pros/cons. In the pro side it encourages the project to be more stable and robust, but adds the weight of guaranteeing that everything works together given the multiple combination of versions released which has of course a price (this can be a burden for some users if we consider that users should also deal with the other dependencies of their own). Today Beam users get this for free with the monolithic release approach. As usual with software trade-offs trade-offs. On Fri, Jun 22, 2018 at 7:53 PM Kenneth Knowles <[email protected]> wrote: > I'm generally in favor of greater decoupling / decentralization where > possible. It is easy to imagine a world in which Beam consists of a few > fairly autonomous projects, some centralized policy, opt-in infrastructure, > just like ~all non-tiny software projects/foundations/companies. > > The granularity matters quite a lot. But in my experience if you choose > subprojects to match clear self-sustaining subcommunities then each one > moves faster and has a stronger community than when you try to erase those > distinctions in a giant monoproject. > > Starting with an active project that already exists and has users and > maintainers works perfectly for getting the granularity right. I'm for > trying it if/when the opportunity arises. There are technical challenges > around shared infrastructure and global quality assurance. It would be an > opportunity to make them concrete and address them. > > Kenn > > > > On Fri, Jun 22, 2018 at 10:24 AM Rafael Fernandez <[email protected]> > wrote: > >> I like the idea of per-module releases for Beam. I know Henning and >> others have thought about that space as well. >> >> BTW, I'm a big fan of scio, and happy to help in any way possible if they >> are interested in turning "de facto" into "de jure" :D >> >> In such a world, Scio could be a very good first use case to drive the >> mechanisms to enable per-module releases. I think it allows us to scale >> better and sets a healthy path for special interest groups to naturally >> emerge and collaborate with their own scope in mind. >> >> >> On Thu, Jun 21, 2018 at 11:48 PM [email protected] < >> [email protected]> wrote: >> >>> >>> >>> On 2018/06/21 17:17:36, Reuven Lax <[email protected]> wrote: >>> > In that case things have changed since I talked to Neville about it >>> last >>> > November. >>> > >>> > On Thu, Jun 21, 2018 at 10:16 AM Rafal Wojdyla <[email protected]> >>> wrote: >>> > >>> > > Nope - it uses standard runners and is fully Beam compliant. >>> > > >>> > > On Thu, Jun 21, 2018 at 1:12 PM, Reuven Lax <[email protected]> >>> wrote: >>> > > >>> > >> My understanding was that under the covers it used the low-level >>> Dataflow >>> > >> service API to run the evaluations. >>> > >> >>> > >> On Thu, Jun 21, 2018 at 10:10 AM Rafal Wojdyla <[email protected]> >>> wrote: >>> > >> >>> > >>> Hi. >>> > >>> Reuven - sorry to hijack the thread - regarding REPL - what do you >>> mean >>> > >>> by it being very Dataflow specific? >>> > >>> >>> > >>> On Thu, Jun 21, 2018 at 12:04 PM, Jean-Baptiste Onofré < >>> [email protected]> >>> > >>> wrote: >>> > >>> >>> > >>>> As the code is not hosted at Apache as Beam, I would not consider >>> SCIO >>> > >>>> as the "official" Scala DSL. >>> > >>>> >>> > >>>> However, I agree that it's "de facto" Scala DSL for Beam ;) >>> > >>>> >>> > >>>> Just "wording" ;) >>> > >>>> >>> > >>>> Regards >>> > >>>> JB >>> > >>>> >>> > >>>> On 21/06/2018 18:00, Robert Bradshaw wrote: >>> > >>>> > I might go so far as to say Scio *is* the official Scala API >>> for Beam. >>> > >>>> > We point to it on our website, and have no plans to create >>> another. It >>> > >>>> > just happens to not be maintained and released by us. >>> > >>>> > On Thu, Jun 21, 2018 at 7:37 AM Jean-Baptiste Onofré < >>> [email protected]> >>> > >>>> wrote: >>> > >>>> >> >>> > >>>> >> Hi Alistair, >>> > >>>> >> >>> > >>>> >> we discussed several times in the past with SCIO guys >>> (especially >>> > >>>> >> Neville), but it seems there's no strong plan right now about a >>> > >>>> donation >>> > >>>> >> of SCIO in Beam. >>> > >>>> >> I think one of the concern is the release cycle, but I think >>> it makes >>> > >>>> >> sense to think about a release per module in Beam. It would >>> allow >>> > >>>> use to >>> > >>>> >> release DSLs, IOs/extensions independently. But that's another >>> story >>> > >>>> ;) >>> > >>>> >> >>> > >>>> >> Regards >>> > >>>> >> JB >>> > >>>> >> >>> > >>>> >> On 21/06/2018 16:34, [email protected] wrote: >>> > >>>> >>> Hi, >>> > >>>> >>> >>> > >>>> >>> Is there any plan to make scio an official scala API for >>> beam? If >>> > >>>> not, is there any plan to have a scala API? >>> > >>>> >>> >>> > >>>> >>> Thanks, >>> > >>>> >>> Alistair >>> > >>>> >>> >>> > >>>> >> >>> > >>>> >> -- >>> > >>>> >> Jean-Baptiste Onofré >>> > >>>> >> [email protected] >>> > >>>> >> http://blog.nanthrax.net >>> > >>>> >> Talend - http://www.talend.com >>> > >>>> >>> > >>>> -- >>> > >>>> Jean-Baptiste Onofré >>> > >>>> [email protected] >>> > >>>> http://blog.nanthrax.net >>> > >>>> Talend - http://www.talend.com >>> > >>>> >>> > >>> >>> > >>> >>> > > >>> > Thanks for the responses everyone. I'm essentially looking for some >>> reassurance scio is still going to be supported in the long term. We'd like >>> to use it for a big project. >>> >>> It states in the scio repo readme that from v0.3.0 it depends on beam >>> and not on dataflow. >>> >>> Thanks, >>> Alistair >>> >>
