I'm a scio contributor, and I have a lot of experience with Scala. However, I would advise for NOT using Scala. There are several problems with maintaining Scala libraries:
- have to build different artifacts for each Scala version - artifacts have dependencies to Scala standard library - it becomes even a bigger problem with Scala 3 migration - Scala is a very complex language and requires a lot of discipline Because of these issues, you see how much time it takes to upgrade Spark to newer Scala version, or libraries from the Twitter ecosystem. Gleb On Sun, Oct 28, 2018 at 10:10 PM Kenneth Knowles <[email protected]> wrote: > Porting to Scio is not necessary. I expect you can use Scala main() + Java > SDK + Scala DPASF no problem. Then Scio users can use it also, of course, > and so can Java SDK users. > > Doesn't Scala compile to jars that are somewhat usable from Java? I've > only ever gone the other way, but I thought it was somewhat both ways. That > would mean Java main() + Java SDK + Scala DPASF is also viable. But there's > still the matter of supporting Scala in our build system & codebase. > Personally SGTM since I know Scala well but I wouldn't want to have code > that only a couple people are comfortable modifying*. From a whole-project > perspective I would yield to the broader Beam dev community on this issue. > > I took a quick look at your Scala and it does look like it would mostly > port to Java quite easily, just a bit more boilerplate. Do you know of > particular things that might be hard? > > Kenn > > *our Jenkins and Gradle groovy scripts are further from Java than Scala > and we seem to be doing OK-but-not-great as far as everyone feeling OK to > modify them > > > On Sat, Oct 27, 2018 at 1:43 AM David Morávek <[email protected]> > wrote: > >> Hello Alejandro, >> >> +1 for java implementation, even though this would probably require more >> effort from your side >> >> The main problem with Scio is that it lives outside beam code base and >> depends on specific version of Beam SDK. The sketching extension (and any >> other module in beam code base) on the other hand uses Beam SDK that is >> build from sources (current checkout), that Scio might not be compatible >> with. >> >> D. >> >> On Sat, Oct 27, 2018 at 8:26 AM Alejandro <[email protected]> wrote: >> >>> Hello, >>> >>> although not exactly your intentions, I am also looking to contribute to >>> Beam, but from a code perspective. >>> >>> I've been discussing with some beam members like Austin and lukasz >>> (CCed) on how to integrate https://github.com/elbaulp/DPASF into Beam. >>> >>> It seems the best place for this algorithms is >>> https://github.com/apache/beam/tree/master/sdks/java/extensions/sketching >>> , >>> but right now I lack the Beam knowledge that allow my to implement it. >>> So I am looking to someone who could help me start. Should I write >>> wrappers that interface my Scala code using >>> https://github.com/spotify/scio? or re implement all in Java? >>> >>> Cheers. >>> >>> On 10/26/2018 11:55 PM, Rose Nguyen wrote: >>> > I've heard of many people referring to the Medium posts related to Beam >>> > for step-by-step tutorials. >>> > >>> > https://medium.com/tag/apache-beam/latest >>> > >>> > On Thu, Oct 25, 2018 at 9:25 PM Austin Bennett >>> > <[email protected] <mailto:[email protected]>> >>> wrote: >>> > >>> > Hi Beam Devs and Users, >>> > >>> > Trying to get a sense from the community on the sorts of things we >>> > think would be useful to build the community (I am thinking not >>> from >>> > an angle of specific code/implementation/functionality, but from a >>> > user/usability -- I want to dive in and make real contributions >>> with >>> > the code, too, but know I also have the interest and skills to help >>> > with education and community aspects, hence my focus on this). >>> > >>> > I had previously suggested a sort of cookbook for focused and >>> > curated examples (code and explination) to help people get started, >>> > on-boarding, using Beam to aid getting up and running and >>> > accomplishing something worthwhile (and quickly), that seems one >>> way >>> > to help grow our user base (and maybe future dev base afterwards >>> > those users become enamored), which did get some positive feedback >>> > when first put out there. >>> > >>> > There are many other areas where featuring others sharing successes >>> > from having used Beam or little tips can be valuable, Pablo's >>> > Awesome Beam is one example of such a >>> > collection: https://github.com/pabloem/awesome-beam or even >>> > centralizing a general place to find any/all Beam >>> > blogs/shared-code/writeups/etc. >>> > >>> > Certainly there is a place for all sorts of contributions and >>> > resources. What do people on these lists think would be >>> > particularly useful? Trying to get a more focused sense of where >>> we >>> > think efforts might be best focused. >>> > >>> > Please share anything (even semi-)related!? >>> > >>> > Thanks, >>> > Austin >>> > >>> > >>> > P.S. I realize that those following this list are rather self >>> > selecting as well, so this might not be the best forum to figure >>> out >>> > what new/novice users need, but I would like to hear what everyone >>> > else here thinks could be useful. >>> > >>> > >>> > >>> > -- >>> > Rose Thị Nguyễn >>> >>> -- >>> elbauldelprogramador.com >>> >> -- Cheers, Gleb
