Thanks Eugene for opening the poll (sorry if I didn't before I was quite busy in the last two days but expected to do it today).
On Fri, Dec 8, 2017 at 1:27 AM, Ahmet Altay <[email protected]> wrote: > > On Thu, Dec 7, 2017 at 3:51 PM, Eugene Kirpichov <[email protected]> > wrote: >> >> I've sent the poll >> https://lists.apache.org/thread.html/5bc2e184a24de9dbc8184ffd2720d1894010497d47d956b395e037df@%3Cuser.beam.apache.org%3E >> Will figure out how to tweet from @ApacheBeam, and sent the Twitter poll >> as well (or ask someone to). > > > I tweeted the poll. > >> >> >> On Wed, Dec 6, 2017 at 1:47 PM Lukasz Cwik <[email protected]> wrote: >>> >>> +1 on moving forward with the plan suggested by kirpichov@ >>> >>> On Wed, Dec 6, 2017 at 9:14 AM, Robert Bradshaw <[email protected]> >>> wrote: >>>> >>>> +1 to moving forward with this plan. >>>> >>>> (FWIW, this seems *less* backwards incompatible than, say, moving from >>>> Spark 1 to Spark 2, which was decided much quicker. I suppose the >>>> Spark change has a lower bound on the number of users it could impact >>>> though.) >>>> >>>> On Wed, Dec 6, 2017 at 9:09 AM, Eugene Kirpichov <[email protected]> >>>> wrote: >>>> > Okay, then let's go forward. Seems that we should: >>>> > - Open a new poll on user@, in light of 2.2 having been released >>>> > - Open a twitter poll >>>> > - Tweet that there's also a poll going on on user@ >>>> > - Runner authors will reach out to respective runner user communities >>>> > - 2 weeks later we gather results and decide >>>> > ? >>>> > >>>> > On Wed, Dec 6, 2017 at 6:16 AM Ismaël Mejía <[email protected]> wrote: >>>> >> >>>> >> +1 For Eugene’s arguments waiting for Beam 3.0 seems still far away, >>>> >> and starting to improve Beam to offer a Java 8 friendly experience >>>> >> seems like an excellent idea. >>>> >> >>>> >> I understand the backwards compatibility argument. We should do the >>>> >> poll in twitter + try to reach more users for comments. If you >>>> >> consider that it is worth, I can open a second poll at user@. >>>> >> >>>> >> In any case we should try to move forward, even if we have more than >>>> >> 5% of users who want to stay on Java 7 we can consider to maintain >>>> >> minor releases of a backwards compatible version where we can >>>> >> backport >>>> >> only critical fixes e.g. security/data related errors but nothing >>>> >> new, >>>> >> in case some user really needs to have them. Of course this can be >>>> >> some extra work (to be discussed). >>>> >> >>>> >> >>>> >> On Tue, Dec 5, 2017 at 7:24 AM, Jean-Baptiste Onofré >>>> >> <[email protected]> >>>> >> wrote: >>>> >> > +1, and sorry again, I thought we got an consensus. >>>> >> > >>>> >> > Regards >>>> >> > JB >>>> >> > >>>> >> > On 12/05/2017 07:10 AM, Kenneth Knowles wrote: >>>> >> >> >>>> >> >> +1 to the poll and also to Reuven's point. >>>> >> >> >>>> >> >> Those without a support contract would have been using JDK 7 >>>> >> >> without >>>> >> >> security updates for years. IMO it seems harmful, as a netizen, to >>>> >> >> encourage >>>> >> >> its use/existence. >>>> >> >> >>>> >> >> If there's no noise from the prior thread, then I would assume no >>>> >> >> one >>>> >> >> on >>>> >> >> user@ has any objection. Anyone else with customers should reach >>>> >> >> out to >>>> >> >> them. >>>> >> >> >>>> >> >> Kenn >>>> >> >> >>>> >> >> On Mon, Dec 4, 2017 at 9:49 PM, Reuven Lax <[email protected] >>>> >> >> <mailto:[email protected]>> wrote: >>>> >> >> >>>> >> >> Technically it's a backwards-incompatible change, however if >>>> >> >> we are >>>> >> >> convinced the risk is low we could do it. >>>> >> >> >>>> >> >> As mentioned on the original thread, it's not clear that all >>>> >> >> Beam >>>> >> >> users read >>>> >> >> user@ - e.g. most Dataflow users definitely do not. I think we >>>> >> >> need >>>> >> >> to >>>> >> >> separately reach out to users of each runner through >>>> >> >> runner-specific >>>> >> >> channels. >>>> >> >> >>>> >> >> Reuven >>>> >> >> >>>> >> >> On Mon, Dec 4, 2017 at 9:37 PM, Eugene Kirpichov >>>> >> >> <[email protected] >>>> >> >> <mailto:[email protected]>> wrote: >>>> >> >> >>>> >> >> On the original thread >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> <https://lists.apache.org/thread.html/2e1890c62d9f022f09b20e9f12f130fe9f1042e391979087f725d2e0@%3Cuser.beam.apache.org%3E> >>>> >> >> , >>>> >> >> Robert and Ismaël were in favor of no major version change >>>> >> >> [Ismaël >>>> >> >> said:/Also I am afraid that if we wait/ >>>> >> >> /until we have enough changes to switch Beam to a new >>>> >> >> major >>>> >> >> version the >>>> >> >> switch to Java 8 will happen too late, probably after Java >>>> >> >> 8's >>>> >> >> end >>>> >> >> of >>>> >> >> life. And I am not exaggerating, Java 8 is planned to EOL >>>> >> >> next >>>> >> >> march >>>> >> >> 2018!/]; JB and now Reuven are in favor of a major version >>>> >> >> change; >>>> >> >> >>>> >> >> nobody so far argued against switching to Java8 in >>>> >> >> general. >>>> >> >> >>>> >> >> I'm personally in favor of no major version change (i.e. >>>> >> >> not >>>> >> >> waiting >>>> >> >> until all other large changes for Beam 3.0 converge, which >>>> >> >> will >>>> >> >> likely >>>> >> >> be many months), because: >>>> >> >> - Reasons Ismaël cited; plus the reason that most people >>>> >> >> are >>>> >> >> likely >>>> >> >> already using Java 8. >>>> >> >> - Going Java8-only earlier will make other Beam 3.0 APIs >>>> >> >> better >>>> >> >> for >>>> >> >> Java8 users, because we (Beam contributors) will have >>>> >> >> experience >>>> >> >> working >>>> >> >> with them within the SDK in Java8 (e.g. writing tests with >>>> >> >> use >>>> >> >> of >>>> >> >> lambdas and noticing whether it's clunky, or whether some >>>> >> >> other >>>> >> >> Beam >>>> >> >> APIs need better Java8 support). >>>> >> >> - Going Java8 will make it more reasonable to include >>>> >> >> (mostly >>>> >> >> or >>>> >> >> only) >>>> >> >> Java8 snippets in Beam documentation, which will obviously >>>> >> >> look >>>> >> >> more >>>> >> >> concise and attractive, addressing one of the common >>>> >> >> concerns >>>> >> >> of >>>> >> >> Beam >>>> >> >> users that it has a heavyweight API compared to >>>> >> >> functional-style >>>> >> >> APIs of >>>> >> >> Spark etc. >>>> >> >> >>>> >> >> I think resolving this via a poll of users would be >>>> >> >> reasonable. >>>> >> >> I'd >>>> >> >> suggest e.g. the following phrasing: >>>> >> >> >>>> >> >> Apache Beam is considering dropping support for Java 7, >>>> >> >> and >>>> >> >> supporting >>>> >> >> only Java 8 and above in a subsequent release. How would >>>> >> >> it >>>> >> >> impact >>>> >> >> your >>>> >> >> usage of Beam? >>>> >> >> - I am already using only Java 8+ for building my Beam >>>> >> >> code >>>> >> >> - I am using Java 7 for building my Beam code, but I would >>>> >> >> have >>>> >> >> no >>>> >> >> trouble switching to Java 8 >>>> >> >> - I am using Java 7 for building my Beam code, and >>>> >> >> dropping >>>> >> >> Java 7 >>>> >> >> would >>>> >> >> be a blocker or hindrance to adopting the new release for >>>> >> >> me >>>> >> >> >>>> >> >> We could tweet this poll on Apache Beam twitter and >>>> >> >> publish on >>>> >> >> user@, >>>> >> >> and, say, if we receive 5% or fewer votes for option 3 >>>> >> >> after >>>> >> >> keeping it >>>> >> >> open for 2 weeks, then adopt Java 8 without a major >>>> >> >> version >>>> >> >> change. >>>> >> >> >>>> >> >> WDYT? >>>> >> >> >>>> >> >> On Mon, Dec 4, 2017 at 8:34 PM Jean-Baptiste Onofré >>>> >> >> <[email protected] >>>> >> >> <mailto:[email protected]>> wrote: >>>> >> >> >>>> >> >> Good idea ! Definitely +1 >>>> >> >> >>>> >> >> Regards >>>> >> >> JB >>>> >> >> >>>> >> >> On 12/05/2017 05:25 AM, Reuven Lax wrote: >>>> >> >> > We should bring this up on the Beam 3.0 thread. >>>> >> >> Since >>>> >> >> it's >>>> >> >> technically a >>>> >> >> > backwards-incompatible change, it might make a good >>>> >> >> item >>>> >> >> for Beam >>>> >> >> 3.0. >>>> >> >> > >>>> >> >> > Reuven >>>> >> >> > >>>> >> >> > On Mon, Dec 4, 2017 at 8:20 PM, Jean-Baptiste >>>> >> >> Onofré >>>> >> >> <[email protected] <mailto:[email protected]> >>>> >> >> > <mailto:[email protected] <mailto:[email protected]>>> >>>> >> >> wrote: >>>> >> >> > >>>> >> >> > My apologizes, I thought we had a consensus >>>> >> >> already. >>>> >> >> > >>>> >> >> > Regards >>>> >> >> > JB >>>> >> >> > >>>> >> >> > On 12/04/2017 11:22 PM, Eugene Kirpichov wrote: >>>> >> >> > >>>> >> >> > Thanks JB for sending the detailed notes >>>> >> >> about >>>> >> >> new >>>> >> >> stuff >>>> >> >> in 2.2.0! A lot >>>> >> >> > of exciting things indeed. >>>> >> >> > >>>> >> >> > Regarding Java 8: I thought our consensus >>>> >> >> was to >>>> >> >> have the >>>> >> >> release notes >>>> >> >> > say that we're *considering* going >>>> >> >> Java8-only, >>>> >> >> and >>>> >> >> use >>>> >> >> that to get more >>>> >> >> > opinions from the user community - but I >>>> >> >> can't >>>> >> >> find >>>> >> >> the >>>> >> >> emails that made >>>> >> >> > me think so. >>>> >> >> > >>>> >> >> > +Ismaël Mejía <mailto:[email protected] >>>> >> >> <mailto:[email protected]> <mailto:[email protected] >>>> >> >> <mailto:[email protected]>>> - do >>>> >> >> > you think we should formally conclude the >>>> >> >> vote >>>> >> >> on >>>> >> >> the >>>> >> >> thread [VOTE] >>>> >> >> > [DISCUSSION] Remove support for Java 7? >>>> >> >> > Or should we take more steps - e.g. perhaps >>>> >> >> tweet a >>>> >> >> link >>>> >> >> to that thread >>>> >> >> > from the Beam twitter account, ask people >>>> >> >> to >>>> >> >> chime >>>> >> >> in, >>>> >> >> and wait for say >>>> >> >> > 2 weeks before declaring a conclusion? >>>> >> >> > >>>> >> >> > Let's also have a process JIRA for going >>>> >> >> Java8. >>>> >> >> I've >>>> >> >> filed one: >>>> >> >> > https://issues.apache.org/jira/browse/BEAM-3285 >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285> >>>> >> >> > >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285 >>>> >> >> <https://issues.apache.org/jira/browse/BEAM-3285>> >>>> >> >> > >>>> >> >> > On Mon, Dec 4, 2017 at 1:58 AM >>>> >> >> Jean-Baptiste >>>> >> >> Onofré >>>> >> >> <[email protected] <mailto:[email protected]> >>>> >> >> > <mailto:[email protected] >>>> >> >> <mailto:[email protected]>> >>>> >> >> <mailto:[email protected] <mailto:[email protected]> >>>> >> >> > <mailto:[email protected] >>>> >> >> <mailto:[email protected]>>>> >>>> >> >> wrote: >>>> >> >> > >>>> >> >> > Just an important note that we forgot >>>> >> >> to >>>> >> >> mention. >>>> >> >> > >>>> >> >> > !! The 2.2.0 release will be the last >>>> >> >> one >>>> >> >> supporting >>>> >> >> Spark 1.x and >>>> >> >> > Java 7 !! >>>> >> >> > >>>> >> >> > Starting from Beam 2.3.0, the Spark >>>> >> >> runner >>>> >> >> will work >>>> >> >> only with >>>> >> >> > Spark 2.x and we >>>> >> >> > will focus only Java 8. >>>> >> >> > >>>> >> >> > Regards >>>> >> >> > JB >>>> >> >> > >>>> >> >> > On 12/04/2017 10:15 AM, Jean-Baptiste >>>> >> >> Onofré >>>> >> >> wrote: >>>> >> >> > > Thanks Reuven ! >>>> >> >> > > >>>> >> >> > > I would like to emphasize on some >>>> >> >> highlights in >>>> >> >> 2.2.0 release: >>>> >> >> > > >>>> >> >> > > - New IOs have been introduced: >>>> >> >> > > * TikaIO leveraging Apache Tika, >>>> >> >> allowing >>>> >> >> the >>>> >> >> deal with a lot >>>> >> >> > of different >>>> >> >> > > data formats >>>> >> >> > > * RedisIO to read and write >>>> >> >> key/value >>>> >> >> pairs >>>> >> >> from a Redis >>>> >> >> > server. This >>>> >> >> > IO will >>>> >> >> > > be soon extended to Redis PubSub. >>>> >> >> > > * FileIO provides transforms for >>>> >> >> working >>>> >> >> with >>>> >> >> files (raw). >>>> >> >> > Especially, it >>>> >> >> > > provides matching file patterns and >>>> >> >> read >>>> >> >> on >>>> >> >> patterns. It can be >>>> >> >> > easily >>>> >> >> > extended >>>> >> >> > > for a specific format (like we do >>>> >> >> in >>>> >> >> AvroIO >>>> >> >> or >>>> >> >> TextIO now). >>>> >> >> > > * SolrIO to interact with Apache >>>> >> >> Solr >>>> >> >> (Lucene) >>>> >> >> > > >>>> >> >> > > - On the other hand, improvements >>>> >> >> have >>>> >> >> been >>>> >> >> performed on >>>> >> >> > existing IOs: >>>> >> >> > > * We started to introduce readAll >>>> >> >> pattern >>>> >> >> in >>>> >> >> IOs (AvroIO, >>>> >> >> > TextIO, JdbcIO, >>>> >> >> > > ...), allowing to pass "request" >>>> >> >> arguments >>>> >> >> via an >>>> >> >> input PCollection. >>>> >> >> > > * ElasticsearchIO has an improved >>>> >> >> support >>>> >> >> of >>>> >> >> different >>>> >> >> > Elasticsearch >>>> >> >> > version >>>> >> >> > > (including Elasticsearch 5.x). It >>>> >> >> also >>>> >> >> now >>>> >> >> supports SSL/TLS. >>>> >> >> > > * HBaseIO is now able to do >>>> >> >> dynamic >>>> >> >> work >>>> >> >> rebalancing >>>> >> >> > > * KinesisIO uses a more accurate >>>> >> >> watermark >>>> >> >> (based on >>>> >> >> > approximateArrivalTimestamp) >>>> >> >> > > * TextIO now supports custom >>>> >> >> delimiter >>>> >> >> and like >>>> >> >> AvroIO, >>>> >> >> > supports the >>>> >> >> > readAll >>>> >> >> > > pattern, >>>> >> >> > > * Performance improvements on >>>> >> >> JdbcIO >>>> >> >> when >>>> >> >> it >>>> >> >> has to read lot >>>> >> >> > of rows >>>> >> >> > > * Kafka write supports >>>> >> >> Exactly-Once >>>> >> >> pattern >>>> >> >> (introduce in >>>> >> >> > Kafka 0.11.x) >>>> >> >> > > >>>> >> >> > > - A new DSL has been introduced: >>>> >> >> the SQL >>>> >> >> DSL ! >>>> >> >> > > >>>> >> >> > > We are now focus on 2.3.0 release >>>> >> >> with >>>> >> >> new >>>> >> >> improvements and >>>> >> >> > features ! >>>> >> >> > > >>>> >> >> > > Stay tuned ! >>>> >> >> > > >>>> >> >> > > JB on behalf of the Apache Beam >>>> >> >> community. >>>> >> >> > > >>>> >> >> > > On 12/02/2017 11:40 PM, Reuven Lax >>>> >> >> wrote: >>>> >> >> > >> The Apache Beam community is >>>> >> >> pleased to >>>> >> >> announce the >>>> >> >> > availability of the >>>> >> >> > >> 2.2.0 release. >>>> >> >> > >> >>>> >> >> > >> This release adds support for >>>> >> >> generic >>>> >> >> file >>>> >> >> sources and sinks >>>> >> >> > (beyond TextIO >>>> >> >> > >> and AvroIO) using FileIO, >>>> >> >> including >>>> >> >> support for >>>> >> >> dynamic >>>> >> >> > filenames using >>>> >> >> > >> readAll; this allows streaming >>>> >> >> pipelines >>>> >> >> to now >>>> >> >> read from files by >>>> >> >> > >> continuously monitoring a >>>> >> >> directory for >>>> >> >> new >>>> >> >> filw. Many other >>>> >> >> > IOs are >>>> >> >> > improved, >>>> >> >> > >> notably including exactly-once >>>> >> >> support >>>> >> >> for >>>> >> >> the >>>> >> >> Kafka sink. Initial >>>> >> >> > support for >>>> >> >> > >> BEAM-SQL is also included in this >>>> >> >> release. >>>> >> >> For a >>>> >> >> more-complete >>>> >> >> > list of major >>>> >> >> > >> changes in the release, please >>>> >> >> refer to >>>> >> >> the >>>> >> >> release notes [2]. >>>> >> >> > >> >>>> >> >> > >> The 2.2.0 release is now the >>>> >> >> recommended >>>> >> >> version; we encourage >>>> >> >> > everyone to >>>> >> >> > >> upgrade from any earlier releases. >>>> >> >> > >> >>>> >> >> > >> We’d like to invite everyone to >>>> >> >> try out >>>> >> >> Apache >>>> >> >> Beam today and >>>> >> >> > consider >>>> >> >> > >> joining our vibrant community. We >>>> >> >> welcome >>>> >> >> feedback, >>>> >> >> > contribution and >>>> >> >> > >> participation through our mailing >>>> >> >> lists, >>>> >> >> issue >>>> >> >> tracker, pull >>>> >> >> > requests, and >>>> >> >> > >> events. >>>> >> >> > >> >>>> >> >> > >> - Reuven Lax, on behalf of the >>>> >> >> Apache >>>> >> >> Beam >>>> >> >> community. >>>> >> >> > >> >>>> >> >> > >> [1] >>>> >> >> https://beam.apache.org/get-started/downloads/ >>>> >> >> <https://beam.apache.org/get-started/downloads/> >>>> >> >> > >>>> >> >> <https://beam.apache.org/get-started/downloads/ >>>> >> >> <https://beam.apache.org/get-started/downloads/>> >>>> >> >> > >> [2] >>>> >> >> > >> >>>> >> >> > >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044> >>>> >> >> > >>>> >> >> >>>> >> >> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044 >>>> >> >> >>>> >> >> >>>> >> >> <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12341044>> >>>> >> >> > >> >>>> >> >> > > >>>> >> >> > >>>> >> >> > -- >>>> >> >> > Jean-Baptiste Onofré >>>> >> >> > [email protected] <mailto:[email protected]> >>>> >> >> <mailto:[email protected] >>>> >> >> <mailto:[email protected]>> >>>> >> >> > <mailto:[email protected] >>>> >> >> <mailto:[email protected]> >>>> >> >> <mailto:[email protected] >>>> >> >> <mailto:[email protected]>>> >>>> >> >> > http://blog.nanthrax.net >>>> >> >> > Talend - http://www.talend.com >>>> >> >> > >>>> >> >> > >>>> >> >> > -- >>>> >> >> > Jean-Baptiste Onofré >>>> >> >> > [email protected] <mailto:[email protected]> >>>> >> >> <mailto:[email protected] >>>> >> >> <mailto:[email protected]>> >>>> >> >> > http://blog.nanthrax.net >>>> >> >> > Talend - http://www.talend.com >>>> >> >> > >>>> >> >> > >>>> >> >> >>>> >> >> -- >>>> >> >> Jean-Baptiste Onofré >>>> >> >> [email protected] <mailto:[email protected]> >>>> >> >> http://blog.nanthrax.net >>>> >> >> Talend - http://www.talend.com >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> > >>>> >> > -- >>>> >> > Jean-Baptiste Onofré >>>> >> > [email protected] >>>> >> > http://blog.nanthrax.net >>>> >> > Talend - http://www.talend.com >>> >>> >
