Hi,I Just wanted to add some points on subjects I took part in:
On the Elasticsearch IO stand point, in addittion to ES 6 support, there was 
also 
- the addition of an exponential retry backoff on the write for all the ES 
versions.- meta-data support for all the ES
versions.
BestEtienne


Le lundi 10 septembre 2018 à 19:56 -0700, Rose Nguyen a écrit :
> September 2018 | Newsletter
> 
> 
> What’s been done
> 
> CI improvement (by: Etienne Chauchot)
> For each new commit on master Nexmark suite is run in both batch and 
> streaming mode in Spark, Flink, Cloud Dataflow
> (thanks to Andrew) and dashboards graphs are produced to track functional and 
> performance regressions.
> Elasticsearch IO Supports Version 6 (by: Dat Tran)
> Elasticsearch IO now supports version 6.x in addition to version 2.x and 
> 5.x.See the merged PR for more details.
> KuduIO Added (by: Tim Robertson)
> Apache Beam master now has KuduIO that will be released with Beam 2.7.0.See 
> BEAM-2661 for more details. 
> 
> 
> 
> What we’re working on...
> 
> Flink Portable Runner (by: Ankur Goenka, Maximilian Michels, Thomas Weise, 
> Ryan Williams)
> Support for streaming side inputs merged                                      
>                          Portable
> Compatibility Matrix tests pass in streaming mode                             
>                  Many more
> ValidatesRunner tests pass (ValidatesRunner is a comprehensive suite for Beam 
> test pipelines)Python Pipelines can be
> tested without bringing up a JobServer first (it is started in a container)   
>  Experimental support for executing
> the SDK harnesses in a process instead of a Docker container    Bug fixes to 
> Beam discovered during working on the
> portabilityState and Timer Support in Python SDK (by: Charles Chen, Robert 
> Bradshaw)
> This change adds the reference DirectRunner implementation of the Python User 
> State and Timers API. With this change,
> a user can execute DoFns with state and timers on the DirectRunner.See the 
> design doc and PR for more details.. 
> New IO - HadoopOutputFormatIO (by: Alexey Romanenko)
> Adding support of MapReduce OutputFormat.See BEAM-5310 for more details. 
> High-level Java 8 DSL (by: David Moravek, Vaclav Plajt, Marek Simunek)
> Adding high-level Java 8 DSL based on Euphoria API projectSee BEAM-3900 for 
> more details. Performance improvements for
> HDFS file writing operations (by: Tim Robertson)
> Autocreate directories when doing an HDFS renameSee PR for more details
> Recognition of non-code contributions (by: Gris Cuevas)
> Got consensus about recognizing non-code contributionsSee discussion for more 
> detailsPlanned launch date: Beam Summit
> London (October 2nd)
> Weekly Community Updates (by: Gris Cuevas)
> Some of the project’s subcomponents run weekly updates in the mailing list, 
> we’ll be consolidating best practices to
> share a weekly community update with all project related must knows in a shell
> 
> 
> 
> What’s planned
> 
> Beam Cookbook (by: Austin Bennett, David Cavazos, Gris Cuevas, Andrea 
> Foegler, Rose Nguyen, Connell O'Callaghan, and
> you!)
> We are creating a cookbook for common data science tasks in Beam and have 
> started brainstormingWe want to have a
> hackathon after the London Summit to generate content from the communityThere 
> will be a session at the summit to
> gather more ideas and input. Watch the dev and users mailing list for a call 
> for contributions soon!.
> Beam 2.7.0 release (by: Charles Chen)
> 
> Beam Mascot (by: Gris Cuevas & Community!)
> We got approval to launch a contest to create a new Apache Beam mascotSee 
> discussion for more details, if you’re
> interested in driving this, reach out in the thread!Planned launch date: Last 
> week of September
> 
> 
> 
> New Members
> 
> New Contributors
>  Đạt Trần, Ho Chi Minh City, VietnamSee BEAM-5107 for more details on 
> “Support ES-6.x for ElasticsearchIO” Ravi
> Pathak, Copenhagen, DenmarkUsing Beam for indexing open data on species at 
> GBIF.orgImproving robustness of SolrIO
> New Committers
> Tim Robertson, Copenhagen, Denmark
> 
> 
> 
> Events, Talks & Meetups
> 
> [Coming Up] Beam Summit @ London, England
> Organized by: Matthias Baetens, Victor Kotai, Alex Van Boxel & Gris CuevasThe 
> Beam Summit London 2018 will take place
> on October 1 and 2 in London. If you’re interested in speaking reach out to 
> gris@apache.orgMore info can be found in
> the blog post and you can get your tickets on Eventbrite
> [Coming Up] ApacheCon @ Montréal, Canada
> Will take place Sep 24-27 Etienne Chauchot will give a talk on Universal 
> Metrics with BeamAlexey Romanenko and Ismaël
> Mejía will give a talk on Building portable and evolvable data-intensive 
> applications with ApacheIsmaël Mejía and
> Eugene Kirpichov will give a talk on Robust, performant and modular APIs for 
> data ingestion with Apache BeamGris
> Cuevas will host a Birds of a Feather session on 9/26: Design Thinking to 
> manage online communities in Open Source
> Projects… It’ll be a Beam get together, we’ll have food & Swag, join us! 
> [Coming Up] DataEngConf @ Barcelona, Spain
> Will take place Sep 25-26Maximilian Michels will give an introduction to Beam 
> and its portability features.
> [Occurred] OSCON @ Portland, OR, USA (by: Holden Karau & Gris Cuevas)
> Holden Karau gave a talk on TFT/TFMA + Beam on Flink (and other related 
> adventures). Watch the video here and see the
> slides hereGris Cuevas gave a talk about active inclusion in Open Source, 
> slides here
> [Occurred] Open Challenge @ Guadalajara, Mexico (by: OSoM, IBM & Google)
> Arianne Navarro, Hector Paredes, Pablo Estrada & Gris Cuevas hosted a 
> Hackathon for Apache Beam and BlueXolo, results
> include 3PR for Beam and 8 Software Engineers introduced to Apache Beam
> [Occurred] Open Source Summit @ Vancouver, Canada
> Gris Cuevas gave a talk on active diversification in Open Source, slides 
> hereIsmael Mejia gave a talk on Apache Beam,
> see details here
> [Occurred] Flink Forward @ Berlin, Germany
> Robert Bradshaw and Maximilian Michels gave talk on Universal Machine 
> Learning with Apache Beam, schedule,
> slidesAljoscha Krettek and Thomas Weise Python Streaming Pipelines with Beam 
> on Flink, schedule, slides 
> 
> 
> 
> Resources
> 
> Setting up a Java Development Env Beam on GCP (by: Jacob Ferriero)This post 
> will help you get a development
> environment up and running to start developing Java Dataflow jobs. By the end 
> you’ll be able to run an Apache Beam
> locally in debug mode, execute code in a REPL to speed your development 
> cycles, and submit your job to Google Cloud
> Dataflow. Medium Post. Coding Apache Beam in your Web Browser (by: Daniel De 
> Leo)But what happens when you’re on the
> go on a computer which doesn’t support your IDE of choice, or you’re using 
> someone else’s computer and need to develop
> Apache Beam pipelines? Google has you covered! Google’s Cloud Shell comes 
> with a built-in Code Editor for
> developing/modifying code (it’s based on Eclipse’s Orion). It’s not as full 
> featured as an IDE but it does beat using
> Vim or Emacs to edit code! Medium Post. Building a real time quant trading 
> engine on Dataflow and Beam (by: Lei He)
> In this post, we are going to build a data pipeline that analyzes real time 
> stock tick data streamed from gCloud
> Pub/Sub, runs them through a pair correlation trading algorithm, and outputs 
> trading signals onto Pub/Sub for
> execution. Medium Post. Apache Beam: Reading from S3 and writing to BigQuery 
> (by: Asa Harland)
> In this article we look at how we can use Apache Beam to extract data from 
> AWS S3 (or Google Cloud Storage), run some
> aggregations over the data and store the result in BigQuery. Medium Post. 
> Apache Beam Events & Meetups
> Join our Slack channel! 
> 
> Until Next Time!
> 
> This edition was curated by our community of contributors, committers and 
> PMCs. It contains work done in August 2018
> and ongoing efforts. We hope to provide visibility to what's going on in the 
> community, so if you have questions, feel
> free to ask in this thread. 
> -- 
> Rose Thị Nguyễn
> 

Reply via email to