Apache Beam London meetup 9: recordings

[Matthias Baetens]

Hi all

The recordings from last year's Apache Beam meetup London are now available
on the YouTube channel (apologies for the delay):
- Apache Beam meetup 9: BBC's journey with Apache Beam

- Apache Beam meetup 9: Apache Beam + Apache Druid

- Apache Beam meetup 9: Attack detection use case at Fastly


Enjoy!
Matthias

Apache Beam meetup London 21st of September

[Matthias Baetens]

Hi everyone!

Join us for the next Apache Beam meetup in London, happening in a bit over
2 weeks on the 21st of September. It will be in-person (but to keep things
COVID-safe, with a limited attendance). More details:
https://www.meetup.com/London-Apache-Beam-Meetup/events/280442419

Looking forward to seeing you there :)
Matthias

Re: October 2020, Beam Community Update

[Matthias Baetens]

This is really awesome indeed, Brittany :)

If I could only read Alexey's mind, but my hunch is that this could come in
blogpost format on the website (as well) and sharing it on the Twitter
handle?

On Tue, 3 Nov 2020 at 20:54, Pablo Estrada  wrote:

> Hi Alexey,
> Do you have any other place in mind? I don't think Brittany has current
> plans to publish this elsewhere, but if you have any good ideas, I imagine
> she could consider them : )
> Best
> -P.
>
> On Tue, Nov 3, 2020 at 8:23 AM Alexey Romanenko 
> wrote:
>
>> Thanks for doing this!
>>
>> Is it going to be published somewhere else than only the mailing lists?
>>
>> On 31 Oct 2020, at 00:03, Brittany Hermann  wrote:
>>
>> Hi everyone,
>>
>> Attached is the October 2020 Beam Community Update. The purpose of this
>> newsletter is purely community focused - giving folks the visibility to the
>> insights of the health and growth of the community. This newsletter will
>> not cover project releases or feature updates. Please let me know if you
>> have any questions and feedback is welcome!
>>
>>
>> https://docs.google.com/document/d/1_t6xKoOQVwgn2edmRVh1ViudmbnNM3BwZyguKAwwjfA/edit?usp=sharing
>>
>>
>>
>>
>>
>>

New Dates For Beam Summit Digital 2020

[Matthias Baetens]

Dear Beam community,

A few weeks ago, we announced the dates for the Beam Digital Summit and we
know the community received this news with excitement. This is a great
opportunity to create and share content about streaming analytics and the
solutions that teams around the world have created using Apache Beam and
its ecosystem.

We have chosen August 24-28th as the new dates. While this has been a
difficult decision, we think it’s the right decision to ensure we produce
the best possible event. We encourage you to send your talk proposals,
anything from use cases, lightning talks, or workshop ideas.

Based on this change, the CFP will remain open until June 15th. We would
love to hear about what you are doing with Beam, how to improve it, and how
to strengthen our community.

We thank you for your understanding! See you soon!

-Griselda Cuevas, Brittany Hermann, Maximilian Michels, Austin Bennett,
Matthias Baetens, Alex Van Boxel

Register now for the Beam Summit 2020!

[Matthias Baetens]

Hi all,

We are happy to share some more information on the *Beam Summit 2020* that
will take place on the *world wide web* this time around :)

🎫 You can now *register on Crowdcast *,
where the conference will take place.
💸 Want to help to make the Beam Summit even more awesome? Consider sharing
the *sponsoring prospectus*
 with
your company!
📢 As Austin shared previously - please submit your proposal *here*
.

Read about this and more in the latest *Beam Summit blogpost
 *&
consider sharing with your colleagues, friends and on social channels!

Cheers,
Matthias, on behalf of the Beam Summit Steering Committee

Re: Beam Katas YouTube

[Matthias Baetens]

Hey Henry - I think you can just fill it out rather than comment. The
people who have access to the Twitter handle monitor the sheet and will
tweet accordingly afaik :)

On Sun, 5 Apr 2020 at 14:42, Henry Suryawirawan 
wrote:

> I just added some comments on the sheet.
> Can someone help me to review and publish the tweet?
>
>
>
> On Thu, Apr 2, 2020 at 12:32 AM Matthias Baetens <
> baetensmatth...@gmail.com> wrote:
>
>> I see :-) That's a good idea, iirc the best way to do that is through
>> this
>> <https://docs.google.com/spreadsheets/d/1mz36njTtn1UJwDF50GdqyZVbX_F0n_A6eMYcxsktpSM/edit>
>> sheet!
>>
>> On Wed, 1 Apr 2020 at 16:01, Henry Suryawirawan 
>> wrote:
>>
>>> Thanks all!
>>>
>>> Matthias, the raw file is not owned by me and managed by the Level Up
>>> YouTube channel team.
>>> Maybe we can do a tweet from @ApacheBeam?
>>>
>>>
>>>
>>>
>>> On Wed, Apr 1, 2020 at 2:27 PM Matthias Baetens <
>>> baetensmatth...@gmail.com> wrote:
>>>
>>>> This is brilliant, Henry! Let me know if you want this posted on the
>>>> Beam YouTube channel as well in case you have access to the raw file.
>>>>
>>>> Cheers,
>>>> Matthias
>>>>
>>>> On Fri, 27 Mar 2020 at 20:02, Pablo Estrada  wrote:
>>>>
>>>>> Nice : D
>>>>>
>>>>> On Fri, Mar 27, 2020 at 12:44 AM Alex Van Boxel 
>>>>> wrote:
>>>>>
>>>>>> That's nicely done! Congrats, going to share this immediately.
>>>>>>
>>>>>> And I actually didn't know where the name Beam came from, now I know
>>>>>> :-)
>>>>>>
>>>>>>  _/
>>>>>> _/ Alex Van Boxel
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 27, 2020 at 4:32 AM Henry Suryawirawan <
>>>>>> hsuryawira...@google.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Just would like to share that recently the Apache Beam Katas is
>>>>>>> featured in the Google Cloud Level Up YouTube video (
>>>>>>> https://www.youtube.com/watch?v=tzZiC4s_OqE).
>>>>>>> Hope that it can be useful for the community.
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Henry
>>>>>>>
>>>>>>>
>>>>>>>

Re: Beam Katas YouTube

[Matthias Baetens]

I see :-) That's a good idea, iirc the best way to do that is through this
<https://docs.google.com/spreadsheets/d/1mz36njTtn1UJwDF50GdqyZVbX_F0n_A6eMYcxsktpSM/edit>
sheet!

On Wed, 1 Apr 2020 at 16:01, Henry Suryawirawan 
wrote:

> Thanks all!
>
> Matthias, the raw file is not owned by me and managed by the Level Up
> YouTube channel team.
> Maybe we can do a tweet from @ApacheBeam?
>
>
>
>
> On Wed, Apr 1, 2020 at 2:27 PM Matthias Baetens 
> wrote:
>
>> This is brilliant, Henry! Let me know if you want this posted on the Beam
>> YouTube channel as well in case you have access to the raw file.
>>
>> Cheers,
>> Matthias
>>
>> On Fri, 27 Mar 2020 at 20:02, Pablo Estrada  wrote:
>>
>>> Nice : D
>>>
>>> On Fri, Mar 27, 2020 at 12:44 AM Alex Van Boxel 
>>> wrote:
>>>
>>>> That's nicely done! Congrats, going to share this immediately.
>>>>
>>>> And I actually didn't know where the name Beam came from, now I know :-)
>>>>
>>>>  _/
>>>> _/ Alex Van Boxel
>>>>
>>>>
>>>> On Fri, Mar 27, 2020 at 4:32 AM Henry Suryawirawan <
>>>> hsuryawira...@google.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Just would like to share that recently the Apache Beam Katas is
>>>>> featured in the Google Cloud Level Up YouTube video (
>>>>> https://www.youtube.com/watch?v=tzZiC4s_OqE).
>>>>> Hope that it can be useful for the community.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Henry
>>>>>
>>>>>
>>>>>

Re: Beam Katas YouTube

[Matthias Baetens]

This is brilliant, Henry! Let me know if you want this posted on the Beam
YouTube channel as well in case you have access to the raw file.

Cheers,
Matthias

On Fri, 27 Mar 2020 at 20:02, Pablo Estrada  wrote:

> Nice : D
>
> On Fri, Mar 27, 2020 at 12:44 AM Alex Van Boxel  wrote:
>
>> That's nicely done! Congrats, going to share this immediately.
>>
>> And I actually didn't know where the name Beam came from, now I know :-)
>>
>>  _/
>> _/ Alex Van Boxel
>>
>>
>> On Fri, Mar 27, 2020 at 4:32 AM Henry Suryawirawan <
>> hsuryawira...@google.com> wrote:
>>
>>> Hello,
>>>
>>> Just would like to share that recently the Apache Beam Katas is featured
>>> in the Google Cloud Level Up YouTube video (
>>> https://www.youtube.com/watch?v=tzZiC4s_OqE).
>>> Hope that it can be useful for the community.
>>>
>>>
>>> Regards,
>>> Henry
>>>
>>>
>>>

Beam Summit North America 2019 - recordings

[Matthias Baetens]

Hi everyone,

It's our pleasure to share the recordings from the Beam Summit North
America 2019.
Please find them in the YouTube playlist

on the Apache Beam channel
.

Best regards,
Matthias

Re: Feedback on how we use Apache Beam in my company

[Matthias Baetens]

This is great, Pierre! Thank you for sharing, very interesting.

Would you and your team be interested to talk about your use case at a
meetup (or Summit) in the future? :)

All the best,
Matthias

On Wed, 9 Oct 2019 at 15:59, Etienne Chauchot  wrote:

> Very nice !
>
> Thanks
>
> ccing dev list
>
> Etienne
> On 09/10/2019 16:55, Pierre Vanacker wrote:
>
> Hi Apache Beam community,
>
>
>
> We’ve been working with Apache Beam in production for a few years now in
> my company (Dailymotion).
>
>
>
> If you’re interested to know how we use Apache Beam in combination with
> Google Dataflow, we shared this experience in the following article :
> https://medium.com/dailymotion/realtime-data-processing-with-apache-beam-and-google-dataflow-at-dailymotion-7d1b994dc816
>
>
>
> Thanks to the developers for your great work !
>
>
>
> Regards,
>
>
>
> Pierre
>
>

Re: Beam meetups: Paris (01/07/2019) and Stockholm (25/07/2019)

[Matthias Baetens]

Thanks for the kind words!

The slides from the Beam meetup Paris are available now as well:
https://drive.google.com/drive/folders/148xkUUFPIkk_UEsShnyd2UuViEVg6-TU?usp=drive_open
 :)

On Wed, 10 Jul 2019 at 00:05, Aizhamal Nurmamat kyzy 
wrote:

> Thank you Matthias for organizing it!
>
> On Mon, Jul 8, 2019 at 8:30 AM Alexey Romanenko 
> wrote:
>
>> Matthias, thank you for organizing the first Beam meetup in Paris!
>> It was very cool even if it was quite hot outside that day in the city,
>> thanks to the weather =)
>>
>> On 26 Jun 2019, at 08:15, Matthias Baetens 
>> wrote:
>>
>> Hi everyone!
>>
>> Happy to announce 2 upcoming Beam Summits:
>> One in *Paris *on *01/07/2019 *with JB talking about Beam IOs,
>> Konstantin Knauf talking about Beam on Flink and Pablo Estrada talking
>> about SQL in Beam!
>> More info and RSVP:
>> https://www.meetup.com/Paris-Apache-Beam-Meetup/events/261775884/
>>
>> On *25/07/2019 *we will have another meetup in Stockholm, with Robert
>> Bradshaw talking about portability, Matt Casters about visual pipeline
>> development using Kettle and Theofilos Kakantousis talking about ML with
>> Beam.
>> More info and RSVP:
>> https://www.meetup.com/Apache-Beam-Stockholm/events/262263586/
>>
>> We have a meetup coming up in London as well, so stay tuned ;-)
>>
>> Cheers,
>> Matthias
>>
>>
>>

Beam meetups: Paris (01/07/2019) and Stockholm (25/07/2019)

[Matthias Baetens]

Hi everyone!

Happy to announce 2 upcoming Beam Summits:
One in *Paris *on *01/07/2019 *with JB talking about Beam IOs, Konstantin
Knauf talking about Beam on Flink and Pablo Estrada talking about SQL in
Beam!
More info and RSVP:
https://www.meetup.com/Paris-Apache-Beam-Meetup/events/261775884/

On *25/07/2019 *we will have another meetup in Stockholm, with Robert
Bradshaw talking about portability, Matt Casters about visual pipeline
development using Kettle and Theofilos Kakantousis talking about ML with
Beam.
More info and RSVP:
https://www.meetup.com/Apache-Beam-Stockholm/events/262263586/

We have a meetup coming up in London as well, so stay tuned ;-)

Cheers,
Matthias

Beam Summit volunteering team

[Matthias Baetens]

Hi everyone,

As you might know, the Beam Summit is currently organised by a small team
 of people committing their free time to make
this happen. We are looking to make this group larger in the future and
particularly could use some help on website maintenance, development and
design.

This is a call for all volunteers interested, please ping over your details
and what you would like to do - this can go for meetups as well.

Many thanks,
Matthias on behalf of the Beam Summit organising team

Beam Summit Europe: speakers and schedule online!

[Matthias Baetens]

Hi everyone,

Happy to share that the speakers  and
schedule  are now online on the website.

Make sure you register on Eventbrite
 if you want to attend and
follow out Twitter channel  for
announcements regarding the speakers over the next few weeks!

Best regards,
Matthias

Meetups in London, Stockholm + recordings

[Matthias Baetens]

Hi everyone,

Happy to announce 2 meetups for the next week(s):
- *London *is having its second meetup of the year on *01/05/2019 in
Level39, Canary Wharf
*
: https://www.meetup.com/London-Apache-Beam-Meetup/events/260526077 with
talks from Lyft, Datatonic and Google.
- *Stockholm* is having its second meetup in its lifetime on *06/05/2019 at
Spotify
:
*
https://www.meetup.com/Apache-Beam-Stockholm/events/260634514 with talks
from Spotify, EQT and Google.

The recording of the 6th meetup in London at Revolut is up on YouTube
 now.

Last but not least: start following the Beam Summit handle on Twitter
 for the latest information on the Beam
Summit  in Berlin (19-20 June 2019) and get your
ticket on Eventbrite
!

Best regards,
Matthias

Beam meetup London at Revolut 03/04/2019

[Matthias Baetens]

Hi everyone,

On Wednesday 03/04, we will have the next Beam meetup in London at the
Revolut offices.

Robert Bradshaw will do a deep dive on portability, while Revolut will
present its use-case on using Beam for fraud detection.

More info and RSVP on the meetup page https://www.meetup.com/London-Apache-Beam-Meetup/events/259496271>.

Looking forward to seeing you there!
Best regards,
Matthias

Re: [Announcement] New Website for Beam Summits

[Matthias Baetens]

Awesome Aizhamal! Great work and thanks for your continued efforts on this
:) Looking forward to the summit.

On Mon, 18 Mar 2019 at 23:17, Aizhamal Nurmamat kyzy 
wrote:

> Hello everybody!
>
> We are thrilled to announce the launch of beamsummit.org dedicated to
> Beam Summits!
>
> The current version of the website provides information about the upcoming
> Beam Summit in Europe on June 19-20th, 2019. We will update it for the
> upcoming summits in Asia and North America accordingly. You can access all
> necessary information about the conference theme, speakers and sessions,
> the abstract submission timeline and the registration process, the
> conference venues and much more that you will find useful until and during
> the Beam Summits 2019.
>
> We are working to make the website easy to use, so that anyone who is
> organizing a Beam event can rely on it. You can find the code for it in
> Github .
>
> The pages will be updated on a regular basis, but we also love hearing
> thoughts from our community! Let us know if you have any questions,
> comments or suggestions, and help us improve. Also, if you are thinking of
> organizing a Beam event, please feel free to reach out
>  for support, and to use the code in GitHub as
> well.
>
> We sincerely hope that you like the new Beam Summit website and will find
> it useful for accessing information. Enjoy browsing around!
>
> Thanks,
> Aizhamal
>

Beam Summit Europe 2019: CfP

[Matthias Baetens]

Hi everyone,

As you might already know, the *Beam Summit Europe 2019* will take place in
*Berlin* this year on *19-20 June*!

Of course, we would love to have you there. That is why we are opening
the *Call
for Speakers .*

We are looking for people to share use-cases for Beam, do a technical deep
dive or deliver a workshop on Beam. We have a few standard slots, but don't
hesitate to do a proposal in case you have something different in mind.

Stay tuned for more announcements - we are working hard on the website,
which should be up and running soon. And of course, don't hesitate to reach
out if you have any questions or suggestions!

Looking forward to seeing you in person later this year.

Best regards,
The Beam Summit organising team

Re: Beam Python streaming pipeline on Flink Runner

[Matthias Baetens]

Hey Ankur,

Thanks for the swift reply. Should I change this in the capability matrix
<https://s.apache.org/apache-beam-portability-support-table> then?

Many thanks.
Best,
Matthias

On Thu, 31 Jan 2019 at 09:31, Ankur Goenka  wrote:

> Hi Matthias,
>
> Unfortunately, unbounded reads including pubsub are not yet supported for
> portable runners.
>
> Thanks,
> Ankur
>
> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <
> baetensmatth...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Last few days I have been trying to run a streaming pipeline (code on
>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>>
>> I am running a Flink cluster locally (v1.5.6
>> <https://flink.apache.org/downloads.html>)
>> I have built the SDK Harness Container: *./gradlew
>> :beam-sdks-python-container:docker*
>> and started the JobServer: *./gradlew
>> :beam-runners-flink_2.11-job-server:runShadow
>> -PflinkMasterUrl=localhost:8081.*
>>
>> I run my pipeline with: *env/bin/python streaming_pipeline.py
>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
>> --input_subscription xxx --output_subscription xxx*
>>
>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>>
>> The job submits fine, but unfortunately fails after a few seconds with
>> the error attached.
>>
>> Anything I am missing or doing wrong?
>>
>> Many thanks.
>> Best,
>> Matthias
>>
>>
>>

Beam Python streaming pipeline on Flink Runner

[Matthias Baetens]

Hi everyone,

Last few days I have been trying to run a streaming pipeline (code on Github
) on a Flink Runner.

I am running a Flink cluster locally (v1.5.6
)
I have built the SDK Harness Container: *./gradlew
:beam-sdks-python-container:docker*
and started the JobServer: *./gradlew
:beam-runners-flink_2.11-job-server:runShadow
-PflinkMasterUrl=localhost:8081.*

I run my pipeline with: *env/bin/python streaming_pipeline.py
--runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
--input_subscription xxx --output_subscription xxx*

All this is running inside a Ubuntu (Bionic) in a Virtualbox.

The job submits fine, but unfortunately fails after a few seconds with the
error attached.

Anything I am missing or doing wrong?

Many thanks.
Best,
Matthias
TimerException{java.lang.RuntimeException: Failed to finish remote bundle}
at 
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$RepeatedTriggerTask.run(SystemProcessingTimeService.java:335)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to finish remote bundle
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.ExecutableStageDoFnOperator$SdkHarnessDoFnRunner.finishBundle(ExecutableStageDoFnOperator.java:624)
at 
org.apache.beam.runners.flink.metrics.DoFnRunnerWithMetricsUpdate.finishBundle(DoFnRunnerWithMetricsUpdate.java:87)
at 
org.apache.beam.runners.core.SimplePushbackSideInputDoFnRunner.finishBundle(SimplePushbackSideInputDoFnRunner.java:118)
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.invokeFinishBundle(DoFnOperator.java:679)
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.checkInvokeFinishBundleByTime(DoFnOperator.java:673)
at 
org.apache.beam.runners.flink.translation.wrappers.streaming.DoFnOperator.lambda$open$1(DoFnOperator.java:378)
at 
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$RepeatedTriggerTask.run(SystemProcessingTimeService.java:330)
... 7 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: 
Error received from SDK harness for instruction 5: Traceback (most recent call 
last):
  File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 148, in _execute
response = task()
  File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 183, in 
self._execute(lambda: worker.do_instruction(work), work)
  File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 256, in do_instruction
request.instruction_id)
  File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 272, in process_bundle
bundle_processor.process_bundle(instruction_id)
  File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 489, in process_bundle
].process_encoded(data.data)
  File 
"/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 126, in process_encoded
self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py", line 182, in 
apache_beam.runners.worker.operations.Operation.output
def output(self, windowed_value, output_index=0):
  File "apache_beam/runners/worker/operations.py", line 183, in 
apache_beam.runners.worker.operations.Operation.output
cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value)
  File "apache_beam/runners/worker/operations.py", line 89, in 
apache_beam.runners.worker.operations.ConsumerSet.receive
cython.cast(Operation, consumer).process(windowed_value)
  File "apache_beam/runners/worker/operations.py", line 497, in 
apache_beam.runners.worker.operations.DoOperation.process
with self.scoped_process_state:
  File "apache_beam/runners/worker/operations.py", line 498, in 
apache_beam.runners.worker.operations.DoOperation.process
self.dofn_receiver.receive(o)
  File "apache_beam/runners/common.py", line 678, in 
apache_beam.runners.common.DoFnRunner.receive
self.process(windowed_value)
  File "apache_beam/runners/common.py", line 684, in 
apache_beam.runners.common.DoFnRunner.process
self._rera

Re: Beam Summits!

[Matthias Baetens]

@Reza Rokni : I was planning to reach out to you for the
Asian edition of the Summit - do you think Singapore would make sense?

On Mon, 7 Jan 2019 at 07:25, Matthias Baetens 
wrote:

> Hey Austin,
>
> Yes, it would be good to get going soon. We started approx. 3 months
> before the event like Thomas mentioned. It is important though to kick-off
> two things at the same time:
> 1) Drafting and getting the proposal approved
> 2) Kicking off the practical organisation
>
> If we would have treated 1) as a blocker for 2), we would've gotten in
> trouble. I was not very involved in the proposal discussions, Gris and
> Ahmet did a great deal of work there and I think they'd be happy to help
> out with any questions you might have; but the proposal would definitely be
> a good basis to start from.
>
> With 2) I have some more info I could share.
> We started by assembling a core team of people we knew were going to do
> the bulk of the organising work (ideally a mix of 'community' forces, a PMC
> member, and employees from companies who seem very keen to join forces)
> Then there is the actual organisational part (which you can divide and
> conquer with the team):
>
>- Fixing the date(s) of your conference (this needs to go into the
>proposal as well, so one of the first things to decide
>- Estimating the budget and finding sponsoring companies: this will
>decide what is possible and what is not
>- Finding a venue that is convenient and fits your needs (how many
>people are you expecting? we could maybe do a small poll on the mailing
>lists and Twitter to see conference interest for this year). Ideally, your
>venue is flexible for change as well (extendable rooms, ...). This goes
>hand in hand with:
>   - Catering / food / coffee
>   - Video crew for recording the sessions
>- Opening a CfP: this is something you probably want to do as soon as
>possible. It will decide largely the content of the summit and what
>sessions you will be able to offer to your audience. If will also be
>crucial in communication later on.
>- This goes hand in hand with constructing your agenda. Draft what you
>   are aiming for and share this with everyone (speakers included) so they 
> can
>   help you build the schedule. This will change a lot (up until the day of
>   the event if people cancel)
>   - Start collecting the speaker presentations
>- Open attendee registrations as soon as possible: this will allow you
>to estimate the number of attendees
>   - A small website could be useful to have an overview of the
>   speakers and schedule, combined with registrating functionality (e.g.
>   https://devfest.gdg.london or use https://splashthat.com)
>- Starting the buzz (communications wise): think about mailing list,
>social media (Twitter, LinkedIn mostly), blogposts, local communities that
>might be interested, reach out to people that frequent your meetup, ask
>sponsoring companies to use their channels, ...
>   - Also think about marketing of the summit (slide branding, banners
>   for social media, ...)
>- Gather a team of people to help out on the day itself and list out
>the on-the-day(s) tasks to divide the work. Have clear responsibilities for
>everyone and have someone to keep the overview. Learning from our summit:
>have someone dedicated to take care of the speakers (welcome them, make
>sure they are comfortable for the talk, make sure they have everything they
>need. A seperate room for speakers is a luxury, but a good one to have).
>- Other things like:
>   - Swag
>   - Speaker gifts
>   - Photographer
>   - Social media branding (ask speakers to use hashtags, ...)
>   - Have a debrief questionnaire
>
> Overall, try to keep closely in sync with your organising team
> (fortnightly calls or weekly when getting closer to the event). A shared
> tracker for you, team (and community - they can give feedback), might be
> useful as well (find ours here
> <https://docs.google.com/spreadsheets/d/1pNoeLD0JBImc1-gja209mSuTVLNna_b1gmQ1r-_Z5t0/edit>
> )
>
> I will share some additional materials later with you. Happy to have a
> quick call to answer additional questions and help out where needed :)
> Best,
> Matthias
>
>
>
>
> On Sun, 6 Jan 2019 at 22:55, Thomas Weise  wrote:
>
>> For the event in SF in April it would be necessary to get going soon.
>>
>> For the type of information that will be required, you could take a look
>> at the proposal from the London event [1].  It was created with ~ 3 months
>> lead time. Note the draft mistakenly used "Apache" i

Re: Beam Summits!

[Matthias Baetens]

n Bennett <
> whatwouldausti...@gmail.com> wrote:
>
>> Hi Matthias, etc,
>>
>> Trying to get thoughts on formalizing a process for getting proposals
>> together.  I look forward to the potential day that there are many people
>> that want (rather than just willing) to host a summit in a given region in
>> a given year.  Perhaps too forward looking.
>>
>> Also, you mentioned planning London wound up with a tight time window.
>> If shooting for April in SF, seems  the clock might be starting to tick.
>> Any advice for how much time needed?  And guidance on getting whatever
>> formal needed through Apache - and does this also necessarily involve a
>> Beam PMC or community vote (probably more related to the first paragraph)?
>>
>> Thanks,
>> Austin
>>
>> On Thu, Dec 20, 2018, 1:09 AM Matthias Baetens > wrote:
>>
>>> Great stuff, thanks for the overview, Austin.
>>>
>>> For EU, there are things to say for both Stockholm and Berlin, but I
>>> think it makes sense to do it on the back of another conference (larger
>>> chance of people being in town with the same interest). I like Thomas
>>> comment - we will attract more people from the US if we don't let it
>>> conflict with the big events there. +1 for doing it around the time of
>>> Berlin Buzzwords.
>>>
>>> For Asia, I'd imagine Singapore would be an option as well. I'll reach
>>> out to some people that are based there to get a grasp on the size of the
>>> community there.
>>>
>>> Best,
>>> -M
>>>
>>>
>>>
>>> On Thu, 20 Dec 2018 at 05:08, Thomas Weise  wrote:
>>>
>>>> I think for EU there is a proposal to have it next to Berlin Buzzwords
>>>> in June. That would provide better spacing and avoid conflict with
>>>> ApacheCon.
>>>>
>>>> Thomas
>>>>
>>>>
>>>> On Wed, Dec 19, 2018 at 3:09 PM Suneel Marthi 
>>>> wrote:
>>>>
>>>>> How about Beam Summit in Berlin on Sep 6 immediately following Flink
>>>>> Forward Berlin on the previous 2 days.
>>>>>
>>>>> Same may be for Asia also following Flink Forward Asia where and
>>>>> whenever it happens.
>>>>>
>>>>> On Wed, Dec 19, 2018 at 6:06 PM Austin Bennett <
>>>>> whatwouldausti...@gmail.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I really enjoyed Beam Summit in London (Thanks Matthias!), and there
>>>>>> was much enthusiasm for continuations.  We had selected that location in 
>>>>>> a
>>>>>> large part due to the growing community there, and we have users in a
>>>>>> variety of locations.  In our 2019 calendar,
>>>>>> https://docs.google.com/spreadsheets/d/1CloF63FOKSPM6YIuu8eExjhX6xrIiOp5j4zPbSg3Apo/
>>>>>> shared in the past weeks, 3 Summits are tentatively slotted for this 
>>>>>> year.
>>>>>> Wanting to start running this by the group to get input.
>>>>>>
>>>>>> * Beam Summit NA, in San Francisco, approx 3 April 2019 (following
>>>>>> Flink Forward).  I can organize.
>>>>>> * Beam Summit Europe, in Stockholm, this was the runner up in voting
>>>>>> falling behind London.  Or perhaps Berlin?  October-ish 2019
>>>>>> * Beam Summit Asia, in Tokyo ??
>>>>>>
>>>>>> What are general thoughts on locations/dates?
>>>>>>
>>>>>> Looking forward to convening in person soon.
>>>>>>
>>>>>> Cheers,
>>>>>> Austin
>>>>>>
>>>>>

Re: Beam Summits!

[Matthias Baetens]

Great stuff, thanks for the overview, Austin.

For EU, there are things to say for both Stockholm and Berlin, but I think
it makes sense to do it on the back of another conference (larger chance of
people being in town with the same interest). I like Thomas comment - we
will attract more people from the US if we don't let it conflict with the
big events there. +1 for doing it around the time of Berlin Buzzwords.

For Asia, I'd imagine Singapore would be an option as well. I'll reach out
to some people that are based there to get a grasp on the size of the
community there.

Best,
-M



On Thu, 20 Dec 2018 at 05:08, Thomas Weise  wrote:

> I think for EU there is a proposal to have it next to Berlin Buzzwords in
> June. That would provide better spacing and avoid conflict with ApacheCon.
>
> Thomas
>
>
> On Wed, Dec 19, 2018 at 3:09 PM Suneel Marthi  wrote:
>
>> How about Beam Summit in Berlin on Sep 6 immediately following Flink
>> Forward Berlin on the previous 2 days.
>>
>> Same may be for Asia also following Flink Forward Asia where and whenever
>> it happens.
>>
>> On Wed, Dec 19, 2018 at 6:06 PM Austin Bennett <
>> whatwouldausti...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I really enjoyed Beam Summit in London (Thanks Matthias!), and there was
>>> much enthusiasm for continuations.  We had selected that location in a
>>> large part due to the growing community there, and we have users in a
>>> variety of locations.  In our 2019 calendar,
>>> https://docs.google.com/spreadsheets/d/1CloF63FOKSPM6YIuu8eExjhX6xrIiOp5j4zPbSg3Apo/
>>> shared in the past weeks, 3 Summits are tentatively slotted for this year.
>>> Wanting to start running this by the group to get input.
>>>
>>> * Beam Summit NA, in San Francisco, approx 3 April 2019 (following Flink
>>> Forward).  I can organize.
>>> * Beam Summit Europe, in Stockholm, this was the runner up in voting
>>> falling behind London.  Or perhaps Berlin?  October-ish 2019
>>> * Beam Summit Asia, in Tokyo ??
>>>
>>> What are general thoughts on locations/dates?
>>>
>>> Looking forward to convening in person soon.
>>>
>>> Cheers,
>>> Austin
>>>
>>

Re: 2019 Beam Events

[Matthias Baetens]

Great stuff, Gris! Looking forward to what 2019 will bring!

The Beam meetup in London will have a new get together early next year as
well :-)
https://www.meetup.com/London-Apache-Beam-Meetup/


On Tue, 4 Dec 2018 at 23:50 Austin Bennett 
wrote:

> Already got that process kicked off with the NY and LA meet ups, now that
> SF is about to be inagurated goal will be to get these moving as well.
>
> For anyone that is in (or goes to) those areas:
> https://www.meetup.com/New-York-Apache-Beam/
> https://www.meetup.com/Los-Angeles-Apache-Beam/
>
> Please reach out to get involved!
>
>
>
> On Tue, Dec 4, 2018 at 3:13 PM Griselda Cuevas  wrote:
>
>> +1 to Pablo's suggestion, if there's interest in "Founding a Meetup group
>> in a particular city, let's create the Meetup page and start getting sign
>> ups. Joana will be reaching out with a comprenhexive list of how to get
>> started and we're hoping to compile a high level calendar of
>> launches/announcements to feed into your meetup.
>>
>> G
>>
>> On Tue, 4 Dec 2018 at 12:04, Daniel Salerno 
>> wrote:
>>
>>> =)
>>> What good news!
>>> Okay, I'll set up the group and try to get interested.
>>> Thank you
>>>
>>>
>>> Em ter, 4 de dez de 2018 às 17:19, Pablo Estrada 
>>> escreveu:
>>>
 FWIW, for some of these places that have interest (e.g. Brazil,
 Israel), it's possible to create a group in meetup.com, and start
 gauging interest, and looking for organizers.
 Once a group of people with interest exists, it's easier to get
 interest / sponsorship to bring speakers.
 So if you are willing to create the group in meetup, Daniel, we can
 monitor it and try to plan something as it grows : )
 Best
 -P.

 On Tue, Dec 4, 2018 at 10:55 AM Daniel Salerno 
 wrote:

>
> It's a shame that there are no events in Brazil ...
>
> =(
>
> Em ter, 4 de dez de 2018 às 13:12, OrielResearch Eila Arich-Landkof <
> e...@orielresearch.org> escreveu:
>
>> agree 👍
>>
>> On Tue, Dec 4, 2018 at 5:41 AM Chaim Turkel  wrote:
>>
>>> Israel would be nice to have one
>>> chaim
>>> On Tue, Dec 4, 2018 at 12:33 AM Griselda Cuevas 
>>> wrote:
>>> >
>>> > Hi Beam Community,
>>> >
>>> > I started curating industry conferences, meetups and events that
>>> are relevant for Beam, this initial list I came up with. I'd love your 
>>> help
>>> adding others that I might have overlooked. Once we're satisfied with 
>>> the
>>> list, let's re-share so we can coordinate proposal submissions, 
>>> attendance
>>> and community meetups there.
>>> >
>>> >
>>> > Cheers,
>>> >
>>> > G
>>> >
>>> >
>>> >
>>>
>>> --
>>>
>>>
>>> Loans are funded by
>>> FinWise Bank, a Utah-chartered bank located in Sandy,
>>> Utah, member FDIC, Equal
>>> Opportunity Lender. Merchant Cash Advances are
>>> made by Behalf. For more
>>> information on ECOA, click here
>>> . For important information
>>> about
>>> opening a new
>>> account, review Patriot Act procedures here
>>> .
>>> Visit Legal
>>>  to
>>> review our comprehensive program terms,
>>> conditions, and disclosures.
>>>
>>
>>
>> --
>> Eila
>> www.orielresearch.org
>> https://www.meetu
>> p.co
>> 
>> m/Deep-Learning-In-Production/
>> 
>>
>>
>> --

Re: Slides from OSCON

[Matthias Baetens]

I added this (with minor delay) to the "Conference Recordings"-playlist on
the Beam channel. Thanks Holden!

On Thu, 6 Sep 2018 at 02:31 Holden Karau  wrote:

> Actually I found the speaker release, looks like I can upload it to
> YouTube - https://youtu.be/ZGyx4GuGEj4
>
> On Tue, Sep 4, 2018 at 10:42 AM Holden Karau  wrote:
>
>> I think I have permission to share it on YouTube, I'll ask the O'Reilly
>> folks when I see them @ Strata next week :)
>>
>> On Tue, Sep 4, 2018 at 10:35 AM Pablo Estrada  wrote:
>>
>>> I found this video:
>>> https://www.safaribooksonline.com/videos/oscon-2018/9781492026075/9781492026075-video321550
>>> Though it requires a suscription to Safari books.
>>> Best
>>> -P.
>>>
>>> On Tue, Sep 4, 2018 at 8:21 AM Matthias Baetens <
>>> baetensmatth...@gmail.com> wrote:
>>>
>>>> Hey Holden, just checking if you were able to find something? :)
>>>> -M
>>>>
>>>>
>>>> On Sat, Aug 25, 2018, 12:07 Holden Karau  wrote:
>>>>
>>>>> Good question, I’ll take a look tonight :)
>>>>>
>>>>> On Sat, Aug 25, 2018 at 6:32 AM Matthias Baetens <
>>>>> baetensmatth...@gmail.com> wrote:
>>>>>
>>>>>> Hey Holden,
>>>>>> Have you found the video in the meanwhile? Happy to upload it to the
>>>>>> Beam channel :)
>>>>>> Cheers,
>>>>>> M
>>>>>>
>>>>>> On Wed, 25 Jul 2018 at 02:52 Holden Karau 
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Friends,
>>>>>>>
>>>>>>> If you missed my talk @ OSCON on TFT/TFMA + Beam on Flink (and other
>>>>>>> related adventures), I've uploaded my slides to
>>>>>>> https://www.slideshare.net/hkarau/powering-tensorflow-with-big-data-using-apache-beam-flink-and-spark-oscon-pdx-2018
>>>>>>> and once the video is done I'll try and find a way to share that as 
>>>>>>> well.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Holden :)
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Twitter: https://twitter.com/holdenkarau
>>>>>>>
>>>>>> --
>>>>>>
>>>>>>
>>>>> --
>>>>> Twitter: https://twitter.com/holdenkarau
>>>>> Books (Learning Spark, High Performance Spark, etc.):
>>>>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>>>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>>>
>>>> --
>>>>
>>>>
>>>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
--

Re: Bay Area Apache Beam Kickoff!

[Matthias Baetens]

Awesome effort awesome - good luck on the first meetup! :)

On Tue, 20 Nov 2018 at 01:37 Austin Bennett 
wrote:

> We have our first meetup scheduled for December 12th in San Francisco.
>
> Andrew Pilloud, a software engineer at Google and Beam committer, will
> demo the latest feature in Beam SQL: a standalone SQL shell. The talk cover
> why SQL is a good fit for streaming data processing, the technical details
> of the Beam SQL engine, and a peek into our future plans.
>
> Kenn Knowles, a founding PMC Member and incoming PMC Chair for the Apache
> Beam project, as well as computer scientist and engineer at Google will
> share about all things Beam. Where it is, where its been, where its going.
>
> More info:
> https://www.meetup.com/San-Francisco-Apache-Beam/events/256348972/
>
> For those in/around town (or that can be) come join in the fun!
>
>
>
>
> --

Re: [Call for items] November Beam Newsletter

[Matthias Baetens]

Looks great, thanks for the effort and for including the Summit blogpost,
Rose!

On Thu, 8 Nov 2018 at 22:55 Rose Nguyen  wrote:

> Hi Beamers:
>
> Time to sync with the community on all the awesome stuff we've been doing!
>
> *Add the highlights from October to now (or planned events and talks) that
> you want to share by 11/14 11:59 p.m. PDT.*
>
> We will collect the notes via Google docs but send out the final version
> directly to the user mailing list. If you do not know how to format
> something, it is OK to just put down the info and I will edit. I'll ship
> out the newsletter on 11/15.
>
> [1]
> https://docs.google.com/document/d/1kKQ4a9RdptB6NwYlqmI9tTcdLAUzDnWi2dkvUi0J_Ww
> --
> Rose Thị Nguyễn
>
--

Recordings and presentations from Beam Summit London 2018

[Matthias Baetens]

Hi everyone,

Very happy to be sharing the great materials from the speakers and
contributors at the Beam Summit we had in London last month in the *new
blogpost* on the website:
https://beam.apache.org/blog/2018/10/31/beam-summit-aftermath.html

You can find the full list of presentations and recordings of the sessions
there, or you can head over to our YouTube channel
.

Thanks for everyone who helped, contributed, was a speaker or sponsor at
the event.

Best,
Matthias
--

Re: Slides from OSCON

[Matthias Baetens]

Hey Holden, just checking if you were able to find something? :)
-M

On Sat, Aug 25, 2018, 12:07 Holden Karau  wrote:

> Good question, I’ll take a look tonight :)
>
> On Sat, Aug 25, 2018 at 6:32 AM Matthias Baetens <
> baetensmatth...@gmail.com> wrote:
>
>> Hey Holden,
>> Have you found the video in the meanwhile? Happy to upload it to the Beam
>> channel :)
>> Cheers,
>> M
>>
>> On Wed, 25 Jul 2018 at 02:52 Holden Karau  wrote:
>>
>>> Hi Friends,
>>>
>>> If you missed my talk @ OSCON on TFT/TFMA + Beam on Flink (and other
>>> related adventures), I've uploaded my slides to
>>> https://www.slideshare.net/hkarau/powering-tensorflow-with-big-data-using-apache-beam-flink-and-spark-oscon-pdx-2018
>>> and once the video is done I'll try and find a way to share that as well.
>>>
>>> Cheers,
>>>
>>> Holden :)
>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>>
>> --
>>
>>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  <https://amzn.to/2MaRAG9>
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>
--

Re: Beam Summit London 2018

[Matthias Baetens]

Hey everyone!

A kind reminder that we will be closing the CfP
<https://goo.gl/forms/nrZOCC1JwEfLtKfA2> and sponsoring opportunities
<https://drive.google.com/file/d/1RnZ52rGaB6BR-EKneBcabdMcg9Pl7z9M/view?usp=sharing>
at
the end of this week.
Don't hesitate to reach out for questions regarding either of them.

Registrations are still open; check out the Eventbrite page
<https://www.eventbrite.com/e/beam-summit-london-2018-tickets-49100625292>.

Best,
Matthias

On Fri, 24 Aug 2018 at 15:29 Pascal Gula  wrote:

> Hi Mahesh,
> I didn't mention any support for Beam python SDK on Spark or Hadoop since
> it doesn't exist ATM.
> I just mentioned them since they are currently used as successful solution
> in terms of data processing framework.
> The use-case I will highlight will focus on DirectRunner for development
> and DataFlowRunner for production.
> Best regards,
> Pascal
>

> On Fri, Aug 24, 2018 at 4:20 PM, Mahesh Vangala 
> wrote:
>
>> Hi Pascal -
>>
>> Just curious to know that in your message you mentioned about beam python
>> work with respect to spark/hadoop.
>> I was under the impression that python beam is not yet available to run
>> on spark/hadoop.
>> Am I missing something?
>> Let me know.
>> Thanks.
>>
>> *--*
>> *Mahesh Vangala*
>> *(Ph) 443-326-1957 <(443)%20326-1957>*
>> *(web) mvangala.com <http://mvangala.com>*
>>
>>
>> On Fri, Aug 24, 2018 at 9:18 AM Matthias Baetens <
>> baetensmatth...@gmail.com> wrote:
>>
>>> Hey Javier,
>>>
>>> Just to make sure you saw my answer, it really depends on the session.
>>> Happy to see the content you have in mind and we discuss times later :)
>>> (certainly if it is a workshop)
>>>
>>> Matthias
>>>
>>> On Fri, 24 Aug 2018 at 11:12 javier ramirez <
>>> javier.ramirez.gom...@gmail.com> wrote:
>>>
>>>> Thanks. I'll stick to 20+10 then when I send my proposal.
>>>>
>>>> Cheers!
>>>>
>>>> On Wed, Aug 22, 2018 at 10:58 AM Matthias Baetens <
>>>> baetensmatth...@gmail.com> wrote:
>>>>
>>>>> Hi Pascal, Javier,
>>>>>
>>>>> Thanks for your interest in submitting a talk!
>>>>> @Pascal: I am happy to check for you if what you have in mind is
>>>>> already being covered in another talk submitted, and happy to help you 
>>>>> find
>>>>> a good topic to talk about :)
>>>>>
>>>>> @Javier: it will depend on the type and subject of the session. If you
>>>>> are planning to do an advanced workshop, it makes sense to allocate more
>>>>> time (>1h). If it is a business case also covering architecture and
>>>>> technical, I think longer sessions should be possible as well (~1h
>>>>> including Q&A). If your talk turns out to have to much content, we can
>>>>> always look at ways to split it up into 2 stand-alone sessions if that
>>>>> would make sense.
>>>>>
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>> On Tue, 21 Aug 2018 at 19:45 javier ramirez <
>>>>> javier.ramirez.gom...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> What'd be the duration of the talks? So I can scope the contents of
>>>>>> my proposal.
>>>>>>
>>>>>> Looking forward to the summit!
>>>>>>
>>>>>> J
>>>>>>
>>>>>> On Tue, 21 Aug 2018, 14:47 Pascal Gula,  wrote:
>>>>>>
>>>>>>> Hi Matthias,
>>>>>>> we (Peat / Plantix) might be interested by submitting a talk and I
>>>>>>> would like to know if we can get access to the list of already submitted
>>>>>>> "Title" to avoid submitting on similar topic!
>>>>>>> Cheers,
>>>>>>> Pascal
>>>>>>>
>>>>>>> On Tue, Aug 21, 2018 at 1:59 PM, Matthias Baetens <
>>>>>>> baetensmatth...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> We are happy to invite you to the first Beam Summit in London.
>>>>>>>>
>>>>>>>> The summit will be held in London at Level39
>>>>>>>>

Re: Slides from OSCON

[Matthias Baetens]

Hey Holden,
Have you found the video in the meanwhile? Happy to upload it to the Beam
channel :)
Cheers,
M

On Wed, 25 Jul 2018 at 02:52 Holden Karau  wrote:

> Hi Friends,
>
> If you missed my talk @ OSCON on TFT/TFMA + Beam on Flink (and other
> related adventures), I've uploaded my slides to
> https://www.slideshare.net/hkarau/powering-tensorflow-with-big-data-using-apache-beam-flink-and-spark-oscon-pdx-2018
> and once the video is done I'll try and find a way to share that as well.
>
> Cheers,
>
> Holden :)
>
>
> --
> Twitter: https://twitter.com/holdenkarau
>
--

Re: Beam Summit London 2018

[Matthias Baetens]

Hey Javier,

Just to make sure you saw my answer, it really depends on the session.
Happy to see the content you have in mind and we discuss times later :)
(certainly if it is a workshop)

Matthias

On Fri, 24 Aug 2018 at 11:12 javier ramirez 
wrote:

> Thanks. I'll stick to 20+10 then when I send my proposal.
>
> Cheers!
>
> On Wed, Aug 22, 2018 at 10:58 AM Matthias Baetens <
> baetensmatth...@gmail.com> wrote:
>
>> Hi Pascal, Javier,
>>
>> Thanks for your interest in submitting a talk!
>> @Pascal: I am happy to check for you if what you have in mind is already
>> being covered in another talk submitted, and happy to help you find a good
>> topic to talk about :)
>>
>> @Javier: it will depend on the type and subject of the session. If you
>> are planning to do an advanced workshop, it makes sense to allocate more
>> time (>1h). If it is a business case also covering architecture and
>> technical, I think longer sessions should be possible as well (~1h
>> including Q&A). If your talk turns out to have to much content, we can
>> always look at ways to split it up into 2 stand-alone sessions if that
>> would make sense.
>>
>> Best,
>> Matthias
>>
>> On Tue, 21 Aug 2018 at 19:45 javier ramirez <
>> javier.ramirez.gom...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> What'd be the duration of the talks? So I can scope the contents of my
>>> proposal.
>>>
>>> Looking forward to the summit!
>>>
>>> J
>>>
>>> On Tue, 21 Aug 2018, 14:47 Pascal Gula,  wrote:
>>>
>>>> Hi Matthias,
>>>> we (Peat / Plantix) might be interested by submitting a talk and I
>>>> would like to know if we can get access to the list of already submitted
>>>> "Title" to avoid submitting on similar topic!
>>>> Cheers,
>>>> Pascal
>>>>
>>>> On Tue, Aug 21, 2018 at 1:59 PM, Matthias Baetens <
>>>> baetensmatth...@gmail.com> wrote:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> We are happy to invite you to the first Beam Summit in London.
>>>>>
>>>>> The summit will be held in London at Level39
>>>>> <https://goo.gl/maps/LAC4haDzSzR2> on *October 1 and 2.*
>>>>> You can register to attend for free on the Eventbrite page
>>>>> <https://www.eventbrite.com/e/beam-summit-london-2018-tickets-49100625292>
>>>>> .
>>>>>
>>>>> If you are interested in talking, please check our CfP form
>>>>> <https://goo.gl/forms/nrZOCC1JwEfLtKfA2> and submit a talk!
>>>>>
>>>>> If you or your company is interested in helping out or sponsoring the
>>>>> summit (to keep it free), you can check out the sponsor booklet
>>>>> <https://drive.google.com/file/d/1RnZ52rGaB6BR-EKneBcabdMcg9Pl7z9M/view?usp=sharing>
>>>>> .
>>>>>
>>>>> We will soon launch a blogpost with more details and announce the
>>>>> agenda closer to date.
>>>>>
>>>>> Thanks to everyone who helped make this happen, looking forward to
>>>>> welcoming you all in London!
>>>>>
>>>>> The Events & Meetups Group
>>>>>
>>>>> --
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Pascal Gula
>>>> Senior Data Engineer / Scientist+49 (0)176 34232684 
>>>> <+49%20176%2034232684>www.plantix.net <http://plantix.net/>
>>>>  PEAT GmbHKastanienallee 4
>>>> 10435 Berlin // Germany 
>>>> <https://maps.google.com/?q=Kastanienallee+4%0D%0A10435+Berlin+//+Germany&entry=gmail&source=g>
>>>>  
>>>> <https://play.google.com/store/apps/details?id=com.peat.GartenBank>Download
>>>>  the App! 
>>>> <https://play.google.com/store/apps/details?id=com.peat.GartenBank>
>>>>
>>>> --
>>
>>
> --

Re: Beam Summit London 2018

[Matthias Baetens]

Hi Pascal, Javier,

Thanks for your interest in submitting a talk!
@Pascal: I am happy to check for you if what you have in mind is already
being covered in another talk submitted, and happy to help you find a good
topic to talk about :)

@Javier: it will depend on the type and subject of the session. If you are
planning to do an advanced workshop, it makes sense to allocate more time
(>1h). If it is a business case also covering architecture and technical, I
think longer sessions should be possible as well (~1h including Q&A). If
your talk turns out to have to much content, we can always look at ways to
split it up into 2 stand-alone sessions if that would make sense.

Best,
Matthias

On Tue, 21 Aug 2018 at 19:45 javier ramirez 
wrote:

> Hi,
>
> What'd be the duration of the talks? So I can scope the contents of my
> proposal.
>
> Looking forward to the summit!
>
> J
>
> On Tue, 21 Aug 2018, 14:47 Pascal Gula,  wrote:
>
>> Hi Matthias,
>> we (Peat / Plantix) might be interested by submitting a talk and I would
>> like to know if we can get access to the list of already submitted "Title"
>> to avoid submitting on similar topic!
>> Cheers,
>> Pascal
>>
>> On Tue, Aug 21, 2018 at 1:59 PM, Matthias Baetens <
>> baetensmatth...@gmail.com> wrote:
>>
>>> Hi everyone,
>>>
>>> We are happy to invite you to the first Beam Summit in London.
>>>
>>> The summit will be held in London at Level39
>>> <https://goo.gl/maps/LAC4haDzSzR2> on *October 1 and 2.*
>>> You can register to attend for free on the Eventbrite page
>>> <https://www.eventbrite.com/e/beam-summit-london-2018-tickets-49100625292>
>>> .
>>>
>>> If you are interested in talking, please check our CfP form
>>> <https://goo.gl/forms/nrZOCC1JwEfLtKfA2> and submit a talk!
>>>
>>> If you or your company is interested in helping out or sponsoring the
>>> summit (to keep it free), you can check out the sponsor booklet
>>> <https://drive.google.com/file/d/1RnZ52rGaB6BR-EKneBcabdMcg9Pl7z9M/view?usp=sharing>
>>> .
>>>
>>> We will soon launch a blogpost with more details and announce the agenda
>>> closer to date.
>>>
>>> Thanks to everyone who helped make this happen, looking forward to
>>> welcoming you all in London!
>>>
>>> The Events & Meetups Group
>>>
>>> --
>>>
>>>
>>
>>
>>
>> --
>>
>> Pascal Gula
>> Senior Data Engineer / Scientist+49 (0)176 34232684 
>> <+49%20176%2034232684>www.plantix.net <http://plantix.net/>
>>  PEAT GmbHKastanienallee 4
>> 10435 Berlin // Germany 
>> <https://maps.google.com/?q=Kastanienallee+4%0D%0A10435+Berlin+//+Germany&entry=gmail&source=g>
>>  <https://play.google.com/store/apps/details?id=com.peat.GartenBank>Download 
>> the App! <https://play.google.com/store/apps/details?id=com.peat.GartenBank>
>>
>> --

Beam Summit London 2018

[Matthias Baetens]

Hi everyone,

We are happy to invite you to the first Beam Summit in London.

The summit will be held in London at Level39
 on *October 1 and 2.*
You can register to attend for free on the Eventbrite page
.

If you are interested in talking, please check our CfP form
 and submit a talk!

If you or your company is interested in helping out or sponsoring the
summit (to keep it free), you can check out the sponsor booklet

.

We will soon launch a blogpost with more details and announce the agenda
closer to date.

Thanks to everyone who helped make this happen, looking forward to
welcoming you all in London!

The Events & Meetups Group

--

Re: Apache Beam Summit in Europe

[Matthias Baetens]

Hi everyone!

Thanks for filling out the survey. We are currently looking into the
practicalities (venue, dates, ...).
To get a better grasp of the size of the audience, Alex has been so kind to
start a poll on Twitter:
https://twitter.com/alexvb/status/1014531906564775937. If you have a spare
minute, it would be great if you could fill this out (retweets are, of
course, greatly appreciated as well).

For the dates, we are thinking of:
- 28-29/09
- 1-2/10 or 3-4/10
- 15-16/10

Happy to hear your preferences and comments.
Best,
Matthias

On Thu, 31 May 2018 at 07:15 Matthias Baetens 
wrote:

> Hi Beam Community,
>
> We are planning to have an Apache Beam Summit in Europe, pretty similar to
> what we hosted earlier in California on March 15th this year.
> If you'd be interested in attending, helping with organization or
> speaking, please answer this Google form
> <https://docs.google.com/forms/d/e/1FAIpQLSeBoxWIOumjGijU_kTccLqGqS9PrQU1RqRVp9Cla5F4HLO1DQ/viewform>
> !
>
> Cheers,
> Matthias & the events and meetups folks
> --
>
>
--

Re: [Events] Big Data in Production Boston Meetup, Today!

[Matthias Baetens]

The video can be found on the Beam YouTube channel as well:
https://youtu.be/pPnY2y-zfaI

On Thu, 28 Jun 2018 at 20:55 Eila Arich-Landkof 
wrote:

> I can download it and share it. You might be able to import it / add it
> directly from Oriel research channel.
> Let me know if this doesn’t work and I will download it
>
> Thanks,
>
> ———
> Eila
> www.orielresearch.org
> www.meetup.com/deep-learning-in-production
>
> On Jun 28, 2018, at 12:22 PM, Matthias Baetens 
> wrote:
>
> Hey Eila,
>
> Do you have the original file and could you share that with me? Is
> everyone in the video and the venue fine with sharing it?
>
> Thanks!
> Matthias
>
> On Thu, 28 Jun 2018 at 14:27 OrielResearch Eila Arich-Landkof <
> e...@orielresearch.org> wrote:
>
>> Sure. great great!
>>
>> On Thu, Jun 28, 2018 at 5:52 AM, Matthias Baetens <
>> baetensmatth...@gmail.com> wrote:
>>
>>> Looks awesome! Shall I add the talk to the Beam YouTube channel as well?
>>>
>>> On Tue, 26 Jun 2018 at 23:40 Griselda Cuevas  wrote:
>>>
>>>> In case you'd like to follow the talk live, here is the livestream
>>>> link:
>>>>
>>>> https://www.youtube.com/watch?v=aSU9y0E-WMA
>>>>
>>>>
>>>>
>>>> On Tue, 26 Jun 2018 at 16:11, Griselda Cuevas  wrote:
>>>>
>>>>> Hi Beam Community,
>>>>>
>>>>> Robert Bradshaw (Google) and Eila Arich-Landkof (Oriel Research) will
>>>>> be speaking about Apache Beam at the Big Data in Production Boston
>>>>> Meetup
>>>>> <https://www.meetup.com/Deep-Learning-In-Production/events/250589154/> 
>>>>> today
>>>>> in the Google office in Cambridge, MA. We'll share the video and slides
>>>>> with the mailing list afterwards.
>>>>>
>>>>> Regards,
>>>>> G
>>>>>
>>>>> *P.S. If you'd like to organize a Meetup/Conference about Apache Beam,
>>>>> don't forget to join our beam-events-meetups Slack channel to join the
>>>>> convo!*
>>>>>
>>>> --
>>>
>>>
>>
>>
>>
>> --
>> Eila
>> www.orielresearch.org
>> https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>
>> p.co <https://www.meetup.com/Deep-Learning-In-Production/>
>> m/Deep-Learning-In-Production/
>> <https://www.meetup.com/Deep-Learning-In-Production/>
>>
>>
>> --
>
>
> --

Re: London Apache Beam meetup 5

[Matthias Baetens]

Also sharing the (audio) recording now:
http://bit.ly/FifthApacheBeamMeetupLondonrecording
Enjoy!

On Mon, 11 Jun 2018 at 16:41 Alexey Romanenko 
wrote:

> Thank you for sharing, Matthias!
>
> WBR,
> Alexey
>
>
> On 10 Jun 2018, at 14:45, Matthias Baetens 
> wrote:
>
> Hi all,
>
> Sharing the slides from the meetup this week here
> <http://bit.ly/ApacheBeamLondonMeetupSlides5>. Enjoy!
>
> Best,
> Matthias
>
> On Sun, 3 Jun 2018 at 19:38 Matthias Baetens 
> wrote:
>
>> Hi all,
>>
>> Happy to announce our 5th Beam meetup in London
>>
>> This Thursday, 7th of June, we’ll have Reuven Lax, senior software
>> engineer at Google and Jean-Baptiste, software architect at Talend at the
>> Qubit <https://goo.gl/maps/75QRfitc3ZE2> office to give you an update on
>> the Beam project.
>>
>> JB will be taking use cases, while Reuven will talk about Beam latest and
>> upcoming features, with, among things, the portability layer and schema
>> support in Beam.
>>
>> I am very happy to announce we will have some Beam swag (not only
>> stickers) this time - so be on time if you don’t want to miss out ;)
>>
>> More info and RSVP on the meetup page
>> <http://bit.ly/FifthApacheBeamMeetupLondon>.
>>
>> Cheers,
>>
>> Matthias
>> --
>>
>>
> --
>
>
>
> --

Re: [Events] Big Data in Production Boston Meetup, Today!

[Matthias Baetens]

Hey Eila,

Do you have the original file and could you share that with me? Is everyone
in the video and the venue fine with sharing it?

Thanks!
Matthias

On Thu, 28 Jun 2018 at 14:27 OrielResearch Eila Arich-Landkof <
e...@orielresearch.org> wrote:

> Sure. great great!
>
> On Thu, Jun 28, 2018 at 5:52 AM, Matthias Baetens <
> baetensmatth...@gmail.com> wrote:
>
>> Looks awesome! Shall I add the talk to the Beam YouTube channel as well?
>>
>> On Tue, 26 Jun 2018 at 23:40 Griselda Cuevas  wrote:
>>
>>> In case you'd like to follow the talk live, here is the livestream link:
>>>
>>> https://www.youtube.com/watch?v=aSU9y0E-WMA
>>>
>>>
>>>
>>> On Tue, 26 Jun 2018 at 16:11, Griselda Cuevas  wrote:
>>>
>>>> Hi Beam Community,
>>>>
>>>> Robert Bradshaw (Google) and Eila Arich-Landkof (Oriel Research) will
>>>> be speaking about Apache Beam at the Big Data in Production Boston
>>>> Meetup
>>>> <https://www.meetup.com/Deep-Learning-In-Production/events/250589154/> 
>>>> today
>>>> in the Google office in Cambridge, MA. We'll share the video and slides
>>>> with the mailing list afterwards.
>>>>
>>>> Regards,
>>>> G
>>>>
>>>> *P.S. If you'd like to organize a Meetup/Conference about Apache Beam,
>>>> don't forget to join our beam-events-meetups Slack channel to join the
>>>> convo!*
>>>>
>>> --
>>
>>
>
>
>
> --
> Eila
> www.orielresearch.org
> https://www.meetu <https://www.meetup.com/Deep-Learning-In-Production/>
> p.co <https://www.meetup.com/Deep-Learning-In-Production/>
> m/Deep-Learning-In-Production/
> <https://www.meetup.com/Deep-Learning-In-Production/>
>
>
> --

Re: [Events] Big Data in Production Boston Meetup, Today!

[Matthias Baetens]

Looks awesome! Shall I add the talk to the Beam YouTube channel as well?

On Tue, 26 Jun 2018 at 23:40 Griselda Cuevas  wrote:

> In case you'd like to follow the talk live, here is the livestream link:
>
> https://www.youtube.com/watch?v=aSU9y0E-WMA
>
>
>
> On Tue, 26 Jun 2018 at 16:11, Griselda Cuevas  wrote:
>
>> Hi Beam Community,
>>
>> Robert Bradshaw (Google) and Eila Arich-Landkof (Oriel Research) will be
>> speaking about Apache Beam at the Big Data in Production Boston Meetup
>>  today
>> in the Google office in Cambridge, MA. We'll share the video and slides
>> with the mailing list afterwards.
>>
>> Regards,
>> G
>>
>> *P.S. If you'd like to organize a Meetup/Conference about Apache Beam,
>> don't forget to join our beam-events-meetups Slack channel to join the
>> convo!*
>>
> --

Re: [FYI] New Apache Beam Swag Store!

[Matthias Baetens]

Great news, big thanks for all the work, Gris! Looking forward to people
wearing this around the globe ;)

On Sat, 9 Jun 2018 at 01:28 Ankur Goenka  wrote:

> Awesome!
>
>
> On Fri, Jun 8, 2018 at 4:24 PM Pablo Estrada  wrote:
>
>> Nice : D
>>
>> On Fri, Jun 8, 2018, 3:43 PM Raghu Angadi  wrote:
>>
>>> Woo-hoo! This is terrific.
>>>
>>> If we are increasing color choices I would like black or charcoal...
>>> Beam logo would really pop on a dark background.
>>>
>>> On Fri, Jun 8, 2018 at 3:32 PM Griselda Cuevas  wrote:
>>>
>>>> Hi Beam Community,
>>>>
>>>> I just want to share with you the exciting news about our brand new
>>>> Apache Beam Swag Store!
>>>>
>>>> You can find it here: https://store-beam.myshopify.com/
>>>>
>>>> *How does it work?*
>>>>
>>>>- You can just select the items you want and check-out. Our Vendor
>>>>ships to anywhere in the world and normally can have swag to be 
>>>> delivered
>>>>within 1 week. Each company or user will need to pay for their own swag.
>>>>- If you are hosting an event or representing Beam at one, reach
>>>>out to me or the beam-events-meetups slack channel, I'll be happy to 
>>>> review
>>>>your event and see if we can sponsor the swag. We'll have codes for this
>>>>occasions thanks to Google, who has sponsored an initial inventory.
>>>>
>>>> If you have feedback, ideas on new swag, questions or suggestions,
>>>> reach out to me and/or Matthias Baetens.
>>>>
>>>> Happy Friday!
>>>> G
>>>>
>>>>
>>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
> --

Re: London Apache Beam meetup 5

[Matthias Baetens]

Hi all,

Sharing the slides from the meetup this week here
<http://bit.ly/ApacheBeamLondonMeetupSlides5>. Enjoy!

Best,
Matthias

On Sun, 3 Jun 2018 at 19:38 Matthias Baetens 
wrote:

> Hi all,
>
> Happy to announce our 5th Beam meetup in London
>
> This Thursday, 7th of June, we’ll have Reuven Lax, senior software
> engineer at Google and Jean-Baptiste, software architect at Talend at the
> Qubit <https://goo.gl/maps/75QRfitc3ZE2> office to give you an update on
> the Beam project.
>
> JB will be taking use cases, while Reuven will talk about Beam latest and
> upcoming features, with, among things, the portability layer and schema
> support in Beam.
>
> I am very happy to announce we will have some Beam swag (not only
> stickers) this time - so be on time if you don’t want to miss out ;)
>
> More info and RSVP on the meetup page
> <http://bit.ly/FifthApacheBeamMeetupLondon>.
>
> Cheers,
>
> Matthias
> --
>
>
--

London Apache Beam meetup 5

[Matthias Baetens]

Hi all,

Happy to announce our 5th Beam meetup in London

This Thursday, 7th of June, we’ll have Reuven Lax, senior software engineer
at Google and Jean-Baptiste, software architect at Talend at the Qubit
 office to give you an update on the Beam
project.

JB will be taking use cases, while Reuven will talk about Beam latest and
upcoming features, with, among things, the portability layer and schema
support in Beam.

I am very happy to announce we will have some Beam swag (not only stickers)
this time - so be on time if you don’t want to miss out ;)

More info and RSVP on the meetup page
.

Cheers,

Matthias
--

Apache Beam Summit in Europe

[Matthias Baetens]

Hi Beam Community,

We are planning to have an Apache Beam Summit in Europe, pretty similar to
what we hosted earlier in California on March 15th this year.
If you'd be interested in attending, helping with organization or speaking,
please answer this Google form

!

Cheers,
Matthias & the events and meetups folks
--

Beam YouTube channel

[Matthias Baetens]

Hi all!

Just reminding you all that we'd be happy to receive contributions of
recordings (talks, meetups, conferences, demo's) to put on the YouTube
channel.

Read more about the process and guidelines here

.

Do not hesitate to reach out to me (directly) if you want your video up
there!

Cheers,
Matthias
--

Re: I'm back and ready to help grow our community!

[Matthias Baetens]

Same here - shame on me. Congratulations on the graduation Gris, very happy
to have you back!

On Tue, 22 May 2018 at 09:19 Ismaël Mejía  wrote:

> I missed somehow this email thread.
> Congratulations Gris and welcome back!
>
> On Fri, May 18, 2018 at 5:34 AM Jesse Anderson 
> wrote:
>
> > Congrats!
>
> > On Thu, May 17, 2018, 6:44 PM Robert Burke  wrote:
>
> >> Congrats & welcome back!
>
> >> On Thu, May 17, 2018, 5:44 PM Huygaa Batsaikhan 
> wrote:
>
> >>> Welcome back, Gris! Congratulations!
>
> >>> On Thu, May 17, 2018 at 4:24 PM Robert Bradshaw 
> wrote:
>
>  Congratulations, Gris! And welcome back!
>  On Thu, May 17, 2018 at 3:30 PM Robin Qiu  wrote:
>
>  > Congratulations! Welcome back!
>
>  > On Thu, May 17, 2018 at 3:23 PM Reuven Lax 
> wrote:
>
>  >> Congratulations! Good to see you back!
>
>  >> Reuven
>
>  >> On Thu, May 17, 2018 at 2:24 PM Griselda Cuevas 
> wrote:
>
>  >>> Hi Everyone,
>
>
>  >>> I was absent from the mailing list, slack channel and our Beam
>  community for the past six weeks, the reason was that I took a leave
> to
>  focus on finishing my Masters Degree, which I finally did on May 15th.
>
>
>  >>> I graduated as a Masters of Engineering in Operations Research
> with a
>  concentration in Data Science from UC Berkeley. I'm glad to be part of
> this
>  community and I'd like to share this accomplishment with you so I'm
> adding
>  two pictures of that day :)
>
>
>  >>> Given that I've seen so many new folks around, I'd like to use
> this
>  opportunity to re-introduce myself. I'm Gris Cuevas and I work at
> Google.
>  Now that I'm back, I'll continue to work on supporting our community
> in two
>  main streams: Contribution Experience & Events, Meetups, and
> Conferences.
>
>
>  >>> It's good to be back and I look forward to collaborating with you.
>
>
>  >>> Cheers,
>
>  >>> Gris
>
--

Third Apache Beam meet up in London

[Matthias Baetens]

Hi all,

We are very excited to announce the third Apache Beam meetup in London *the
5th of March*.

Tyler Akidau (Google) will be talking about Streaming SQL, while Victor
Kotai (Qubit) will talk about putting a Beam pipeline in production and how
to monitor it, while I (Datatonic) will do a talk on how to leverage Beam
for a Machine Learning use case.

More info and RSVP: http://bit.ly/3rdApacheBeamMeetupLondon

We'll do our best to record and share the session again as well.

Best regards,
Matthias

Re: London Apache Beam meetup 2: 11/01

[Matthias Baetens]

As promised (but with some delay): the recording
<http://bit.ly/SecondApacheBeamMeetupLondonRecording> of our second Beam
meetup in London! Enjoy :-)

On Tue, Jan 9, 2018 at 12:30 PM, Carlos Alonso  wrote:

> Cool, thanks!!
>
>
> On Mon, Jan 8, 2018 at 1:38 PM Matthias Baetens <
> matthias.baet...@datatonic.com> wrote:
>
>> Yes, we put everything in place to record this time and hope to share the
>> recordings soon after the meetup. Stay tuned!
>>
>> On 8 Jan 2018 10:32, "Carlos Alonso"  wrote:
>>
>> Will it be recorded?
>>
>> On Fri, Jan 5, 2018 at 5:11 PM Matthias Baetens <
>> matthias.baet...@datatonic.com> wrote:
>>
>>> Hi all,
>>>
>>> Excited to announce the second Beam meet up located in the *Qubit
>>> offices <https://goo.gl/maps/sVmFYrVys1S2> *next *Thursday 11/01.*
>>>
>>> We are very excited to have JB flying in to talk about IO and Splittable
>>> DoFns and Vadim Sobolevski to share on how FutureFlow uses Beam in a
>>> finance use case.
>>>
>>> More info and RSVP here <http://bit.ly/2zcUy5A>. We are looking forward
>>> to welcome you all!
>>>
>>> Best regards,
>>> Matthias
>>>
>>
>>


-- 


*Matthias Baetens*


*datatonic | data power unleashed*

office +44 203 668 3680  |  mobile +44 74 918 20646

Level24 | 1 Canada Square | Canary Wharf | E14 5AB London


We've been announced
<https://blog.google/topics/google-cloud/investing-vibrant-google-cloud-ecosystem-new-programs-and-partnerships/>
as
one of the top global Google Cloud Machine Learning partners.

Re: Strata Conference this March 6-8

[Matthias Baetens]

Sure, I'd be very happy to organise something. This is about Strata San
Jose though right? Maybe we can organise a remote session in which we can
join (depending on when you would organise the BoF) or have a channel
set-up if the talks would be broadcasted?

Also: will there be any Beam talks on Strata London or is this not known
yet? Keen to get involved and set things up around that date as well.

On Wed, Jan 17, 2018 at 8:37 AM, Jean-Baptiste Onofré 
wrote:

> That's a great idea ! I'm sure that Matthias (organizer of the Beam London
> Meetup) can help us to plan something.
>
> Regards
> JB
>
>
> On 01/17/2018 08:57 AM, Ismaël Mejía wrote:
>
>> Maybe a good idea to try to organize a Beam meetup in london in the
>> same dates in case some of the people around can jump in and talk too.
>>
>> On Wed, Jan 17, 2018 at 2:51 AM, Ron Gonzalez 
>> wrote:
>>
>>> Works for me...
>>>
>>> On Tuesday, January 16, 2018, 5:45:33 PM PST, Holden Karau
>>>  wrote:
>>>
>>>
>>> How would folks feel about during the afternoon break (3:20-4:20) on the
>>> Wednesday (same day as Eugene's talk)? We could do the Philz which is a
>>> bit
>>> of a walk but gets us away from the big crowd and also lets folks not
>>> attending the conference but in the area join us.
>>>
>>> On Tue, Jan 16, 2018 at 5:29 PM, Ron Gonzalez 
>>> wrote:
>>>
>>> Cool, let me know if you guys finally schedule it. I will definitely try
>>> to
>>> make it to Eugene's talk but having an informal BoF in the area would be
>>> nice...
>>>
>>> Thanks,
>>> Ron
>>>
>>> On Tuesday, January 16, 2018, 5:06:53 PM PST, Boris Lublinsky
>>>  wrote:
>>>
>>>
>>> All for it
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublin...@lightbend.com
>>> https://www.lightbend.com/
>>>
>>> On Jan 16, 2018, at 7:01 PM, Ted Yu  wrote:
>>>
>>> +1 to BoF
>>>
>>> On Tue, Jan 16, 2018 at 5:00 PM, Dmitry Demeshchuk >> >
>>> wrote:
>>>
>>> Probably won't be attending the conference, but totally down for a BoF.
>>>
>>> On Tue, Jan 16, 2018 at 4:58 PM, Holden Karau 
>>> wrote:
>>>
>>> Do interested folks have any timing constraints around a BoF?
>>>
>>> On Tue, Jan 16, 2018 at 4:30 PM, Jesse Anderson <
>>> je...@bigdatainstitute.io>
>>> wrote:
>>>
>>> +1 to BoF. I don't know if any Beam talks will be on the schedule.
>>>
>>> We could do an informal BoF at the Philz nearby or similar?
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Twitter: https://twitter.com/h oldenkarau
>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Dmitry Demeshchuk.
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>



-- 


*Matthias Baetens*


*datatonic | data power unleashed*

office +44 203 668 3680  |  mobile +44 74 918 20646

Level24 | 1 Canada Square | Canary Wharf | E14 5AB London


We've been announced
<https://blog.google/topics/google-cloud/investing-vibrant-google-cloud-ecosystem-new-programs-and-partnerships/>
as
one of the top global Google Cloud Machine Learning partners.

Re: London Apache Beam meetup 2: 11/01

[Matthias Baetens]

Yes, we put everything in place to record this time and hope to share the
recordings soon after the meetup. Stay tuned!

On 8 Jan 2018 10:32, "Carlos Alonso"  wrote:

Will it be recorded?

On Fri, Jan 5, 2018 at 5:11 PM Matthias Baetens  wrote:

> Hi all,
>
> Excited to announce the second Beam meet up located in the *Qubit offices
> <https://goo.gl/maps/sVmFYrVys1S2> *next *Thursday 11/01.*
>
> We are very excited to have JB flying in to talk about IO and Splittable
> DoFns and Vadim Sobolevski to share on how FutureFlow uses Beam in a
> finance use case.
>
> More info and RSVP here <http://bit.ly/2zcUy5A>. We are looking forward
> to welcome you all!
>
> Best regards,
> Matthias
>

London Apache Beam meetup 2: 11/01

[Matthias Baetens]

Hi all,

Excited to announce the second Beam meet up located in the *Qubit offices
 *next *Thursday 11/01.*

We are very excited to have JB flying in to talk about IO and Splittable
DoFns and Vadim Sobolevski to share on how FutureFlow uses Beam in a
finance use case.

More info and RSVP here . We are looking forward to
welcome you all!

Best regards,
Matthias

Re: London Apache Beam meetup 2: call for speakers

[Matthias Baetens]

@Gris: This is a call for speakers - we launched a similar one through the
meetup channels. Anyone interested in talking, we are looking forward to
hear from them!

@Andras: that's exactly what we were planning on in our second edition.
Happy to share through the official Beam channels if they are up and
running, we will also share on the meetup page & through the channels of
Datatonic <https://twitter.com/teamdatatonic> and Qubit
<https://twitter.com/qubit> :)

On Thu, Nov 9, 2017 at 9:07 PM, AndrasNagy  wrote:

> Thx.
> Great pls post the links here too ;)
>
> On 9 Nov 2017 10:05 pm, "Griselda Cuevas"  wrote:
>
>> Great suggestion (about the Youtube channel). I have the recording of the
>> talks we gave here in SF this past 11/1 but I need to get it from our tech
>> team and upload it somewhere, will look into a youtube channel :)
>>
>>
>> Gris Cuevas Zambrano
>>
>> g...@google.com
>>
>> Open Source Strategy
>>
>> 345 Spear Street, San Francisco, 94105
>> <https://maps.google.com/?q=345+Spear+Street,+San+Francisco,+94105&entry=gmail&source=g>
>>
>>
>>
>> On 9 November 2017 at 14:01, AndrasNagy  wrote:
>>
>>> Hey.
>>> Please make sure all talks are going to be recorded and later uploaded
>>> to youtube or ustream.
>>>
>>> We are all really interested but many people (include myself) do not
>>> have the opportunity to attend.
>>>
>>> Thx
>>>
>>> On 9 Nov 2017 21:55, "Griselda Cuevas"  wrote:
>>>
>>>> Maybe send a call for speakers and see if we have anyone from the
>>>> community visiting London who could be a speaker?
>>>>
>>>> It'd be great to have one London Meetup on 12/5 because we're hosting
>>>> one in SF and one in NY so it'd be awesome to have three on the same day!
>>>>
>>>>
>>>> Gris Cuevas Zambrano
>>>>
>>>> g...@google.com
>>>>
>>>> Open Source Strategy
>>>>
>>>> 345 Spear Street, San Francisco, 94105
>>>> <https://maps.google.com/?q=345+Spear+Street,+San+Francisco,+94105&entry=gmail&source=g>
>>>>
>>>>
>>>>
>>>> On 8 November 2017 at 15:09, Matthias Baetens <
>>>> baetensmatth...@gmail.com> wrote:
>>>>
>>>>> No worries JB, I'll send you a message on how we can plan around this
>>>>> (reschedule the meetup or postpone your session).
>>>>> Thanks for the heads-up, have fun in Singapore!
>>>>>
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>> Op di 7 nov. 2017 om 04:52 schreef Jean-Baptiste Onofré <
>>>>> j...@nanthrax.net>:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> unfortunately, I have to decline the invite as I will be at Strata
>>>>>> Singapore in
>>>>>> the same time :(
>>>>>>
>>>>>> I'm very sorry about that. You can count on me for the 3rd edition !
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 11/07/2017 01:41 AM, Matthias Baetens wrote:
>>>>>> > Hi all!
>>>>>> >
>>>>>> > Hope you are well.
>>>>>> > We are back for a second edition of the London Apache Beam meetup,
>>>>>> aiming for
>>>>>> > the 5th of December.
>>>>>> >
>>>>>> > We are pretty excited to announce that our first speaker will be
>>>>>> > Jean-Baptiste Onofré <https://github.com/jbonofre> himself!
>>>>>> >
>>>>>> > If you have an interesting *use-case* to share and are in London on
>>>>>> the *5th of
>>>>>> > December*, don't hesitate to reach out to me :)
>>>>>> > Else: keep track of the meetup page
>>>>>> > <https://www.meetup.com/London-Apache-Beam-Meetup/> to be updated
>>>>>> on our
>>>>>> > activity in the space.
>>>>>> >
>>>>>> > Best,
>>>>>> > Matthias
>>>>>> > --
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> jbono...@apache.org
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>>>>
>>>>> --
>>>>>
>>>>>
>>>>
>>>>
>>


-- 


*Matthias Baetens*


*datatonic | data power unleashed*

office +44 203 668 3680  |  mobile +44 74 918 20646

Level24 | 1 Canada Square | Canary Wharf | E14 5AB London


We've been announced
<https://blog.google/topics/google-cloud/investing-vibrant-google-cloud-ecosystem-new-programs-and-partnerships/>
as
one of the top global Google Cloud Machine Learning partners.

Re: London Apache Beam meetup 2: call for speakers

[Matthias Baetens]

No worries JB, I'll send you a message on how we can plan around this
(reschedule the meetup or postpone your session).
Thanks for the heads-up, have fun in Singapore!

Best,
Matthias

Op di 7 nov. 2017 om 04:52 schreef Jean-Baptiste Onofré :

> Hi,
>
> unfortunately, I have to decline the invite as I will be at Strata
> Singapore in
> the same time :(
>
> I'm very sorry about that. You can count on me for the 3rd edition !
>
> Regards
> JB
>
> On 11/07/2017 01:41 AM, Matthias Baetens wrote:
> > Hi all!
> >
> > Hope you are well.
> > We are back for a second edition of the London Apache Beam meetup,
> aiming for
> > the 5th of December.
> >
> > We are pretty excited to announce that our first speaker will be
> > Jean-Baptiste Onofré <https://github.com/jbonofre> himself!
> >
> > If you have an interesting *use-case* to share and are in London on the
> *5th of
> > December*, don't hesitate to reach out to me :)
> > Else: keep track of the meetup page
> > <https://www.meetup.com/London-Apache-Beam-Meetup/> to be updated on our
> > activity in the space.
> >
> > Best,
> > Matthias
> > --
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--

London Apache Beam meetup 2: call for speakers

[Matthias Baetens]

Hi all!

Hope you are well.
We are back for a second edition of the London Apache Beam meetup, aiming
for the 5th of December.

We are pretty excited to announce that our first speaker will be
Jean-Baptiste Onofré  himself!

If you have an interesting *use-case* to share and are in London on the *5th
of December*, don't hesitate to reach out to me :)
Else: keep track of the meetup page
 to be updated on our
activity in the space.

Best,
Matthias
--

Re: PipelineTest with TestStreams: unable to serialize

[Matthias Baetens]

Hi Aleksandr,

Awesome. That did the trick. Did bump in to this relevant SO answer
<https://stackoverflow.com/a/28033607/6316101>, but overlooked those
TupleTags, good spot!

@Eugene: thanks a lot for the tip. Will give it a try soon ;)

Cheers,
Matthias

On Fri, Nov 3, 2017 at 9:00 PM, Eugene Kirpichov 
wrote:

> When debugging serialization exceptions, I always find it very helpful to
> use  -Dsun.io.serialization.extendedDebugInfo=true .
>
> On Fri, Nov 3, 2017 at 9:21 AM Aleksandr  wrote:
>
>> Hello,
>> Probably error is in your tuple tag classes, which are anonymous classes.
>> It means that your test is trying to serialise testpipeline.
>>
>> Best regards
>> Aleksandr Gortujev
>>
>>
>>
>> 3. nov 2017 3:33 PM kirjutas kuupäeval "Matthias Baetens" <
>> matthias.baet...@datatonic.com>:
>>
>> Hi all,
>>
>> I'm currently trying to write a TestStream to validate the windowing
>> logic in a Beam pipeline.
>>
>> I'm creating a teststream of Strings and applying the different
>> PTransforms to the stream, ending with a PAssert on some of the events I
>> created
>>
>> TestStream events = TestStream.create(AvroCoder.of(String.class))
>>  .addElements("", "")
>>  .advanceWatermarkToInfinity();
>>
>> PCollection>> eventsSessionised = 
>> p.apply(events)
>>
>>  .apply(new Processing(new 
>> TupleTag() {
>>  }, new TupleTag() {
>>  }, new TupleTag() {
>>  }, eventsEnrichedKeyedTag, "", "", 
>> "")).get(eventsEnrichedKeyedTag)
>>  .apply(new 
>> Sessionisation(SESSION_GAP_SIZE_HOURS, SESSION_CUT_OFF, 
>> ALLOWED_LATENESS_MINUTES))
>>  .apply(new Aggregation(uniqueEventsTag, new 
>> TupleTag() {
>>  })).get(uniqueEventsTag).apply(ParDo.of(new 
>> EventToKV()));
>>
>>
>> PAssert.that(eventsSessionised).inOnTimePane(new IntervalWindow(baseTime, 
>> endWindow1)).containsInAnyOrder(e1,
>>  e2);
>>
>> Running the test function with in a main functions (new
>> IngestionPipeLineTest().testOnTimeEvents();) causes the following error:
>>
>> Exception in thread "main" java.lang.IllegalArgumentException: unable to 
>> serialize
>>
>> pointing at a custom DoFn which runs fine running the main pipeline.
>>
>>
>> Not sure why this error gets thrown all of a sudden, any pointers / help 
>> would be greatly appreciated.
>>
>> Full stacktrace:
>>
>> Exception in thread "main" java.lang.IllegalArgumentException: unable to 
>> serialize xxx
>>  at 
>> org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:53)
>>  at 
>> org.apache.beam.sdk.util.SerializableUtils.clone(SerializableUtils.java:90)
>>  at 
>> org.apache.beam.sdk.transforms.ParDo$SingleOutput.(ParDo.java:591)
>>  at org.apache.beam.sdk.transforms.ParDo.of(ParDo.java:435)
>>  at xxx.transforms.Processing.expand(Processing.java:52)
>>  at xxx.transforms.Processing.expand(Processing.java:1)
>>  at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:514)
>>  at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:454)
>>  at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:284)
>>  at 
>> xxx.IngestionPipeLineTest.testOnTimeEvents(IngestionPipeLineTest.java:96)
>>  at xxx.IngestionPipeLineTest.main(IngestionPipeLineTest.java:155)
>> Caused by: java.io.NotSerializableException: 
>> org.apache.beam.sdk.testing.TestPipeline
>>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
>>  at 
>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>>  at 
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>>  at 
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>>  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
>>  at 
>> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
>>  at 
>> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
>>  at 
>> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
>>  a

PipelineTest with TestStreams: unable to serialize

[Matthias Baetens]

Hi all,

I'm currently trying to write a TestStream to validate the windowing logic
in a Beam pipeline.

I'm creating a teststream of Strings and applying the different PTransforms
to the stream, ending with a PAssert on some of the events I created

TestStream events = TestStream.create(AvroCoder.of(String.class))
.addElements("", "")
.advanceWatermarkToInfinity();

PCollection>> eventsSessionised = p.apply(events)

.apply(new Processing(new 
TupleTag() {
}, new TupleTag() {
}, new TupleTag() {
}, eventsEnrichedKeyedTag, "", "", 
"")).get(eventsEnrichedKeyedTag)
.apply(new 
Sessionisation(SESSION_GAP_SIZE_HOURS, SESSION_CUT_OFF,
ALLOWED_LATENESS_MINUTES))
.apply(new Aggregation(uniqueEventsTag, new 
TupleTag() {
})).get(uniqueEventsTag).apply(ParDo.of(new 
EventToKV()));


PAssert.that(eventsSessionised).inOnTimePane(new
IntervalWindow(baseTime, endWindow1)).containsInAnyOrder(e1,
e2);

Running the test function with in a main functions (new
IngestionPipeLineTest().testOnTimeEvents();) causes the following error:

Exception in thread "main" java.lang.IllegalArgumentException: unable
to serialize

pointing at a custom DoFn which runs fine running the main pipeline.


Not sure why this error gets thrown all of a sudden, any pointers /
help would be greatly appreciated.

Full stacktrace:

Exception in thread "main" java.lang.IllegalArgumentException: unable
to serialize xxx
at 
org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:53)
at 
org.apache.beam.sdk.util.SerializableUtils.clone(SerializableUtils.java:90)
at 
org.apache.beam.sdk.transforms.ParDo$SingleOutput.(ParDo.java:591)
at org.apache.beam.sdk.transforms.ParDo.of(ParDo.java:435)
at xxx.transforms.Processing.expand(Processing.java:52)
at xxx.transforms.Processing.expand(Processing.java:1)
at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:514)
at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:454)
at org.apache.beam.sdk.values.PCollection.apply(PCollection.java:284)
at 
xxx.IngestionPipeLineTest.testOnTimeEvents(IngestionPipeLineTest.java:96)
at xxx.IngestionPipeLineTest.main(IngestionPipeLineTest.java:155)
Caused by: java.io.NotSerializableException:
org.apache.beam.sdk.testing.TestPipeline
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at 
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at 
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at 
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at 
org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:49)
... 10 more


Best,

Matthias

Re: October Apache Beam Newsletter

[Matthias Baetens]

Great overview of what happened in Beam-land. Thanks a lot!

On Tue, Oct 10, 2017 at 4:17 AM, James  wrote:

> Cool, very informational, thanks!
>
> On Tue, Oct 10, 2017 at 2:39 AM Griselda Cuevas  wrote:
>
>> Hi Apache Beam Community,
>>
>> Our first Apache Beam Newsletter is here!, I'm sharing a table of
>> contents of what's in this edition, which covers everything that has
>> happened in the project from June 2017 until October 2017.
>>
>> You can find the full content in this Google Doc:
>> https://docs.google.com/document/d/1BbpQne-9ng93G-_-
>> UKH2C4UNafcEQcLALtu38qsXfI8/edit?usp=sharing
>>
>> Enjoy!
>>
>> * * * * * October 2017 Newsletter Table of Contents * * * * *
>>
>> >> What's Been Done
>> - Beam SQL DSL APIs
>> - Nexmark
>> - Splittable DoFn
>> - Improvements to reading and writing files
>> - Improvements to BigQueryIO
>> - New I/O connectors
>> - Docker development images and reproducible-builds
>> - Website updates: New Beam Execution Model page & the Mobile Gaming
>> Walkthrough was updated w/ new sample code for Python
>>
>> >> What We Are Working On
>> - Portability
>> - Splittable DoFn for Python SDK
>> - Website: Improve CoGroupByKey docs & website navigation/usability
>>
>> >> What's Planned
>> - FileIO.write()
>>
>> >>  New Members (Welcome!)
>> - Daniel Harper, BBC, London (UK)
>>
>> >>  Talks & Meetups
>> - Talks @ YOW Data Sydney and Strata NY
>> - Meetups @ London & dinner in NY
>> - Speakers & Meetup Founders group
>>
>> >>  Resources
>> - Capability Matrix
>> - Contribution Guide
>> - Featured talk & sample intro talk deck
>>
>


-- 


*Matthias Baetens*


*datatonic | data power unleashed*

office +44 203 668 3680  |  mobile +44 74 918 20646

Level24 | 1 Canada Square | Canary Wharf | E14 5AB London


We've been announced
<https://blog.google/topics/google-cloud/investing-vibrant-google-cloud-ecosystem-new-programs-and-partnerships/>
as
one of the top global Google Cloud Machine Learning partners.

Re: Apache Beam meetup London 1: debrief

[Matthias Baetens]

Awesome JB, glad you are happy. I'm having a call with Gris from Googl and
Victor (he's co-organising these with me) for the next meetup, probably in
November. Do you have any plans of being in London in that month? We would
be happy to adjust the timing to your needs to be able to have you talk!

Thanks :)
Best,
Matthias

Op di 10 okt. 2017 om 14:23 schreef Jean-Baptiste Onofré :

> Thanks for the update !
>
> That's a great stuff !
>
> Looking forward next Beam meetup and hope to be able to participate !
>
> Regards
> JB
>
> On 10/10/2017 03:11 PM, Matthias Baetens wrote:
> > Hi all!
> >
> > We had our first Apache Beam meetup last week, and we just released a
> small
> > blogpost
> > <
> http://blog.datatonic.com/2017/10/first-apache-beam-meetup-in-london.html> 
> about
> it!
> > Stay tuned for more of these or reach out if you have something
> interesting to
> > share :)
> >
> > Also: if you are thinking to set up something similar in your city, I am
> happy
> > to join forces and share thoughts and ideas. Don't doubt to reach out to
> > Griselda (in cc) as well - she has been a great help for us so far!
> >
> > Best,
> > Matthias
> > --
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--

Apache Beam meetup London 1: debrief

[Matthias Baetens]

Hi all!

We had our first Apache Beam meetup last week, and we just released a small
blogpost

about
it!
Stay tuned for more of these or reach out if you have something interesting
to share :)

Also: if you are thinking to set up something similar in your city, I am
happy to join forces and share thoughts and ideas. Don't doubt to reach out
to Griselda (in cc) as well - she has been a great help for us so far!

Best,
Matthias
--

Re: London Apache Beam meet up

[Matthias Baetens]

Awesome, JB, happy to hear you like the initiative! I will get in touch
with you offline to talk about the details :)
Matthias

On Fri, Sep 8, 2017 at 7:10 AM, Jean-Baptiste Onofré 
wrote:

> Cool !!!
>
> I would be happy to be there and give a talk if you want.
>
> Regards
> JB
>
> On 09/08/2017 12:01 AM, Matthias Baetens wrote:
>
>> Hi all,
>>
>> After a bit of delay I managed to schedule our first Apache Beam meet up <
>> https://www.meetup.com/London-Apache-Beam-Meetup/events/243123359/>, on
>> *October 3rd* here in *London*!
>>
>> I will try to build a community to discuss both use cases (and value to
>> the business) as well as do more technical deep dives and in general have a
>> conversation about Beam. Hope you like the initiative and the first few
>> speakers I managed to pull together.
>>
>> I'm very keen to hear about anyone who wants to *talk about Beam in one
>> of the future meet ups*, or who just wants to come over and connect with
>> your peers.
>> *Feedback on the agenda* (the topics and who is speaking) is also greatly
>> appreciated, any ideas for a next iteration are welcome.
>> Last but not least: I'd love to hear about any *channels to share this
>> *and get it out in the open (through social media or just your friends /
>> colleagues that may be interested) - the more people attend, the merrier.
>>
>> Cheers,
>> Matthias
>>
>>
>>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>



-- 


*Matthias Baetens*


*datatonic | data power unleashed*

office +44 203 668 3680  |  mobile +44 74 918 20646

Level24 | 1 Canada Square | Canary Wharf | E14 5AB London


We've been announced
<https://blog.google/topics/google-cloud/investing-vibrant-google-cloud-ecosystem-new-programs-and-partnerships/>
as
one of the top global Google Cloud Machine Learning partners.

London Apache Beam meet up

[Matthias Baetens]

Hi all,

After a bit of delay I managed to schedule our first Apache Beam meet up
,
on *October
3rd* here in *London*!

I will try to build a community to discuss both use cases (and value to the
business) as well as do more technical deep dives and in general have a
conversation about Beam. Hope you like the initiative and the first few
speakers I managed to pull together.

I'm very keen to hear about anyone who wants to *talk about Beam in one of
the future meet ups*, or who just wants to come over and connect with your
peers.
*Feedback on the agenda* (the topics and who is speaking) is also greatly
appreciated, any ideas for a next iteration are welcome.
Last but not least: I'd love to hear about any *channels to share this *and
get it out in the open (through social media or just your friends /
colleagues that may be interested) - the more people attend, the merrier.

Cheers,
Matthias

Re: ReadFromDatastore in the cloud

[Matthias Baetens]

Hi Vikas,

Thanks for your reply! I was able to run both as well, and I think I
figured out why it wasn't working: the project where I was staging was
different from the one I was reading datastore from - there was a warning
in the logs (saying PERMISSION_DENIED) while it failed with an error only
in the read/flatten-step.

But it's all working now, thanks for your help! :)

Matthias

On Fri, Jan 13, 2017 at 6:51 AM, Vikas Kedigehalli 
wrote:

> Hi Matthias,
>
>   Glad you are trying out the python beam sdk. The datastoreio should work
> on any runner.
>
> I was able to run the datastore wordcount example
> <https://github.com/apache/beam/blob/python-sdk/sdks/python/apache_beam/examples/cookbook/datastore_wordcount.py>
> successfully (both locally (DirectRunner) and on GCP). Here is the command,
>
> Locally -> "python -m apache_beam.examples.cookbook.datastore_wordcount
> --output gs:// --project '' --kind ''
> --read_only"
>
> GCP -> "python -m apache_beam.examples.cookbook.datastore_wordcount
>  --output gs:// --project '' --kind ''
> --read_only --staging_location gs:// --runner
> DataflowPipelineRunner --job_name "
>
> Did you get a chance to look at the job worker logs, for any errors that
> the pipeline is throwing? (that should give us a better idea).
>
> Regards,
> Vikas
>
> On Thu, Jan 12, 2017 at 7:39 AM, Matthias Baetens <
> matthias.baet...@datatonic.com> wrote:
>
>> Hi all,
>>
>> Using the Python SDK (the one installed using pip install
>> google-cloud-dataflow) I have implemented a very simple pipeline trying to
>> read from Datastore and print the result on a dataset with just 3 entities:
>>
>> entities = p \
>>   | 'read from datastore' >> 
>> ReadFromDatastore(project='project-name', query=ds_query) \
>>   | 'printing' >> beam.Map( lambda row : println(row) )
>>
>>
>> Running this locally, this seems to work fine. Running it on the cloud this 
>> results in the following graph:
>>
>> [image: Inline image 1]
>>
>> but the execution stops in the GroupByKey step after which the rest of
>> the pipeline fails. Anything that should be added code-wise to make this
>> working?
>> Or is this only working locally for now?
>>
>> Thanks :)
>>
>> Matthias
>>
>>

ReadFromDatastore in the cloud

[Matthias Baetens]

Hi all,

Using the Python SDK (the one installed using pip install
google-cloud-dataflow) I have implemented a very simple pipeline trying to
read from Datastore and print the result on a dataset with just 3 entities:

entities = p \
  | 'read from datastore' >>
ReadFromDatastore(project='project-name', query=ds_query) \
  | 'printing' >> beam.Map( lambda row : println(row) )


Running this locally, this seems to work fine. Running it on the cloud
this results in the following graph:

[image: Inline image 1]

but the execution stops in the GroupByKey step after which the rest of the
pipeline fails. Anything that should be added code-wise to make this
working?
Or is this only working locally for now?

Thanks :)

Matthias

Re: CombineWithContext

[Matthias Baetens]

Hi Kenneth,

Thanks a lot for the fast reply, works like a charm now :)

Best regards,

Matthias

On Tue, Jan 3, 2017 at 8:34 PM, Kenneth Knowles  wrote:

> Hi Matthias,
>
> It seems there is a bit of an API inconsistency here. ParDo has the
> .withSideInputs(Iterable>) version but also a
> varargs version withSideInputs(PCollectionView...) but Combine only
> has the Iterable version.
>
> So you have the right idea, and just need to put your PCollectionView into
> a trivial iterable to pass it to Combine's withSideInputs. Leaving out type
> annotations for readability:
>
> logs_keyed.apply(
> Combine.perKey(new CombineLogs(sideInput))
> .withSideInputs(Collections.singletonList(sideInput)));
>
> Just replying without checking with a compiler, so I hope this helps but
> let me know if you hit another snag.
>
> We'll get these APIs consistent.
>
> Kenn
>
> On Tue, Jan 3, 2017 at 11:29 AM, Matthias Baetens <
> matthias.baet...@datatonic.com> wrote:
>
>> Hi everyone,
>>
>> I have been trying to write a CombineFnWithContext; in the Context I want
>> to have a map containing values to filter the input values.
>>
>> If I'm correct, I can access the sideinput in the addInput(Accum accum,
>> Log input, Context c) operation using c.sideInput(). When constructing the
>> main flow of the pipeline though, I tried to do things similarly to what is
>> outlined here
>> <https://beam.apache.org/documentation/programming-guide/#transforms-sideio> 
>> for
>> ParDo's (creating a PCollectionView and specifying the side input using
>> .withSideInputs()) - but then an Iterable> is
>> required rather than a PCollectionView<>):
>>
>> PCollection> logs_aggregated =
>> logs_keyed.apply(Combine.perKey(new
>> CombineLogs(sideInput)).withSideInputs(sideInput));
>>
>> How should I go about this, because the syntax is confusing me and I was
>> not able to find a clear example floating around the internet.
>>
>> Thanks!
>>
>> Best regards,
>>
>> Matthias
>>
>
>


-- 

*Matthias Baetens*


*datatonic | data power unleashed*
office +44 203 668 3680  |  mobile +44 74 918 20646

Level24 | 1 Canada Square | Canary Wharf | E14 5AB London


Check out our latest blogposts (as featured on Google Cloud blog
<https://cloud.google.com/blog/big-data/2016/11/real-time-data-visualization-and-machine-learning-for-london-traffic-analysis>)
about processing London traffic data
<http://blog.datatonic.com/2016/10/traffic-in-london-episode-i-live.html> and
predicting congestion
<http://blog.datatonic.com/2016/11/traffic-in-london-episode-ii-predicting.html>
.

CombineWithContext

[Matthias Baetens]

Hi everyone,

I have been trying to write a CombineFnWithContext; in the Context I want
to have a map containing values to filter the input values.

If I'm correct, I can access the sideinput in the addInput(Accum accum, Log
input, Context c) operation using c.sideInput(). When constructing the main
flow of the pipeline though, I tried to do things similarly to what is
outlined here

for
ParDo's (creating a PCollectionView and specifying the side input using
.withSideInputs()) - but then an Iterable> is
required rather than a PCollectionView<>):

PCollection> logs_aggregated =
logs_keyed.apply(Combine.perKey(new
CombineLogs(sideInput)).withSideInputs(sideInput));

How should I go about this, because the syntax is confusing me and I was
not able to find a clear example floating around the internet.

Thanks!

Best regards,

Matthias