Re: Bay Area Apache Beam Kickoff!

[Sobhan Badiozamany]

Would this meetup be recorded, or better yet, is it broadcasted online?

Otherwise, it would be great if you share the slides afterwards.

Thanks,
Sobi

On Tue, Nov 20, 2018 at 2:37 AM Austin Bennett 
wrote:

> We have our first meetup scheduled for December 12th in San Francisco.
>
> Andrew Pilloud, a software engineer at Google and Beam committer, will
> demo the latest feature in Beam SQL: a standalone SQL shell. The talk cover
> why SQL is a good fit for streaming data processing, the technical details
> of the Beam SQL engine, and a peek into our future plans.
>
> Kenn Knowles, a founding PMC Member and incoming PMC Chair for the Apache
> Beam project, as well as computer scientist and engineer at Google will
> share about all things Beam. Where it is, where its been, where its going.
>
> More info:
> https://www.meetup.com/San-Francisco-Apache-Beam/events/256348972/
>
> For those in/around town (or that can be) come join in the fun!
>
>
>
>
>

-- 


*Sobhan Badiozamany*
*Data Architect*
*Gears of Leo*M: +46704094664
E: sobhan.badiozam...@leovegas.com

*This email and any files transmitted with it, are confidential, may be
legally privileged and are intended solely for the use of the individual or
entity to whom they are addressed. Any views or opinions expressed are
solely those of the sender and do not necessarily represent those of the
company. Copyright of this email and any attachments belongs to LeoVegas
Mobile Gaming Group. If this email was sent to you in error, please notify
the sender and delete it from your system.*

Re: Is it possible to run a perl scrip in Dataflow worker?

[Sobhan Badiozamany]

Hi Nima,

I think the answer depends on the use-case, but what you suggest is on the list 
of practices that hurt scalability of pipelines as it will be an example of 
“Expensive Per-Record Operations”, look it up here:
https://cloud.google.com/blog/products/gcp/writing-dataflow-pipelines-with-scalability-in-mind

Cheers,
Sobi

Sent from my iPhone

> On Oct 23, 2018, at 23:35, Nima Mousavi  wrote:
> 
> Hi, 
> 
> We have a dataflow pipeline written in Apache python beam, and are wondering 
> if we can run a third party code (written in perl) in the pipeline. We 
> basically want to run 
> 
> perl myscript.pl $DATA 
> 
> for every DATA in a PCollection passed to a DoFn
> 
> and write the result back into Bigquery.  We could have setup a server for 
> myscript.pl, and send HTTP/RPC request to the server from each worker 
> instead. But we are wondering if it is possible to run the script directly 
> inside the Beam worker? Or even through a docker container packaging our perl 
> script? If yes, how? what do you think of this approach? Any caveat we should 
> be aware of?
> 
> Thanks!

Slack Channel

[Sobhan Badiozamany]

Hi,

Could you please add me to the slack channel?

Thanks,
Sobi