Re: Common model for runners

[Robert Bradshaw]

This is all present in the master branch, the WIP nature is that as we
build atop it, we may discover that changes to the proto representation are
necessary.


On Sat, Mar 24, 2018 at 6:26 AM zlgonzalez <zlgonza...@yahoo.com> wrote:

> Thanks Robert. I'll take a look at this for now.
>
> What WIP branch is involved with creating a language independent
> representation of the data flow?
>
> Thanks,
> Ron
>
>
>
> Sent via the Samsung Galaxy S7 active, an AT 4G LTE smartphone
>
>  Original message 
> From: Robert Bradshaw <rober...@google.com>
> Date: 3/21/18 12:18 AM (GMT+02:00)
> To: dev@beam.apache.org
> Subject: Re: Common model for runners
>
> The proto representation isn't (yet) part of the public API, and is still
> under active development. However, if you're curious you can see it via
> calling
>
> pipeline.to_runner_api()
>
> in Python or manually invoking classes under
>
>
> https://github.com/apache/beam/tree/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction
>
> for Java. There's probably some equivalent for Go.
>
> (Eventually, the way to get at this would be to implement a Runner whose
> input would be the proto description itself, but currently both Python and
> Java represent Pipelines as native objects when invoking runners.)
>
>
>
> On Tue, Mar 20, 2018 at 2:58 PM Ron Gonzalez <zlgonza...@yahoo.com> wrote:
>
>> Hi,
>>   When I build a data flow using the Beam SDK, can someone point me to
>> the code that represents the underlying representation of the beam model
>> itself?
>>   Is there an API that lets me retrieve the underlying protobuf-based
>> graph for the data flow? Perhaps some pointers to what code in the runner
>> retrieves this model in order to execute it in the specific engine?
>>
>> Thanks,
>> Ron
>>
>

Re: Common model for runners

[zlgonzalez]

Thanks Robert. I'll take a look at this for now.
What WIP branch is involved with creating a language independent representation 
of the data flow?
Thanks,Ron


Sent via the Samsung Galaxy S7 active, an AT 4G LTE smartphone
 Original message From: Robert Bradshaw <rober...@google.com> 
Date: 3/21/18  12:18 AM  (GMT+02:00) To: dev@beam.apache.org Subject: Re: 
Common model for runners 
The proto representation isn't (yet) part of the public API, and is still under 
active development. However, if you're curious you can see it via calling
    pipeline.to_runner_api()

in Python or manually invoking classes under
    
https://github.com/apache/beam/tree/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction

for Java. There's probably some equivalent for Go. 
(Eventually, the way to get at this would be to implement a Runner whose input 
would be the proto description itself, but currently both Python and Java 
represent Pipelines as native objects when invoking runners.)


On Tue, Mar 20, 2018 at 2:58 PM Ron Gonzalez <zlgonza...@yahoo.com> wrote:
Hi,  When I build a data flow using the Beam SDK, can someone point me to the 
code that represents the underlying representation of the beam model itself?  
Is there an API that lets me retrieve the underlying protobuf-based graph for 
the data flow? Perhaps some pointers to what code in the runner retrieves this 
model in order to execute it in the specific engine?
Thanks,Ron

Re: Common model for runners

[Henning Rohde]

Go currently prints out the model pipeline (as well as the Dataflow
representation) if you use the Dataflow runner. Pass --dry_run=true to not
actually submit a job, but just print out the representations. The graphx
package can also be used to generate a model pipeline manually.


On Tue, Mar 20, 2018 at 3:19 PM Robert Bradshaw  wrote:

> The proto representation isn't (yet) part of the public API, and is still
> under active development. However, if you're curious you can see it via
> calling
>
> pipeline.to_runner_api()
>
> in Python or manually invoking classes under
>
>
> https://github.com/apache/beam/tree/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction
>
> for Java. There's probably some equivalent for Go.
>
> (Eventually, the way to get at this would be to implement a Runner whose
> input would be the proto description itself, but currently both Python and
> Java represent Pipelines as native objects when invoking runners.)
>
>
>
> On Tue, Mar 20, 2018 at 2:58 PM Ron Gonzalez  wrote:
>
>> Hi,
>>   When I build a data flow using the Beam SDK, can someone point me to
>> the code that represents the underlying representation of the beam model
>> itself?
>>   Is there an API that lets me retrieve the underlying protobuf-based
>> graph for the data flow? Perhaps some pointers to what code in the runner
>> retrieves this model in order to execute it in the specific engine?
>>
>> Thanks,
>> Ron
>>
>

Re: Common model for runners

[Robert Bradshaw]

The proto representation isn't (yet) part of the public API, and is still
under active development. However, if you're curious you can see it via
calling

pipeline.to_runner_api()

in Python or manually invoking classes under


https://github.com/apache/beam/tree/master/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction

for Java. There's probably some equivalent for Go.

(Eventually, the way to get at this would be to implement a Runner whose
input would be the proto description itself, but currently both Python and
Java represent Pipelines as native objects when invoking runners.)



On Tue, Mar 20, 2018 at 2:58 PM Ron Gonzalez  wrote:

> Hi,
>   When I build a data flow using the Beam SDK, can someone point me to the
> code that represents the underlying representation of the beam model itself?
>   Is there an API that lets me retrieve the underlying protobuf-based
> graph for the data flow? Perhaps some pointers to what code in the runner
> retrieves this model in order to execute it in the specific engine?
>
> Thanks,
> Ron
>

Common model for runners

[Ron Gonzalez]

Hi,  When I build a data flow using the Beam SDK, can someone point me to the 
code that represents the underlying representation of the beam model itself?  
Is there an API that lets me retrieve the underlying protobuf-based graph for 
the data flow? Perhaps some pointers to what code in the runner retrieves this 
model in order to execute it in the specific engine?
Thanks,Ron