Re: Are there pipeline API's for ETL?

[kant kodali]

Hi Vino,

Another use case would be I want to build a dag of batch sources, sinks and 
transforms and I want to schedule the jobs periodically. One can say similar to 
airflow but a Flink api would be lot better!

Sent from my iPhone

> On Jan 10, 2020, at 6:42 PM, vino yang  wrote:
> 
> 
> Hi kant,
> 
> Can you provide more context about your question? What do you mean about 
> "pipeline API"?
> 
> IMO, you can build an ETL pipeline via composing several Flink transform 
> APIs. About choosing which transform APIs, it depends on your business logic. 
> 
> Here are the generic APIs list.[1]
> 
> Best,
> Vino
> 
> [1]: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html
> 
> kant kodali  于2020年1月11日周六 上午9:06写道：
>> Hi All,
>> 
>> I am wondering if there are pipeline API's for ETL?
>> 
>> Thanks!
>> 
>> 

Re: Are there pipeline API's for ETL?

[kant kodali]

Hi Vino,

I am new to Flink. I was thinking more like a dag builder api where I can build 
a dag of source,sink and transforms and hopefully fink take cares of the entire 
life cycle of the dag.

An example would be CDAP pipeline api.

Sent from my iPhone

> On Jan 10, 2020, at 6:42 PM, vino yang  wrote:
> 
> 
> Hi kant,
> 
> Can you provide more context about your question? What do you mean about 
> "pipeline API"?
> 
> IMO, you can build an ETL pipeline via composing several Flink transform 
> APIs. About choosing which transform APIs, it depends on your business logic. 
> 
> Here are the generic APIs list.[1]
> 
> Best,
> Vino
> 
> [1]: 
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html
> 
> kant kodali  于2020年1月11日周六 上午9:06写道：
>> Hi All,
>> 
>> I am wondering if there are pipeline API's for ETL?
>> 
>> Thanks!
>> 
>> 

Re: Are there pipeline API's for ETL?

[vino yang]

Hi kant,

Can you provide more context about your question? What do you mean about
"pipeline API"?

IMO, you can build an ETL pipeline via composing several Flink transform
APIs. About choosing which transform APIs, it depends on your business
logic.

Here are the generic APIs list.[1]

Best,
Vino

[1]:
https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/index.html

kant kodali  于2020年1月11日周六 上午9:06写道：

> Hi All,
>
> I am wondering if there are pipeline API's for ETL?
>
> Thanks!
>
>
>

Are there pipeline API's for ETL?

[kant kodali]

Hi All,

I am wondering if there are pipeline API's for ETL?

Thanks!