Re: Retrieve written records of a sink after job

Hi Fabian, thanks for the feedback. Right now I'm doing exactly as you said. Since I was seeing this as a useful API extension I just proposed this addition and so I asked for feedbacks. Of course, it doesn't make much sense if I'm the only one asking for it :) Best, Flavio On Mon, Feb 19, 2018

Re: Retrieve written records of a sink after job

Hi Flavio, Not sure if I would add this functionality to the sinks. You could also add a MapFunction with a counting Accumulator right before each sink. Best, Fabian 2018-02-14 14:11 GMT+01:00 Flavio Pompermaier : > So, if I'm not wrong, the right way to do this is using

Re: Retrieve written records of a sink after job

So, if I'm not wrong, the right way to do this is using accumulators..what do you think about my proposal to add an easy way to add to a sink an accumulator for the written/outputed records? On Wed, Feb 14, 2018 at 1:08 PM, Chesnay Schepler wrote: > Technically yes, a subset

Re: Retrieve written records of a sink after job

Technically yes, a subset of metrics is stored in the ExecutionGraph when the job finishes. (This is for example where the webUI derives the values from for finished jobs). However these are on the task level, and will not contain the number of incoming records if your sink is chained to

Re: Retrieve written records of a sink after job

The problem here is that I don't know the vertex id of the sink..would it be possible to access the sink info by id? And couldn't be all those info attached to the JobExecutionResult (avoiding to set up all the rest connection etc)? On Wed, Feb 14, 2018 at 12:44 PM, Chesnay Schepler

Re: Retrieve written records of a sink after job

The only way to access this info from the client is the REST API or the Metrics REST API

Re: Retrieve written records of a sink after job

Actually I'd like to get this number from my Java class in order to update some external dataset "catalog", so I'm asking if there's some programmatic way to access this info (from JobExecutionResult for example). On Wed, Feb 14, 2018 at 12:25 PM, Chesnay Schepler wrote: >

Re: Retrieve written records of a sink after job

Do you want to know how many records the sink received, or how many the sink wrote to the DB? If it's the first you're in luck because we measure that already, check out the metrics documentation. If it's the latter, then this issue is essentially covered by FLINK-7286 which aims at allowing