Hi Ben,
thanks for the update.
Correct me if I'm wrong: you are proposing kind of monitoring of the
pipelines, and be able to trace what's going on during the execution of
the pipeline.
It's a very important feature, especially for stream (and data integration).
The document greatly describes the data display. Do you have any plan to
implement kind of "checkpoint"/alerting depending of some predicates on
the data (it's something that I had in mind for the Beam data
integration DSL) ? It's maybe the TRIGGER type ?
Thanks again, great document and idea.
Regards
JB
On 03/15/2016 01:25 AM, Ben Chambers wrote:
Hi!
The following document describes work that we're planning on doing to allow
every steps in a pipeline to include more information about what is
actually going on. The goal is to allow UIs and diagnostic tools to display
details about what is happening inside each step by including details what
is otherwise just serialized blobs of code.
Everyone should be able to comment on the following link:
https://docs.google.com/document/d/11enEB9JwVp6vO0uOYYTMYTGkr3TdNfELwWqoiUg5ZxM/edit?usp=sharing
We will be creating a Jira issue to track the implementation of the
associated SDK changes.
Thanks,
Ben
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
