Hi Stephen,
Great work ! I will take a look during the week end and will provide
feedbacks.
Thanks !
Regards
JB
On 01/28/2017 02:29 AM, Stephen Sisk wrote:
Here's the doc I've been working on:
https://docs.google.com/document/d/1nGGP2sLb5fLamB_dnkHVHC8BVjDD_SE46mQPIPkK5cQ/edit?usp=sharing
The general purpose of the doc is to share high level design thoughts and
process for authoring an IO transform. Topics include:
* List of example IOs that people can refer to, and for what reasons
* Steps for implementers
* Read transform design
* Write transform design (mostly a TODO for now)
Specifics of implementations, notably including the two items that started
recent discussions of the IO Authoring doc (the service interface and the
fake/mock pattern), are probably best covered by the transform style guide
that Eugene's working on and the "testing IO transforms" doc[1].
This is definitely a first draft. There is a lot that could be written on
the topic, and I have struggled a bit with how much detail to put in, but
since the jira issue has been open a while, this is a simple set of things
that I thought wouldn't be too controversial, and that would be forward
compatible with SplittableDoFn coming (thus, spending too much time
discussing sources isn't useful.)
I look forward to contributions from folks like Eugene, JB, and Dan who
have a lot more experience in this area. I'd like to get something up on
the website sooner rather than later, so if folks want to add large
chunks/sections, I might push us towards getting up a smaller, correct
version in a first PR, and folks can add more sections/design
patterns/etc... in later PRs.
Thanks!
Stephen
[1] Testing IO transforms -
https://docs.google.com/document/d/153J9jPQhMCNi_eBzJfhAg-NprQ7vbf1jNVRgdqeEE8I/edit?usp=sharing
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
