No one else has commented on this, so even though it seems like a great project
I guess there is not enough support from committers at this time. Please do
check back in again. I really want to see how things go with this, and
hopefully we can pull this in, in the future.
- Bobby
On Tuesday, January 5, 2016 12:52 PM, Adrianos Dadis <add...@gmail.com>
wrote:
Hi Bobby and all,
thanks for your interest and your comments (even if some seems negative :)
).
About project age (7 days). I use Storm in various use cases and I know it
(from user side) pretty well (I guess). For Morphlines, I use it very rare,
but I always use it with Apache Flume, which is already implemented as
Interceptor. So, I decided to implement a simple maven module that will
help me in future use cases and might help other people too. Configurable
ELT could be achieved with Flume too, but you loose the distributed nature
that Storm provides, so I decide to build something on top of Storm. It was
very easy and as I liked the result I suggest it here, if community likes
the idea to fork and make a formal pull request.
About community reputation of my project. This small project was
implemented not to be a single github repository, but as a reference to
integrate Storm with Morphlines. That's why I make it a simple maven module
instead of a github repository. So, I guess, it will never gain a big
reputation because is just a small maven module of a larger maven project
(that I will try to extend with various useful examples with Storm and
other frameworks).
About TODOs. I have left it there because these are open issues and I do
not like to hide them. I have already check other external projects too and
seems that some of them (check hive) might have open issues too (from what
I understand). If this project will be integrated in Storm, then these
TODOs might removed from code, but then they will be opened as JIRA bugs,
so we will not loose tricky points. In my current project, there is not
issue tracker, so I left them as TODOs as first approach. Also, community
might have better ideas than me of how to solve these potential issues.
I agree that the most important issue is about community adoption and
maintenance. I can surely help providing fixes or documentation on this,
but I also believe that others committers/PMC members opinion and help is
very crucial too. So, let people decide :)
If you believe that we have to ask "user" emailing list too, then just
inform me to send the same concept there too (including any existing
replies).
Sorry to all for the long response. I do not want to push things or any
like this, I just want to express all thoughts in my mind.
Regards,
Adrianos Dadis.
On Tue, Jan 5, 2016 at 12:43 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> Adrianos,
> I have never really taken a look at morphlines before. It looks really
> interesting, and I think it would be great to have official integration for
> it as a part of storm (especially like you said if we can do something
> combining it and flux). My only real concern would be around community
> adoption and maintenance, besides the TODOs that seem to be left in the
> code :). Integrating it with storm really requires people who are going to
> maintain it and provide updates/bug fixes for it. I personally don't feel
> like I can commit to that at this time, but I would like to hear from
> others, especially other committers/PMC members, if this is something that
> they can get behind and support. The project is only 7 days old so it
> hasn't had time to build up a community.
> - Bobby
>
> On Monday, January 4, 2016 11:24 AM, Adrianos Dadis <add...@gmail.com>
> wrote:
>
>
> Hi all,
>
> I have built a simple project that integrates Kite SDK Morphlines with
> Storm.
> The main main concept is to give to user the oportunity to create a simple
> topology that can run configurable ETL.
>
> Details:
> - source code:
>
> https://github.com/qiozas/sourcevirtues-samples/tree/master/sv-etl-storm-morphlines
> - blog post:
>
> http://sourcevirtues.com/2016/01/04/configurable-etl-processing-using-apache-storm-and-kite-sdk-morphlines/
> - Morphlines: http://kitesdk.org/docs/current/morphlines/
>
> Main Bolt is MorphlinesBolt. There are 3 main configurable sections:
> a) Morphlines configuration file (could also include java code or not).
> b) Mapper of incoming Tuple to Morphlines execution input.
> c) (Optional) Mapper of Morphlines output to new Tuple.
>
> If you find this implementation useful, then I can try to fork and pull
> this code in your repository according to your contribution procedure.
>
> Current implementation is not final and you might find bugs or things that
> need changes, but I will try it. Trident approach is not yet implemented,
> as I would like to first implement a stable version of pure Storm and then
> add a new User Story for Trident users.
>
> I find configurable ETL idea very useful, so please tell me if like this
> concept or not, in order to contribute this code :)
> Using Morphlines and Flux seems a valid approach for configurable ETL.
>
> As this is my first email in dev emailing list, I would like to say great
> thanks to all of you for this great streaming engine. I think Storm is
> great because of its simplicity. You can built too many concepts using this
> generic engine. Great job!!!
>
> Regards,
> Adrianos Dadis.
>
>
>
>
