[
https://issues.apache.org/jira/browse/PIG-3453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13972242#comment-13972242
]
Mridul Jain commented on PIG-3453:
----------------------------------
We completed the implementation for "Pig on Storm" last month and are now
testing it in production so that we could potentially opensource it. Here is an
abstract:
“We propose PIG as the primary language for expressing realtime stream
processing logic and provide a working prototype on Storm. We also illustrate
how legacy code written for MR in PIG, can run with minimal to no changes, on
Storm. This includes running the existing PIG UDFs, seamlessly on Storm. Though
PIG or Storm do not take any position on state, we have provided built-in
support for advanced state semantics like sliding windows, global mutable state
etc, which are required in real world applications. We take a detailed look
into a prototype application (realtime anomaly detection/trending system),
which elucidates the performance characteristics of this framework and rich
expressibility of complex programming logic via PIG on streaming.
Finally, we propose a "Hybrid Mode" where a single PIG script can express logic
for both realtime streaming and batch jobs and also defines data exchange
mechanisms between the two, without breaking the semantic & syntactic sanctity
of PIG. The underlying system figures out what parts of the this PIG script to
run on MR and what on Storm, automatically."
> Implement a Storm backend to Pig
> --------------------------------
>
> Key: PIG-3453
> URL: https://issues.apache.org/jira/browse/PIG-3453
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.13.0
> Reporter: Pradeep Gollakota
> Assignee: Jacob Perkins
> Labels: storm
> Fix For: 0.13.0
>
> Attachments: storm-integration.patch
>
>
> There is a lot of interest around implementing a Storm backend to Pig for
> streaming processing. The proposal and initial discussions can be found at
> https://cwiki.apache.org/confluence/display/PIG/Pig+on+Storm+Proposal
--
This message was sent by Atlassian JIRA
(v6.2#6252)
