[
https://issues.apache.org/jira/browse/STORM-138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979447#comment-13979447
]
ASF GitHub Bot commented on STORM-138:
--------------------------------------
GitHub user jsgilmore opened a pull request:
https://github.com/apache/incubator-storm/pull/84
Pluggable serialization for multilang
This pull request addresses storm issue STORM-138
(https://issues.apache.org/jira/browse/STORM-138) and makes the Storm multilang
JSON protocol plugable. This pull request was originally reviewed and accepted
by James Xu.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jsgilmore/incubator-storm master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-storm/pull/84.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #84
----
commit 1a19f3ddaca0d2ab2c388d70d03d7ebd63590879
Author: John Gilmore <[email protected]>
Date: 2013-10-01T08:59:47Z
Shell component serialisation is now plugable with a default JSON
serialiser being used. Moved multilang classes to multilang directory.
commit a7918f8ef91b82b5b8fd91c91b36378a6c99caad
Author: John Gilmore <[email protected]>
Date: 2013-10-07T07:33:53Z
Moved ShellBolt, ShellProcess and ShellSpout back to their old packages.
Updated dependencies.
commit 3701bfcbe3ed90e1ddf73605a0e5cd0bf4f0dbd4
Author: John Gilmore <[email protected]>
Date: 2013-10-07T09:47:48Z
Added comments to the new multilang objects.
commit 3219a4f05f938f4ce5bdbb9f006e49cbec5ee89f
Author: John Gilmore <[email protected]>
Date: 2013-10-08T11:52:00Z
multilang serializer is not handled by stormConf. Renamed Emission and
Immission to ShellMsg and BoltMsg respectively. Documented ISerializer
interface.
commit ee0678c7f8bed531a8514166c7254a2190f3c4a9
Author: John Gilmore <[email protected]>
Date: 2013-10-08T11:52:00Z
multilang serializer is now handled by stormConf. Renamed Emission and
Immission to ShellMsg and BoltMsg respectively. Documented ISerializer
interface.
commit 06aad3f99d82bcc29c4a827e892d48aaf25e2fd6
Author: John Gilmore <[email protected]>
Date: 2013-10-08T11:53:59Z
Merge branch 'master' of github.com:jsgilmore/storm
commit 18c0ce778bb252e1f8b266d3d3b0a568d283257f
Author: John Gilmore <[email protected]>
Date: 2013-10-09T09:38:49Z
Merge branch 'master' of https://github.com/nathanmarz/storm
commit 52948f56511a57cd273431b794819ddc9a343bee
Author: John Gilmore <[email protected]>
Date: 2013-10-09T09:51:20Z
Added newline to NoOutputException
commit 63afb94cd0d8aa769898ba80ed2b020a6379b52f
Author: John Gilmore <[email protected]>
Date: 2013-10-09T13:29:21Z
Re-added need_task_ids option
commit 32b371857ddec5669f65fa779f6a0b098c1cb0a1
Author: John Gilmore <[email protected]>
Date: 2013-10-09T14:40:05Z
Added json msg and id parsing.
commit 5699373be3842310171aa9b51059d94c03dca82a
Author: John Gilmore <[email protected]>
Date: 2013-10-09T15:00:10Z
Allow for null boltMsg when polling
commit 16ba0c7fbba5b49ed4054f7aba7fc7e15e144c77
Author: John Gilmore <[email protected]>
Date: 2013-10-10T12:28:08Z
Removed pure whitespace changes of project.clj and bootstrap.clj
commit dfbd045966a9a909aaa0e7e31eac59639a86c3c9
Author: John Gilmore <[email protected]>
Date: 2013-10-10T12:58:51Z
Added some trailing whitespace to Config.java
commit 84360dd32c77b050e28bf4c8f3483ce393027c5e
Author: John Gilmore <[email protected]>
Date: 2013-11-07T12:46:37Z
Merge branch 'master' of https://github.com/nathanmarz/storm
commit 80304231ff550f74b99a88684b23031208bec919
Author: John Gilmore <[email protected]>
Date: 2013-11-18T14:30:44Z
Merge branch 'master' of https://github.com/nathanmarz/storm
commit 5ba093862560aa3ca2cff002e2e38e1e37753670
Author: John Gilmore <[email protected]>
Date: 2013-11-18T14:31:44Z
Added debugging info to ShellSpout and pulled in latest upstream master.
commit 8cde5508b5e4828982aa4f1ff2e70ba6fd10f4b9
Author: John Gilmore <[email protected]>
Date: 2013-11-29T12:47:31Z
Added null check for anchors. Log a component's error stream under it name.
commit 46c02be8181d196e48484807e34215d5e5fb61b4
Author: John Gilmore <[email protected]>
Date: 2014-04-23T07:12:01Z
Merge branch 'master' of https://github.com/nathanmarz/storm
Conflicts:
conf/defaults.yaml
commit f5f41a0b54a852da95a2fa659b1199d727571313
Author: John Gilmore <[email protected]>
Date: 2014-04-23T07:27:39Z
Merge branch 'master' of github.com:jsgilmore/incubator-storm
----
> Pluggable serialization for multilang
> -------------------------------------
>
> Key: STORM-138
> URL: https://issues.apache.org/jira/browse/STORM-138
> Project: Apache Storm (Incubating)
> Issue Type: New Feature
> Reporter: James Xu
> Priority: Minor
>
> https://github.com/nathanmarz/storm/issues/373
> Currently JSON is used to serialize tuples for multilang. It would be great
> if the serialization mechanism were pluggable so that using richer types with
> multilang would be possible.
> ---------
> francis-liberty: Hello, I am a newbie here, and I wanted to pick up this
> issue. I also noticed a recent PR here #697 by jsgilmore, is it feasible for
> this issue, too?
> I looked around the source code, and I would like to talk about my opinions
> on this issue here.
> For now, ShellProcess only supports JSON to communicate with multilang
> process: read, write. And, ShellSpout and ShellBolt talk with ShellProcess
> through JSON, too. This is all because ShellProcess's interface use
> JSONObject only. Conceptually, ShellProcess should encapsulate the multilang
> details, and talk with Bolt and Spout using Tuple. (jsgilmore invented two
> new classes, Immission and Emission. But I think all information Bolt and
> Spout need is in Tuple already, no need for new data structures.) So, I think
> it would be much cleaner to do serialization in ShellProcess only, and both
> ShellSpout and ShellBolt don't know anything about how ShellProcess convert
> between Tuple and strings.
> So, I suppose I can do the work of
> 1. change the interface of ShellProcess to return and accept Tuple data
> structure, instead of JSONObject.
> 2. make ShellSpout and ShellBolt work on Tuple, all information like task_id,
> stream_id and tuples should be retrieve/encapsulate in this data structure.
> 3. what other serialization format would you like to add? I think in the end
> we need to add some example other than JSON to storm-starter storm.py/rb,
> which I would also like to work on.
> ----------
> jsgilmore: Hi, all serialisation is done in the JSONSerialiser, so no
> serialisation is done in ShellBolt, ShellProcess or ShellSpout. They just
> send around the Emission and Immission classes. The point of the ISerializer
> interface is to achieve the separation of serialisation.
> I come from the multilang side of Storm, so I'm not that familiar with the
> internal Storm structures. If there is a class that the ISerializer interface
> can use, instead of the Emission and Immission classes, I'm open to it.
> I would recommend that further discussion of PR #697 rather happen in the PR
> thread itself though.
> I created an issue to add protocol buffer serialisation for multilang to
> Storm in issue #654 , but I didn't see this issue. The whole purpose of PR
> #697 is to solve this issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
