[
https://issues.apache.org/jira/browse/STORM-42?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rick Kellogg updated STORM-42:
------------------------------
Component/s: storm-core
> Support sending arbitrary data types back and forth through DRPC
> ----------------------------------------------------------------
>
> Key: STORM-42
> URL: https://issues.apache.org/jira/browse/STORM-42
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: James Xu
>
> https://github.com/nathanmarz/storm/issues/41
> Right now, you can only send Strings back and forth due to the limitations of
> Thrift. It would be nice if DRPC were structured in a way so that you could
> put any interface you want on it. So the Thrift interface would be one
> implementation, and another implementation could just send Java objects back
> and forth (perhaps by using Kryo).
> -----------------------------------------------------------------------------------------------------
> thatsnotright: Why would you define the DRPCRequest to have function args as
> a string? What is the limitation I am not seeing of Thrift? There already
> exists kryo and java serialization support, why not define func_args to be
> binary and serialize in to that?
> -----------------------------------------------------------------------------------------------------
> dmk23: So are there any specific plans to make this feature happen? Thrift
> obviously can be used to define objects of arbitrary complexity and this
> would also allow for cross-language DRPC calls.
> We currently have a set of custom Thrift services written in Java that are
> called from Python clients. We'd be interested in converting our Java backend
> into Storm topologies but RPC data type limitation is a significant barrier,
> especially given that this was to work across languages.
> -----------------------------------------------------------------------------------------------------
> nathanmarz: Yea, the right solution to this is to just do binary inputs and
> outputs through DRPC. Actually it should be something like:
> list<list<binary>> execute(string function, binary args)
> Then you can do the serialization/deserialization around that. The return
> type there is because Trident returns a list of tuples for the output of
> drpc, where each tuple is a list of values. Each value is in serialized form
> here.
> -----------------------------------------------------------------------------------------------------
> dmk23: I understand anything can be manually serialized / deserialized and
> pushed through the existing "execute" call.
> But this looks quite clumsy / redundant since these DRPC service definitions
> could be easily expressed with Thrift, then converted into easy-to-use
> generated code in any language and imported into Storm through Java. Given
> that Thrift is already used within Storm this seems like an obvious solution.
> Would be nice to have support for having Storm import arbitrary Thrfit
> service definition and have DRPC services natively expose interfaces / data
> structures as defined in those IDLs...
> -----------------------------------------------------------------------------------------------------
> suchetchachra:A temporary workaround for the string size limitation (64K)
> could be to save the object to be returned to a distributed cache and return
> the key (String). On the client side, the key can be used to retrieve the
> object from the cache.
> -----------------------------------------------------------------------------------------------------
> dimazaur: Hello, Nathan.
> Is there any plans about drpc binary support?
> We are thinking about move to Storm system on our servers.
> And this String type limitation is blocking us.
> Thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
