That does appear to be the case after being pointed at the code in
Flume. Biting off implementation of a protocol was a bit more than I
felt I could manage, especially in Node.js, a language I am very new
too. I really have no idea how complicated that would be, could be that
it is not as bad as I thought. I've been coding for a number of years
in a number of other languages, but this is the first project in Node.js
that I have worked on.
JensG on stack overflow found this for me:
http://stackoverflow.com/questions/23377767/node-js-to-flume-ng/23383545?noredirect=1#comment35920812_23383545
https://issues.apache.org/jira/browse/FLUME-1894
| args.protocolFactory(new TCompactProtocol.Factory());
args.inputTransportFactory(new TFastFramedTransport.Factory());
args.outputTransportFactory(new TFastFramedTransport.Factory());
args.processor(new ThriftSourceProtocol.Processor<ThriftSourceHandler>(new
ThriftSourceHandler()));|
From the Java Flume source, ThriftSource.java. The FastFramedTransport
is wire compatible with Framed from what I understand, so that is not an
issue. However, Compact and Binary do not seem to be compatible.
I was getting very hopeful once I was successfully communicating with
blocking code, I didn't think the step to async would be that bad, I
assumed the framework would do the heavy lifting behind the scenes and
allow me to just register callbacks.
Sounds like that may not be the case.
How difficult do you expect it would be to implement TCompact in JS? I
do like the idea of native support, and am happy to do it if it is
something I can do in a reasonable time frame.
Aaron
On 5/3/14, 12:36 PM, Randy Abernethy wrote:
Hey Aaron,
So is it true that Flume-NG is requiring compact protocol for Thrift
clients?
If that is so I think the easiest way forward might be to add the compact
protocol to the Node.js library in Node directly. Particularly because the
Node.js end point code is not designed like the end point layers in other
supported languages (the async bit being the key factor).
I would be happy to help you get this sorted either way.
-Randy
On Sat, May 3, 2014 at 9:33 AM, Aaron Mefford <[email protected]> wrote:
I am trying to get communication setup between Node.JS and Flume-NG over
thrift. I seem to keep running up against brick walls in the process
however. The first of those was that all of the existing projects that
seemed to do this were seemingly abandoned about 3 years ago. The next
problem came when I found that the javascript stubs for Thrift do not
support the CompactProtocol.
I finally settled on needing to write a C++ extension for node, and have
proceeded in that vein. I have now made it to the point where I can send a
message from javascript to Flume-NG using thrift in synchronous mode.
However as Node.Js needs non-blocking IO I now need to take the next step
and implement it with callbacks in async mode. I was at a complete loss
yesterday until I found one lone web post referring to the ability to
request the stubs be built with cob support, which I have now done and feel
like I am soo close, but I cannot see how to construct the objects required
to do so. There seems to be absolutely no documentation or code samples
available for this on the web, at least according to the Google. I assume
the archives of this list are included in Google, and as such have not
specifically consulted them. If my question is largely answered within
those, I will find another way to search them.
At this point it looks like my stub constructor needs and TAsyncChannel,
and a TProtocolFactory for it's contstructor. While I think I can see how
to get the TProtocolFactory, I am not sure how to provide a Factory that
will contstruct Protocols with the proper TTransport, TFramedTransport.
With the TAsyncChannel it seems all the classes provided in the
distribution are virtual, and there was not one generated from the .thrift
file in the stubs. Is it expected for the user to implement the
TAsyncChannel?
Can anyone point me in the right direction on this? I feel like I am so
close, but just missing something, it has been a very long week trying to
solve this communication need.
Aaron
