RE: Ordered Map in Thrift
This doesn't exist as a Thrift feature. You'll need to separately maintain a list to preserve ordering using Thrift off-the-shelf. Another route to take would be to use some of the annotations to tell Thrift what data-types to use for the map type. If you provide container types that implement ordering, they can be swapped out. I don't recall off the top of my head what the syntax is on these type annotations or how deep the support is -- perhaps someone else can chime in there, or you might look through the implementation. -Original Message- From: Dheeraj Goswami [mailto:dheerajgosw...@yahoo.com] Sent: Monday, January 31, 2011 6:19 PM To: user@thrift.apache.org Subject: Ordered Map in Thrift Hi,I have the requirement to use Map but keep it ordered (Like LinkedHashMap in java). Is there a way to define thrift IDL to generate the ordered map in Java and C++ languages? thanks, --dheeraj
RE: How to use TBufferedTransport in Java ?
Also will note that part of the reason TBufferedTransport isn't generally necessary in Java is that by default your Java VM will typically already have a stage of output buffering on its Socket implementation. Wrapping that up in a TBufferedTransport would just burn up CPU/memory to no advantage. The buffering/streaming that the white paper describes isn't actually beneficial in practice for typical moderate-payload RPC use cases, which is what most people use Thrift for. The goal there was to not make it *impossible* to do those things with Thrift, but if you're just issuing basic RPCs, the framed transport makes things a lot easier to debug as each payload is sent over the wire with a frame header containing its size. -Original Message- From: Shrinath M [mailto:shrinat...@webyog.com] Sent: Wednesday, March 23, 2011 9:21 PM To: user@thrift.apache.org Cc: Bryan Duxbury Subject: Re: How to use TBufferedTransport in Java ? Why would you recommend Framed Transport ? Isn't that the opposite to whats described in section 4.2 in the white paper ? And can you show me an example usage of TIOStreamTransport on Java ? I didn't see one proper example anywhere... I've doubts like should we create a socket ? Can we have multiple buffers accessing the different streams like we can do on Framed Transport ? On Wed, Mar 23, 2011 at 10:57 PM, Bryan Duxbury br...@rapleaf.com wrote: You are correct - there is no BufferedTransport in Java. This is because we have TIOStreamTransport, which can be wrapped around a Java Buffer(Input|Output)Stream to raw buffering if that's what you need. I would strongly encourage you to use Framed Transport (on both sides) instead. On Wed, Mar 23, 2011 at 4:26 AM, Shrinath M shrinat...@webyog.com wrote: I am unable to find any way to get the TBufferedTransport working in Java. I also read somewhere http://www.facebook.com/topic.php?uid=15325934266topic=9390 that its missing totally from Java implementation of thrift. If so, what is the alternative ? -- Regards Shrinath.M -- Regards Shrinath.M
RE: Version 0.6.0 on FreeBSD doesn't build the shared.library
I've seen a similar issue where the boost path was wrong, or boost wasn't installed. That doesn't seem to show up in your error log, but it may be a similar issue where a library dependency is causing the --with-cpp to be ignored. I'd double-check just to be sure here, not sure what to recommend beyond that other than digging through the log. Do you have the command-line output of running configure available? config.log looks a bit noisier, might be easier to compare the two with a plain old stdout trace. -Original Message- From: Toomas Aas [mailto:toomas@raad.tartu.ee] Sent: Tuesday, April 26, 2011 1:50 PM To: Toomas Aas Cc: user@thrift.apache.org Subject: Re: Version 0.6.0 on FreeBSD doesn't build the shared.library Help, please... I've been stuck on this for a week. Trying to compile Thrift 0.6.0 on FreeBSD 8.1, and ./configure refuses to build the C++ library even if specifically told to do so with the --with-cpp switch. I also tried with the freshly-released 0.6.1, but the result is the same - configure script decides not to build the C++ library. From the JIRA pages I see that at least some people have managed to build Thrift on FreeBSD not too long ago, so it must be possible. I tried to compare the config.log output of (successful) 0.4.0 build with the unsuccessful 0.6.x build, but both logs contain a lot of errors and it's really hard to tell which of them are significant. My latest config.log output is here: ftp://ftp.raad.tartu.ee/pub/it/toomas/config.log -- Toomas Aas
RE: use thrift to create an interface from ruby to C++ and use it as webservice
This is certainly doable with Thrift. Technically, it won't be a web service, as Thrift doesn't offer and off-the-shelf C++ HTTP server implementation. You would have to embed Thrift in another server if you want HTTP. It's worth noting that the C++ servers we have for Thrift are not really designed to be used on the open internet. They tend to assume that you are in a protected intranet environment, behind a firewall. If this is just for a thesis project and not something you realistically need to worry about being defended against potentially malicious internet traffic, Thrift should be able to get you off the ground running very quickly. Cheers, mcslee -Original Message- From: Matias Hernandez Arellano [mailto:msd...@archlinux.cl] Sent: Tuesday, April 26, 2011 2:51 PM To: user@thrift.apache.org Subject: use thrift to create an interface from ruby to C++ and use it as webservice (Sorry for my english) Hi!, this is my first attempt to use Thrift, i arrive from a stackoverflow answer. First i tell about the idea/problem next the question. I have an application created with C++ that takes an image and make some processing with this, then return a result. This have to be donde with C++ for performance purpose. The application run with OpenCV, armadillo and FANN, the return value can be True or False (1 or 0). The idea is to have an application running on a mobile device (or any other device) and take a picture and send to a server where the C++ application take it, process and return a response. The mobile device have to able to send the image and wait for the response to take and action. The first idea was create a webservice to expose the C++ functionality, i thought about rails, but i can't find a way to solve this problem, so finally i found Thrift. The question. It's possible to use Thrift for this? My idea: Create a Server written in C++ to run the processing application (create this server with Thrift). And create, with thrift, the code to use this server, and use this code to make a implementation in the mobile device. The application is for my thesis for Computer Science. Thanks in advance. -- Matías Hernandez Arellano Ingeniero de Software/Proyectos en VisionLabs S.A CDA Archlinux-CL www.msdark.archlinux.cl
RE: use thrift to create an interface from ruby to C++ and use it as webservice
That should work fine, though it'll require you to write a bit more plumbing code for passing the image and file data around. Assuming the ruby/php webserver just takes care of this for you (which most of them do), it's probably not a bad approach. Thrift will make communicating between the ruby/php web process and a C++ server very easy. -Original Message- From: Matias Hernandez Arellano [mailto:msd...@archlinux.cl] Sent: Tuesday, April 26, 2011 4:36 PM To: user@thrift.apache.org Subject: Re: use thrift to create an interface from ruby to C++ and use it as webservice Thanks for your response... So, i get another idea: create a php/ruby interface to use the GET http method to specify the image that need to upload .. (http://myserver.dom/upload.php?file=/dir/to/image/image.jpeg i know this is not secure but this will work i guess) and this interface communicate with the C++ application (a thrift server) to make the calculations and then send a response to the device ... Am i right?? thrift can do the communication between a web client and C++ so i think this won't be a problem (the idea).. Thanks a lot for your time. El 26-04-2011, a las 18:17, Mark Slee escribió: This is certainly doable with Thrift. Technically, it won't be a web service, as Thrift doesn't offer and off-the-shelf C++ HTTP server implementation. You would have to embed Thrift in another server if you want HTTP. It's worth noting that the C++ servers we have for Thrift are not really designed to be used on the open internet. They tend to assume that you are in a protected intranet environment, behind a firewall. If this is just for a thesis project and not something you realistically need to worry about being defended against potentially malicious internet traffic, Thrift should be able to get you off the ground running very quickly. Cheers, mcslee -Original Message- From: Matias Hernandez Arellano [mailto:msd...@archlinux.cl] Sent: Tuesday, April 26, 2011 2:51 PM To: user@thrift.apache.org Subject: use thrift to create an interface from ruby to C++ and use it as webservice (Sorry for my english) Hi!, this is my first attempt to use Thrift, i arrive from a stackoverflow answer. First i tell about the idea/problem next the question. I have an application created with C++ that takes an image and make some processing with this, then return a result. This have to be donde with C++ for performance purpose. The application run with OpenCV, armadillo and FANN, the return value can be True or False (1 or 0). The idea is to have an application running on a mobile device (or any other device) and take a picture and send to a server where the C++ application take it, process and return a response. The mobile device have to able to send the image and wait for the response to take and action. The first idea was create a webservice to expose the C++ functionality, i thought about rails, but i can't find a way to solve this problem, so finally i found Thrift. The question. It's possible to use Thrift for this? My idea: Create a Server written in C++ to run the processing application (create this server with Thrift). And create, with thrift, the code to use this server, and use this code to make a implementation in the mobile device. The application is for my thesis for Computer Science. Thanks in advance. -- Matías Hernandez Arellano Ingeniero de Software/Proyectos en VisionLabs S.A CDA Archlinux-CL www.msdark.archlinux.cl Matías Hernandez Arellano Ingeniero de Software/Proyectos en VisionLabs S.A CDA Archlinux-CL www.msdark.archlinux.cl
RE: use thrift to create an interface from ruby to C++ and use it as webservice
It's open source, you can just cite the code! Other than that, I suppose there's just the whitepaper. I wouldn't think a citation is really necessary for this sort of application, but feel free. -Original Message- From: Matias Hernandez Arellano [mailto:msd...@archlinux.cl] Sent: Tuesday, April 26, 2011 4:46 PM To: user@thrift.apache.org Subject: Re: use thrift to create an interface from ruby to C++ and use it as webservice Thanks for your time!!!... I will try this idea... Thanks again... and i let you know... Ah!!! .. is there any document to include (like a citation) in my thesis document? or just the whitepaper? El 26-04-2011, a las 18:17, Mark Slee escribió: This is certainly doable with Thrift. Technically, it won't be a web service, as Thrift doesn't offer and off-the-shelf C++ HTTP server implementation. You would have to embed Thrift in another server if you want HTTP. It's worth noting that the C++ servers we have for Thrift are not really designed to be used on the open internet. They tend to assume that you are in a protected intranet environment, behind a firewall. If this is just for a thesis project and not something you realistically need to worry about being defended against potentially malicious internet traffic, Thrift should be able to get you off the ground running very quickly. Cheers, mcslee -Original Message- From: Matias Hernandez Arellano [mailto:msd...@archlinux.cl] Sent: Tuesday, April 26, 2011 2:51 PM To: user@thrift.apache.org Subject: use thrift to create an interface from ruby to C++ and use it as webservice (Sorry for my english) Hi!, this is my first attempt to use Thrift, i arrive from a stackoverflow answer. First i tell about the idea/problem next the question. I have an application created with C++ that takes an image and make some processing with this, then return a result. This have to be donde with C++ for performance purpose. The application run with OpenCV, armadillo and FANN, the return value can be True or False (1 or 0). The idea is to have an application running on a mobile device (or any other device) and take a picture and send to a server where the C++ application take it, process and return a response. The mobile device have to able to send the image and wait for the response to take and action. The first idea was create a webservice to expose the C++ functionality, i thought about rails, but i can't find a way to solve this problem, so finally i found Thrift. The question. It's possible to use Thrift for this? My idea: Create a Server written in C++ to run the processing application (create this server with Thrift). And create, with thrift, the code to use this server, and use this code to make a implementation in the mobile device. The application is for my thesis for Computer Science. Thanks in advance. -- Matías Hernandez Arellano Ingeniero de Software/Proyectos en VisionLabs S.A CDA Archlinux-CL www.msdark.archlinux.cl Matías Hernandez Arellano Ingeniero de Software/Proyectos en VisionLabs S.A CDA Archlinux-CL www.msdark.archlinux.cl
RE: Thrift IDL
1 a long integer field? The term long is vague, Thrift is explicit. Use i32 for 32-bit integers, i64 for 64-bit integers. You probably want i64. 2. a const int field (equivalent to 'static final int field' in java) ? Struct fields can't be constants. If you just want to define a symbolic constant for use in your code you can say: const i32 MY_CONSTANT = 23423; Or you can just give a field a default-value if you want, but it won't be immutable. Cheers, mcslee -Original Message- From: Anait Markosian [mailto:amarkos...@lyris.com] Sent: Thursday, May 26, 2011 1:18 PM To: user@thrift.apache.org Subject: Thrift IDL How does one represent in thrift struct : 1 a long integer field? 2. a const int field (equivalent to 'static final int field' in java) ? Thanks!
Re: Java Object
Thrift is statically typed and does not support heterogenous containers. The objects in your map need to be of a concrete type. On 2/20/12 6:54 PM, 韶隆吴 yechen1...@gmail.com wrote: How I can I used the java Object using the object in the thrift, like Map String,Object so write
Re: asymmetrical __isset
Still I wonder why it was designed this way. If I compare Thrift
generated files with those created by Protobuf, it becomes immediately
obvious how much smaller is the Thrift generated code. Maybe that is
the reason why it was named Thrift, because it is parsimonious! So
maybe it was designed this way to keep the generated code to the bare
minimum (for example, by not causing an exception on transmission if
the field is not set, as that would require one check of variable
every write).
Your technical analysis is correct, as well as the historical inference!
The required and optional modifiers were not present in the first
version of the IDL, and the original intent was to make using Thrift as
lightweight and transparent as possible. There was an assumption that many
clients would be written in scripting languages, where wrapping everything
up in setters/getters would feel unnatural - we didn't want to make
modifying fields require a function call (via a setter) or require the
programmer to manually maintain all the isset flags. So all fields would
be sent regardless of isset, with one important exception - languages with
nil/null don't send empty fields.
The required/optional keywords were added later to make it easier to
automate some of this checking, and to make it possible for clients in
null-less languages (i.e. C++) to explicitly omit optional fields.
The downside to this approach is that it does make the default behavior
more opaque. Hope that sheds some light on it.
On 4/5/12 8:00 AM, Piscium grok...@gmail.com wrote:
On 5 April 2012 15:05, Rush Manbert r...@manbert.com wrote:
Hi Piscium,
I'm using an older version of Thrift, but my code is all C++.
Try it with your Thrift IDL written like this and I think you'll see
what I meant:
namespace cpp test
struct Test {
1: optional i32 npotatoes, // optional field because it has the
optional keyword
2: i32 nonions, // required field (but will be sent
regardless of the __isset state)
}
Thanks, Rush. You are right, an optional field is sent only if __isset is
set.
But there is more. I realise now that there are actually three types
of fields: optional, required and half-way (when not explicitly
marked as either optional or required). half-way fields are a bit
like half-pregnant, somewhere in-between optional and required!
A half-way field is like a required field in that it is always sent
on transmission, and it is like an optional field in that it does not
cause an exception if not present on reception.
For obvious reasons a half-way field is not symmetrical. Moreover a
required field is not symmetrical either, because there is no
exception on transmission if it is not set. The apparent conclusion is
that the only type of field that is symmetrical is the optional.
Still I wonder why it was designed this way. If I compare Thrift
generated files with those created by Protobuf, it becomes immediately
obvious how much smaller is the Thrift generated code. Maybe that is
the reason why it was named Thrift, because it is parsimonious! So
maybe it was designed this way to keep the generated code to the bare
minimum (for example, by not causing an exception on transmission if
the field is not set, as that would require one check of variable
every write).
Below is the modified Thrift file and generated code.
=
namespace cpp test
struct CountrySoup {
1: optional i32 npotatoes,
2: i32 nonions, // half-way field - neither required nor optional
3: required i32 cabbages,
}
==
namespace test {
const char* CountrySoup::ascii_fingerprint =
A5D9FD141E77B608AA21FD4BD8487AC1;
const uint8_t CountrySoup::binary_fingerprint[16] =
{0xA5,0xD9,0xFD,0x14,0x1E,0x77,0xB6,0x08,0xAA,0x21,0xFD,0x4B,0xD8,0x48,0x7
A,0xC1};
uint32_t CountrySoup::read(::apache::thrift::protocol::TProtocol* iprot) {
uint32_t xfer = 0;
std::string fname;
::apache::thrift::protocol::TType ftype;
int16_t fid;
xfer += iprot-readStructBegin(fname);
using ::apache::thrift::protocol::TProtocolException;
bool isset_cabbages = false;
while (true)
{
xfer += iprot-readFieldBegin(fname, ftype, fid);
if (ftype == ::apache::thrift::protocol::T_STOP) {
break;
}
switch (fid)
{
case 1:
if (ftype == ::apache::thrift::protocol::T_I32) {
xfer += iprot-readI32(this-npotatoes);
this-__isset.npotatoes = true;
} else {
xfer += iprot-skip(ftype);
}
break;
case 2:
if (ftype == ::apache::thrift::protocol::T_I32) {
xfer += iprot-readI32(this-nonions);
this-__isset.nonions = true;
} else {
xfer += iprot-skip(ftype);
}
break;
case 3:
if (ftype == ::apache::thrift::protocol::T_I32) {
xfer += iprot-readI32(this-cabbages);
isset_cabbages = true;
} else {
xfer += iprot-skip(ftype);
}
Re: how to reconnect server after rebooting server
Are you 100% sure that your server is up? Connection refused would imply
that the server isn't listening for new connections on the port you're
trying to connect to...
What type of transport are you using? Could also try invoking -close()
before open(), or creating a new transport - though it *should* work as
written here.
On 4/19/12 9:45 AM, Tiger Y eastdawny...@gmail.com wrote:
Hello,
I tried to execute the code below before each RPC call at client side,
if ( !some_transport-isOpen() )
{
some_transport-open();
}
some_client-ping();
...
but some_transport-open() failed sometimes and got message like
connection refused by blablabla ...
So how to reconnect server rightly?
Thanks.
Re: thrift server questions
This is fairly complicated and doesn't sound like a good use of the linux file-permissioning system. Thrift clients are not Linux users. Assuming you're using TCP (the common case), they are data from an IP/port. You have no way of knowing which user that corresponds to - you can encode it in the application message, but that would be easily spoof-able and you'd definitely need a strong security layer in place to feel confident changing your server's capabilities based upon those messages. This also implies you would be running your server as root, which is generally not recommended. I think it'd probably be better to use group-permissioning on the filesystem, so the server can share file access with the right set of users. You control file security by changing group membership and file permissions, not with application logic in your server. Run your server as a member of the relevant group, no sudo, no setuid/setgid, etc. At a higher-level, what are you trying to accomplish here? Are your server and clients sharing access to a filesystem? If the clients also have access to the filesystem, why don't they operate directly on it? Cheers, Mark On 4/26/12 3:52 PM, Steve Angelovich sangelov...@lgc.com wrote: I am building a server in java based on thrift. The server needs to be able to interact with different users and to read and write files. The problem or question that I'm running into is how can I make sure these file based operation happen as the correct user? I'm currently starting my server using an xinetd script. Would it make any sense to do a sudo to the same user as the client when starting the server if that is even possible? Is there a model or a best practice guide for linux when creating a service that needs to interact with multiple different users? Am I missing something? Is this easy complicated or where do I start? I've looked for some ideas on the net but I'm not finding anything that looks promising. Thanks for any insights, Steve -- This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.
Re: Support array
It is also worth just quickly noting that native arrays in C++ present a few nontrivial hurdles: - Their length needs to be separately maintained - If they are on the stack, all the object storage is on the stack (as opposed to STL) - If they are dynamic, you now need to manage new/delete calls across Thrift call-boundaries Native arrays in Java are more achievable, but the lack of parity with the other container types and Java idioms does not feel worth it. Cheers, mcslee On 5/7/12 12:23 PM, Bryan Duxbury br...@rapleaf.com wrote: I do not foresee us offering array datatypes. On Sun, May 6, 2012 at 8:36 AM, Lingfeng Zhuang zlfc...@gmail.com wrote: Hi all, Does thrift support array datatype like double[]? I know it support listdouble, but I prefer double[] in some cases. Regards zlf
Re: OAuth support out of the box
Definitely no OAuth support built in. Thrift operates a couple layers lower on the stack than that. Certainly possible to implement OAuth with Thrift as the transport and request mechanism, but it's far from out-of-the-box. On 5/22/12 10:22 PM, corno corno crnco...@gmail.com wrote: Hi there, Does Thrift come with any OAuth support or is it fairly easily pluggable to have it work according to spec when using Thrift via the http/s communication channel? I've had a read around on the net and it doesn't seem to be standard with it. Seems that even Evernote has quite an own-rolled version of OAuth on their Thrift API. Thanks in advance for any extra info regarding this.
RE: Compression on serialization
The compact protocol doesn't do compression, the word compact refers to the way it encodes structure and type metadata. Thrift is intended for strongly-typed structured data serialization, not compression. A file is already serialized - it sounds like what you really want is to compress serialized data. Would recommend using zlib or gzip for that. From: Anand Hegde [anandghe...@gmail.com] Sent: Tuesday, June 12, 2012 10:15 PM To: user@thrift.apache.org Subject: Compression on serialization I serialized a 116k file to thrift binary format using the compact protocol but even after serializing, the size is 116K. Isnt the serializer supposed to reduce the file size? The file contains only strings, if that info helps.
RE: a thrift server performance estimate
Completely dependent on your system configuration and the cost of your RPC calls. This question can't really be answered without more info about the application. Some things to consider: - How expensive are these calls? Do they use a lot of CPU? Block on reading files? - You'll be holding a server socket open during each of these calls - you may run into open file descriptor considerations - If the calls are asynchronous, do you close the client as soon as putting in the request? Cheers, Mark From: panfei [cnwe...@gmail.com] Sent: Tuesday, June 19, 2012 3:00 AM To: user@thrift.apache.org Subject: a thrift server performance estimate Hi all : I have a API to call asynchronous method A of thrift server 1, and A only calls another asynchronous method of thrift server 2, I want to know , how many concurrent requests can be processed in this way. thanks. -- 不学习，不知道
Re: reuse Ttransport or not ?
Agreed with Rush. A couple other considerations worth mentioning here: - There is a bit of TCP overhead every time you create and then tear down a connection, if you're trying to squeeze out really high performance and this is the same client talking to the same server, it's preferable not to pay that every time - Re-using the client does mean that the socket is also held open longer on the server side. Timeouts may come into play here if there are significant delays between client calls. If your server specifies a read timeout - it *will* close the client connection after that period elapses with no new call being made. So, as long as your server is not going to be maxed out on connections from tons of different clients, it's generally preferable to re-use. On 7/19/12 10:02 AM, Rush Manbert r...@manbert.com wrote: On Jul 18, 2012, at 11:45 PM, panfei wrote: Hi all: there maybe a lot of RPC calls from the client to the server, I want to know , in this case, should I create a transport and keep it opened to do the RPC calls , or should I create a transport on every RPC call ? and why ? thanks ! -- 不学习，不知道 On a *nix system, opening a transport for each RPC call uses a socket. When you close the transport, the system keeps the socket around for a while, so it is unavailable for re-use. Depending on the rate at which you make RPC calls, your system can get starved for sockets, and you will need to add retry logic to handle that. So it's better to open a single client and use it for multiple calls. I'm guessing you would see the same effect from a Windows client. - Rush
RE: Using multi-threaded clients with Thrift
sometimes having more threads than the number of cores is
desirable when you have a thread pool with worker threads doing
network/disk I/O
Yep, totally agreed, and this is certainly an intuitive way of programming.
But in all of these systems, either each worker thread has exclusive ownership
over the file descriptors/sockets it uses, or you have to introduce locking
around those shared resources.
The latter is typically a much larger performance drain, so that's why Thrift
doesn't do it by default (or at all - it is up to the application layer to lock
appropriately).
I suppose it is possible to keep most of code in a worker thread model
and just use a queue of requests at the lowest level where I need to
make Thrift RPC calls
Yep, exactly. Each thread can have a semaphore. You do all your application
logic in worker threads. When a worker thread needs a network thing to happen,
it places the request in a queue for the network thread, along with your
semaphore. The network thread runs an infinite loop, pulling items off the
queue and signaling the sempahore when they are complete. Your worker thread
code can still *look* synchronous and not have any callbacks.
Here is really rough pseudocode:
network_queue;
worker_thread {
semaphore s;
run() {
// do application logic
request = thrift_request();
request.sempahore = s;
lock(network_queue) {
network_queue.enqueue(request);
s.increment();
}
// block until a networker signals
s.wait();
// request is now populated with our response data
// more application logic
return whatever;
}
}
network_thread {
while (true) {
lock(network_queue) {
request = network_queue.dequeue();
}
request.process();
request.semaphore.signal();
}
}
You can have multiple network threads all working on this queue. Typically
makes sense to have as many networker threads as you have cores.
Cheers,
mcslee
From: Akshat Aranya [aara...@gmail.com]
Sent: Wednesday, August 08, 2012 2:06 PM
To: user@thrift.apache.org
Subject: Re: Using multi-threaded clients with Thrift
On Wed, Aug 8, 2012 at 4:55 PM, Mark Slee ms...@fb.com wrote:
The Thrift transport layer is not thread-safe. It is essentially a wrapper on
a socket.
You can't interleave writing things to a single socket from multiple threads
without locking. You also don't know what order the responses will come back
in. Each thread is effectively calling read(). To have this work in a
multi-threaded environment would require another layer of abstraction that
parceled out responses on the socket and determined which data should go to
which thread. This would be less efficient in the common case of a single
transport per thread.
You certainly could build this functionality on top of the Thrift
abstractions, but the base layers are designed to be very lightweight and
pretty closely mimic raw sockets.
If so, is the only way to make it work in a
multi-threaded environment is to use an independent connection (i.e.,
a new Transport) per thread? That seems kind of wasteful and
inefficient.
In practice, assuming your number of threads is on the order of you number of
cores, this is not inefficient and additional sockets aren't very expensive.
Having each thread own its own socket obviates the need for locking around
all accesses the shared socket resource, which tends to be much more costly.
Another common design in a multi-threaded environment is to have a single
networker thread (or a low fixed number of them). This thread owns a
transport, and the clients put in requests to this thread to perform an
operation and then block, waiting to receive a callback when the operation
they requested is complete.
Cheers,
mcslee
Thanks for the information, Mark. This is not a criticism of Thrift,
but sometimes having more threads than the number of cores is
desirable when you have a thread pool with worker threads doing
network/disk I/O. I have programmed with both a message passing model
and a worker thread model, and in my opinion, the latter results in
more readable code and is easier to program with, especially with
languages such as C++ that require explicit memory management. I
suppose it is possible to keep most of code in a worker thread model
and just use a queue of requests at the lowest level where I need to
make Thrift RPC calls. I was hoping that Thrift would do this for me
out of the box. :-D
Cheers,
Akshat
RE: Using Thrift on Android causing error
i have not solved it!!! From: ankit patel [aprocks.pa...@gmail.com] Sent: Wednesday, August 08, 2012 6:11 AM To: user@thrift.apache.org Subject: Using Thrift on Android causing error have you solved it???
RE: How to extend multiple Services?
This is not supported, Thrift does not implement multiple inheritance.
From: Tarun Goyal [join2ta...@gmail.com]
Sent: Wednesday, September 19, 2012 11:14 PM
To: user@thrift.apache.org
Subject: How to extend multiple Services?
Hey,
How to extend multiple services in one service?
service IBase{
void ping(),
int test(),
}
service ID1 extends IBase{
int test(),
void add(),
}
service ID2 extends IBase{
int test(),
void multiply(),
}
service ID3 extends ID1, extends ID2 {
int test(),
}
When i am compiling it. It is throwing errors. Cant we extend multiple
services?
Thanks,
Tarun
