[protobuf] Re: Timeouts for reading from a CodedInputStream
On Sep 28, 6:07 pm, Kenton Varda wrote: > On Tue, Sep 28, 2010 at 5:38 PM, Evan Jones wrote: > > On Sep 28, 2010, at 18:36 , Patrick wrote: > > >> I also have the problem that the RPC I wrote comes in a threaded model > >> and a multi-process model. The multi-process one makes some things a > >> bit harder. I was hoping to utilize a shm mutex to signal termination > >> but this would only work if my message parsing loop timed out every so > >> often and, therefore, could check the mutex. > > > This should be pretty easy to achieve by supplying your own implementation > > of FileInputStream that uses select() and a non-blocking read() rather than > > just read(). It can then fail the call to Next() whenever it is convenient. > > Oh duh, why didn't I think of that? Sounds simple enough. I'll give it a go. Thanks you both for your help. - Patrick -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Timeouts for reading from a CodedInputStream
On Tue, Sep 28, 2010 at 5:38 PM, Evan Jones wrote: > On Sep 28, 2010, at 18:36 , Patrick wrote: > >> I also have the problem that the RPC I wrote comes in a threaded model >> and a multi-process model. The multi-process one makes some things a >> bit harder. I was hoping to utilize a shm mutex to signal termination >> but this would only work if my message parsing loop timed out every so >> often and, therefore, could check the mutex. >> > > This should be pretty easy to achieve by supplying your own implementation > of FileInputStream that uses select() and a non-blocking read() rather than > just read(). It can then fail the call to Next() whenever it is convenient. > Oh duh, why didn't I think of that? -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Timeouts for reading from a CodedInputStream
On Sep 28, 2010, at 18:36 , Patrick wrote: I also have the problem that the RPC I wrote comes in a threaded model and a multi-process model. The multi-process one makes some things a bit harder. I was hoping to utilize a shm mutex to signal termination but this would only work if my message parsing loop timed out every so often and, therefore, could check the mutex. This should be pretty easy to achieve by supplying your own implementation of FileInputStream that uses select() and a non- blocking read() rather than just read(). It can then fail the call to Next() whenever it is convenient. Evan -- Evan Jones http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: Timeouts for reading from a CodedInputStream
Not sure who "Ken" is, but I don't have much to add here. The protobuf parsing code is not designed for non-blocking I/O (which would be excessively complicated). If you need non-blocking I/O, the only thing you can do is read to a separate buffer, and only invoke the protobuf parser when you've received the whole message. Either that, or I guess you could use signals to interrupt parsing. Don't use signals. On Tue, Sep 28, 2010 at 3:36 PM, Patrick wrote: > Thanks Evan. I don't really want to buy the cow to get a glass of > milk. > > I also have the problem that the RPC I wrote comes in a threaded model > and a multi-process model. The multi-process one makes some things a > bit harder. I was hoping to utilize a shm mutex to signal termination > but this would only work if my message parsing loop timed out every so > often and, therefore, could check the mutex. > > I'm sure Ken has thought about this problem before. I'm curious on his > thoughts and if there are plans on supporting polling reads. > > On Sep 28, 12:41 pm, Evan Jones wrote: > > On Sep 28, 2010, at 15:33 , Patrick wrote: > > > > > This is all fine and dandy except when I want to shutdown the server > > > or connection (not client initiated). The ReadTag (as well as the > > > other Read functions) blocks until data is received but I want it to > > > timeout after a specified amount of time. So in essence a polling read > > > instead of a blocking one. This will allow me to check that the > > > connection is still valid and either re-enter my message parsing > > > function or cleanup and exit. > > > > One quick hack that might work: if you have threads anyway, if you > > close the file descriptor in the other thread, the read will fail. > > This causes input.ReadTag() to return 0. > > > > The more complex hack is to supply your own ZeroCopyInputStream > > implementation, and in your implementation of ::Next, implement your > > own time out logic. > > > > In my implementation, I manage this by manually managing my own > > buffer, so I never call the CodedInputStream routines unless I know > > there is sufficient data. This may not be ideal for your application, > > so your milage may vary. > > > > Good luck, > > > > Evan Jones > > > > -- > > Evan Joneshttp://evanjones.ca/ > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Timeouts for reading from a CodedInputStream
Thanks Evan. I don't really want to buy the cow to get a glass of milk. I also have the problem that the RPC I wrote comes in a threaded model and a multi-process model. The multi-process one makes some things a bit harder. I was hoping to utilize a shm mutex to signal termination but this would only work if my message parsing loop timed out every so often and, therefore, could check the mutex. I'm sure Kenton has thought about this problem before. I'm curious on his thoughts and if there are plans on supporting polling reads. On Sep 28, 12:41 pm, Evan Jones wrote: > On Sep 28, 2010, at 15:33 , Patrick wrote: > > > This is all fine and dandy except when I want to shutdown the server > > or connection (not client initiated). The ReadTag (as well as the > > other Read functions) blocks until data is received but I want it to > > timeout after a specified amount of time. So in essence a polling read > > instead of a blocking one. This will allow me to check that the > > connection is still valid and either re-enter my message parsing > > function or cleanup and exit. > > One quick hack that might work: if you have threads anyway, if you > close the file descriptor in the other thread, the read will fail. > This causes input.ReadTag() to return 0. > > The more complex hack is to supply your own ZeroCopyInputStream > implementation, and in your implementation of ::Next, implement your > own time out logic. > > In my implementation, I manage this by manually managing my own > buffer, so I never call the CodedInputStream routines unless I know > there is sufficient data. This may not be ideal for your application, > so your milage may vary. > > Good luck, > > Evan Jones > > -- > Evan Joneshttp://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Timeouts for reading from a CodedInputStream
Thanks Evan. I don't really want to buy the cow to get a glass of milk. I also have the problem that the RPC I wrote comes in a threaded model and a multi-process model. The multi-process one makes some things a bit harder. I was hoping to utilize a shm mutex to signal termination but this would only work if my message parsing loop timed out every so often and, therefore, could check the mutex. I'm sure Ken has thought about this problem before. I'm curious on his thoughts and if there are plans on supporting polling reads. On Sep 28, 12:41 pm, Evan Jones wrote: > On Sep 28, 2010, at 15:33 , Patrick wrote: > > > This is all fine and dandy except when I want to shutdown the server > > or connection (not client initiated). The ReadTag (as well as the > > other Read functions) blocks until data is received but I want it to > > timeout after a specified amount of time. So in essence a polling read > > instead of a blocking one. This will allow me to check that the > > connection is still valid and either re-enter my message parsing > > function or cleanup and exit. > > One quick hack that might work: if you have threads anyway, if you > close the file descriptor in the other thread, the read will fail. > This causes input.ReadTag() to return 0. > > The more complex hack is to supply your own ZeroCopyInputStream > implementation, and in your implementation of ::Next, implement your > own time out logic. > > In my implementation, I manage this by manually managing my own > buffer, so I never call the CodedInputStream routines unless I know > there is sufficient data. This may not be ideal for your application, > so your milage may vary. > > Good luck, > > Evan Jones > > -- > Evan Joneshttp://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
