Re: [zeromq-dev] Last call for Extensible Statistics Transmission Protocol (ESTP) v1.0
After reading your answer and the definition type in the spec I admit they all make sense, I did not understood what DELTA was supposed to be but I am not sure why since the definition is not that long. On 14 June 2012 22:10, Paul Colomiets p...@colomiets.name wrote: Hi Schmurfy, On Thu, Jun 14, 2012 at 2:20 PM, Schmurfy schmu...@gmail.com wrote: DERIVE: that's were I am getting skeptical, once we agree that the type used at the end of the chain (you advice using DERIVE RRD type to store counters) (say RRD) is not the one used in the protocol is there really any reason to have a DERIVE type in the protocol ? Note that it is COUNTER that's is represented by DERIVE type, not DERIVE represented as a COUNTER. So I don't think argument applies, as not every DERIVE is a COUNTER (particularly, the one having value that can go down). Would a client send the derive computed by itself or send the raw value and let the server do this ? In the later case this is just a counter. Yes, the type is considered to be used by stateless implementations. All COUNTER uses are also for stateless implementations, and also potentially can be calculated by sender. Potentially the difference is mostly in handling message loss: message loss in COUNTER results to averaging the value, message loss in DELTA value means that period is totally lost (and system may have high load during that period, which was a reason of a message loss, and message loss will not count against network quota as described below). By the way, collectd uses only DERIVE with zero minimum, which is essentially a COUNTER type for ESTP. Having no real DERIVE type I see as a bug in collectd (which I'll discuss in their ML soon), but actually means that no use cases for real DERIVE type have been emerged in collectd. All in all, I've changed my mind 3 times while writing this email, so will probably take a break and will think more :) But here is basic round up: Pros 1. Seems natural, complementary type to COUNTER 2. Usual type for RRD users 3. Some imaginary use-cases are there, the real ones may appear in future 4. A type that may be hard to add in future (all implementations should be updated) Cons: 1. Additional type (maintenance burden) 2. All imaginary use-cases are covered by GAUGE and good GUI (which shows change over period) 3. No real use cases found *(please, propose anyone!)* 4. Resets of the value (if the underlying value is volatile) work bad 5. Can be calculated by sender (same as COUNTER) DELTA: what is the difference between GAUGE and DELTA ? The difference is that scale of GAUGE doesn't depend on interval (e.g. CPU %). But scale of DELTA very much depends: e.g. 10 messages per second means 600 messages per minute. In other words: 10 with interval of 10 is not the same as 10 with interval of 60 (compare with CPU%). So if client for some reason chooses to change interval of the value reporting, the old values (e.g. stored as messages per second) can still be used. The changing interval is very frequent use case for GUI: it may be more interesting to see messages per hour, instead seeing messages per second at hour intervals (the latter may be nice too, but this type gives you a choice). Basically DELTA value is usually implemented as a COUNTER with remembering value at the start interval and sending difference (or alternatively by resetting counter at read), the statistics collection application then divides it by seconds elapsed to store nice messages per second value. Here is something to think about on types, say I have a probe sending the number of bytes sent by the network card to my central server, now what I want to graph is: - the speed at which data are sent - the total number of bytes sent (say I want to check how much I will pay at the end of the month) For this I would prefer having my client sending one metric which is the number of bytes sent and then let the server store the data, one metric received could lead to storing two, three or maybe more RRD metrics (if used) but why the client should care about that ? Sure use a COUNTER type. Storing both values is a collection application's problem. Note that actually storing the bytes per second rate with double precision (as RRD does), gives you quite precise estimate of a traffic per month (and per day or week), without storing values twice (and it's actually better in case of counter reset or wrap than the difference between counter values at the start of the month and at the end) I may be completely wrong on the goal of the types but for me they should just define what the client is sending and not how the data will be ultimately stored on disk if we want something flexible. So are the current types a definition of what is sent or an indication of how to store the data ? Yes is the definition of what client is sending. But DELTA type gives
Re: [zeromq-dev] Last call for Extensible Statistics Transmission Protocol (ESTP) v1.0
I dd not gave it too much thought before starting implementing it but why using : as a separator (which is later used in the time) but suddenly stop using it halfway and use space which can even appear multiple times ? Why not just go for something like this: ESTP*org.example*sys**cpu*2012-06-02T09:36:45*10*7.2 In ruby (and I am sure it is as easy in most high level languages) it could be parsed as simply as: header, host, app, res, metric, time, interval, value = ESTP*org.example*sys**cpu*2012-06-02T09:36:45*10*7.2:a.split(*) time = Time.parse(time) value, value_type = value.split(:) extension fields are omitted but I am sure you see my point. I don't see why we should have something much more complex than needed unless there is a good reason. On 15 June 2012 11:33, Schmurfy schmu...@gmail.com wrote: After reading your answer and the definition type in the spec I admit they all make sense, I did not understood what DELTA was supposed to be but I am not sure why since the definition is not that long. On 14 June 2012 22:10, Paul Colomiets p...@colomiets.name wrote: Hi Schmurfy, On Thu, Jun 14, 2012 at 2:20 PM, Schmurfy schmu...@gmail.com wrote: DERIVE: that's were I am getting skeptical, once we agree that the type used at the end of the chain (you advice using DERIVE RRD type to store counters) (say RRD) is not the one used in the protocol is there really any reason to have a DERIVE type in the protocol ? Note that it is COUNTER that's is represented by DERIVE type, not DERIVE represented as a COUNTER. So I don't think argument applies, as not every DERIVE is a COUNTER (particularly, the one having value that can go down). Would a client send the derive computed by itself or send the raw value and let the server do this ? In the later case this is just a counter. Yes, the type is considered to be used by stateless implementations. All COUNTER uses are also for stateless implementations, and also potentially can be calculated by sender. Potentially the difference is mostly in handling message loss: message loss in COUNTER results to averaging the value, message loss in DELTA value means that period is totally lost (and system may have high load during that period, which was a reason of a message loss, and message loss will not count against network quota as described below). By the way, collectd uses only DERIVE with zero minimum, which is essentially a COUNTER type for ESTP. Having no real DERIVE type I see as a bug in collectd (which I'll discuss in their ML soon), but actually means that no use cases for real DERIVE type have been emerged in collectd. All in all, I've changed my mind 3 times while writing this email, so will probably take a break and will think more :) But here is basic round up: Pros 1. Seems natural, complementary type to COUNTER 2. Usual type for RRD users 3. Some imaginary use-cases are there, the real ones may appear in future 4. A type that may be hard to add in future (all implementations should be updated) Cons: 1. Additional type (maintenance burden) 2. All imaginary use-cases are covered by GAUGE and good GUI (which shows change over period) 3. No real use cases found *(please, propose anyone!)* 4. Resets of the value (if the underlying value is volatile) work bad 5. Can be calculated by sender (same as COUNTER) DELTA: what is the difference between GAUGE and DELTA ? The difference is that scale of GAUGE doesn't depend on interval (e.g. CPU %). But scale of DELTA very much depends: e.g. 10 messages per second means 600 messages per minute. In other words: 10 with interval of 10 is not the same as 10 with interval of 60 (compare with CPU%). So if client for some reason chooses to change interval of the value reporting, the old values (e.g. stored as messages per second) can still be used. The changing interval is very frequent use case for GUI: it may be more interesting to see messages per hour, instead seeing messages per second at hour intervals (the latter may be nice too, but this type gives you a choice). Basically DELTA value is usually implemented as a COUNTER with remembering value at the start interval and sending difference (or alternatively by resetting counter at read), the statistics collection application then divides it by seconds elapsed to store nice messages per second value. Here is something to think about on types, say I have a probe sending the number of bytes sent by the network card to my central server, now what I want to graph is: - the speed at which data are sent - the total number of bytes sent (say I want to check how much I will pay at the end of the month) For this I would prefer having my client sending one metric which is the number of bytes sent and then let the server store the data, one metric received could lead to storing two, three or maybe more RRD metrics (if used) but why the client should care about
Re: [zeromq-dev] Statistics protocol (ESTP) v0.2
I think you can post about ESTP on the collectd mailing list to bring more people into the boat, I am sure there is people interested into this there :) On 13 June 2012 22:44, Paul Colomiets p...@colomiets.name wrote: Hi, On Wed, Jun 13, 2012 at 9:42 PM, Schmurfy schmu...@gmail.com wrote: Latest version looks fine :) I will try implementing it on one project tomorrow see if I find other potential problems. Nice. I will wait a week or so until declaring 1.0. And will try to implement more stuff during this time too. -- Paul ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Last call for Extensible Statistics Transmission Protocol (ESTP) v1.0
I started implementing it and I have some comments: the types as defined in RRD are not that great we seem to agree on that since you renamed one already so why not going a step further ? Here is some thoughts about the current types: GAUGE: not much to say about this one, it is pretty self explanatory. COUNTER: the value sent can only grows and will never reset (except on overflow), fine too DERIVE: that's were I am getting skeptical, once we agree that the type used at the end of the chain (you advice using DERIVE RRD type to store counters) (say RRD) is not the one used in the protocol is there really any reason to have a DERIVE type in the protocol ? Would a client send the derive computed by itself or send the raw value and let the server do this ? In the later case this is just a counter. DELTA: what is the difference between GAUGE and DELTA ? Here is something to think about on types, say I have a probe sending the number of bytes sent by the network card to my central server, now what I want to graph is: - the speed at which data are sent - the total number of bytes sent (say I want to check how much I will pay at the end of the month) For this I would prefer having my client sending one metric which is the number of bytes sent and then let the server store the data, one metric received could lead to storing two, three or maybe more RRD metrics (if used) but why the client should care about that ? I may be completely wrong on the goal of the types but for me they should just define what the client is sending and not how the data will be ultimately stored on disk if we want something flexible. So are the current types a definition of what is sent or an indication of how to store the data ? On 11 June 2012 22:59, Paul Colomiets p...@colomiets.name wrote: Hi, I'm going to pronounce protocol version v1.0, as its now complete, and most issues discussed here are fixed. Now it's an opportunity to find last issues, before freezing the specification. The last changes are: * Type is denoted by colon and english letter, instead of cryptic character * There is x type marker that allows experimental types Spec is now in github organization, which should aggregate projects using the protocol: https://github.com/estp/estp/blob/master/specification.rst I'll be happy if someone could fix grammar, as I'm not native English speaker (We are not at IETF yet, so will fix grammar later if needed, but now is a better time). There is also collectd plugin implementation (it supports v0.2 proto at the time of writing, but will be updated soon): https://github.com/estp/collectd-zeromq-estp After protocol is declared stable, the following steps will be next: * Define collectd extension, which allows to transfer data losslessly between collectd instances (and implement it in plugin) * Implement collectd plugin for crossroads (mostly same as zeromq one) * Define a recommendation for the metric names to use for various application classes and propose to their communities, HTTP servers being the first in my plan -- Paul ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Statistics protocol (ESTP) v0.2
Latest version looks fine :) I will try implementing it on one project tomorrow see if I find other potential problems. On 11 June 2012 12:21, Schmurfy schmu...@gmail.com wrote: On 10 June 2012 14:12, Paul Colomiets p...@colomiets.name wrote: Hi Schmurfy, On Sun, Jun 10, 2012 at 1:11 PM, Schmurfy schmu...@gmail.com wrote: I am really not sure on this, while I like sensible defaults when writing code to speed things up I prefer protocols to clearly say what they try to express, if there is a default I agree that gauge would be the most common type but it introduces one problem for me: you have to deal with the presence or absence of this field and as you mentioned the number of fields will change between a gauge value and a counter value. How do you handle multiple flags scenario without separator ? 43btgh would really be ugly to parse. Well, I consider it type, not flags. So the type is only one. Being it non-letter gives ability to use letters to describe flags, or parameters easier. If you know the value will always be something like: 23.4|ghty (with hty being hypothetical flags and g the type) then you can just split the string with | and look for known flags in the second part which becomes a little harder is the flags are directly added at the end of the value (34.56ght) since in this later case you need to do some magic to separate what is two different informations for me. I've tried to implement a collectd plugin: https://github.com/estp/collectd-zeromq-estp It was easy. Even in C. In scripting languages there are even more tools (e.g. regular expressions). I stil don't like being forced to use regexp when a simple split could do the job just for the sake of allowing lazy parsers, for a protocol as simple as this I consider you either code a full parser or nothing, why stopping halfway ? If there is a separator (the pipe here) then having gauge as default bother me less because a quick split will reveals there is no flags defined without further operations. I agree mostly on the extensibility but if you wanted to add more options to the value the logical choice would be to add other one letters flags in this case. No, the important point is that lazy parser can skip parameters, and almost always be right. Until now I never needed to add much to the value itself and its type, knowing counters size may be useful to detect overflow though but I never worked with counters. This is also the reason why gauge is default. Most people will just use it, without thinking about types. I think the real question is what informations we could want linked to a value, allowing anything and everything to be added leter may not be the more efficient way ;) (that's funny because some parts of the discussion start to look like low level concerns for zeromq /libxs) I've added forward compatibility section. Can you look through? Looks fine. P.S.: Latest spec is now at https://github.com/estp/estp/blob/master/specification.rst -- Paul ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Statistics protocol (ESTP) v0.2
On 10 June 2012 14:12, Paul Colomiets p...@colomiets.name wrote: Hi Schmurfy, On Sun, Jun 10, 2012 at 1:11 PM, Schmurfy schmu...@gmail.com wrote: I am really not sure on this, while I like sensible defaults when writing code to speed things up I prefer protocols to clearly say what they try to express, if there is a default I agree that gauge would be the most common type but it introduces one problem for me: you have to deal with the presence or absence of this field and as you mentioned the number of fields will change between a gauge value and a counter value. How do you handle multiple flags scenario without separator ? 43btgh would really be ugly to parse. Well, I consider it type, not flags. So the type is only one. Being it non-letter gives ability to use letters to describe flags, or parameters easier. If you know the value will always be something like: 23.4|ghty (with hty being hypothetical flags and g the type) then you can just split the string with | and look for known flags in the second part which becomes a little harder is the flags are directly added at the end of the value (34.56ght) since in this later case you need to do some magic to separate what is two different informations for me. I've tried to implement a collectd plugin: https://github.com/estp/collectd-zeromq-estp It was easy. Even in C. In scripting languages there are even more tools (e.g. regular expressions). I stil don't like being forced to use regexp when a simple split could do the job just for the sake of allowing lazy parsers, for a protocol as simple as this I consider you either code a full parser or nothing, why stopping halfway ? If there is a separator (the pipe here) then having gauge as default bother me less because a quick split will reveals there is no flags defined without further operations. I agree mostly on the extensibility but if you wanted to add more options to the value the logical choice would be to add other one letters flags in this case. No, the important point is that lazy parser can skip parameters, and almost always be right. Until now I never needed to add much to the value itself and its type, knowing counters size may be useful to detect overflow though but I never worked with counters. This is also the reason why gauge is default. Most people will just use it, without thinking about types. I think the real question is what informations we could want linked to a value, allowing anything and everything to be added leter may not be the more efficient way ;) (that's funny because some parts of the discussion start to look like low level concerns for zeromq /libxs) I've added forward compatibility section. Can you look through? Looks fine. P.S.: Latest spec is now at https://github.com/estp/estp/blob/master/specification.rst -- Paul ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Statistics protocol (ESTP) v0.2
On 8 June 2012 23:54, Paul Colomiets p...@colomiets.name wrote: Hi Schmurfy, Marten, On Fri, Jun 8, 2012 at 1:21 PM, Schmurfy schmu...@gmail.com wrote: In the intent of building a protocol which is easy to read as a human why using obscure symbols to denote the metric type ? I am talking about ^,' and + Yes. I've just realized that too. The original intention was to make types parametric: 1000'-10:10 -- a derive type with limits from -10 to 10 1234^32bit -- counter with wrap point at 32bit This way it looks more natural -- as a separator. Then I looked at collectd and realized that counter that wraps is not used there, because it's safer to use derive with minimum of 0. And type with minimum and maximum is coded inside configuration there, so it's hard to integrate custom limits in collectd, so I've decided to leave this to the later extensions of the protocol. Does it make sense now? Yes it makes a little more sense but I think a letter clearly describing the type is better, see below. I kinda like the statsd approach on this (https://github.com/etsy/statsd ), why not using something like this (for the value field): gauge: 32.45|g counter: 129800|c derive: 56|d Hm, statd uses semi-readable types. But uses cryptic |@ to denote interval :) I did not said I like all the protocol :) always having the type would also be nicer as opposite that having gauge considered as default. I think gauge is most simple and understandable type, so it seems obvious that it's default. See more reasoning below. I am really not sure on this, while I like sensible defaults when writing code to speed things up I prefer protocols to clearly say what they try to express, if there is a default I agree that gauge would be the most common type but it introduces one problem for me: you have to deal with the presence or absence of this field and as you mentioned the number of fields will change between a gauge value and a counter value. How do you handle multiple flags scenario without separator ? 43btgh would really be ugly to parse. If you know the value will always be something like: 23.4|ghty (with hty being hypothetical flags and g the type) then you can just split the string with | and look for known flags in the second part which becomes a little harder is the flags are directly added at the end of the value (34.56ght) since in this later case you need to do some magic to separate what is two different informations for me. If there is a separator (the pipe here) then having gauge as default bother me less because a quick split will reveals there is no flags defined without further operations. I agree mostly on the extensibility but if you wanted to add more options to the value the logical choice would be to add other one letters flags in this case. Until now I never needed to add much to the value itself and its type, knowing counters size may be useful to detect overflow though but I never worked with counters. I think the real question is what informations we could want linked to a value, allowing anything and everything to be added leter may not be the more efficient way ;) (that's funny because some parts of the discussion start to look like low level concerns for zeromq /libxs) There is also the option to have the type as a proper field: ESTP:org.example:sys::cpu: 2012-06-02T09:36:45 10 7.2 g I'm trying to avoid that, because it is possible that some future revision of the protocol would allow several values in the single packet (if single value per packet will be a lot of pain in practice). So I'm trying to keep the protocol extensible to adding more values at the end. Moving type to the place before the value, might help, but will look a bit ugly if several values would have different types. Also parsing a field would encourage to compare whole word: if type == c But the current check is expected to be done like the following: COUNTER_MARKER = ^ ... if COUNTER_MARKER in value: So adding more specific typing would be easier (see possible typing extension above). Similar reasoning applies to something like 1234|c, turning it into: 1234|c:32bit would be a pain. Similar thing applies to making gauge default. It's nice to have fixed number of fields, as that makes parsing easier, but if 1234|d will be extended to 1234|d|-10|10 to have same field separator for the words, that doesn't help. And if 1234|d will be extended to 1234|d:-10:10 in the future, it doesn't help either, as a lot of implementations will check d:-10:10 == d so will treat the type as undefined instead of normal derived type. How about using just a single latin letter without a separator, and leave gauge as default? Any follow up thoughts? see above. The resource name is explained but not present in the fields description for the example (above), is it the part after the :: in the application name ? It's between the colons
Re: [zeromq-dev] Statistics protocol (ESTP) v0.2
In the intent of building a protocol which is easy to read as a human why using obscure symbols to denote the metric type ? I am talking about ^,' and + I kinda like the statsd approach on this (https://github.com/etsy/statsd), why not using something like this (for the value field): gauge: 32.45|g counter: 129800|c derive: 56|d always having the type would also be nicer as opposite that having gauge considered as default. There is also the option to have the type as a proper field: ESTP:org.example:sys::cpu: 2012-06-02T09:36:45 10 7.2 g The resource name is explained but not present in the fields description for the example (above), is it the part after the :: in the application name ? Other that these remarks this is starting to look like something I could use :) Ps: I sent you a pull request for a typo On 8 June 2012 08:30, Paul Colomiets p...@colomiets.name wrote: Hi Pieter, On Fri, Jun 8, 2012 at 4:11 AM, Pieter Hintjens p...@imatix.com wrote: Paul, I've made a pull request with a few spelling/grammar fixes. Thanks a lot. Looks interesting, would be fun to see some running code. Sure. Will be done shortly. -- Paul ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Statistics reporting using zeromq/crossroads
I not really sure about using subscriptions, why do you want to use this schema ? Currently what I have is many collectd sending their data to another collectd, the applications sends their statistics to the master collectd for the most part (we have embedded systems each with its own collectd instance). So for my use the closest pattern would be PUSH/PULL, what is your use case for subscriptions ? After digging into the core of collectd when I implemented my zeromq plugin (and a lua plugin too) I really feel like it was designed to handle a really large amount of data and/or short cycles and it makes sacrifices toward that goal. I like libxs/zeromq but I don't think this would fit all need, if you don't already use these libraries into your application you may reconsider adding them only to provide statistics, in my experiments I tend to prefer allowing multiple entry points (that's one reason I like collectd btw). As an example sending an udp packet should be fairly easy in any language out there and you don't have to maintain a connection, if the application and the collectd daemon (or equivalent service) are close or if you consider some loss acceptable this is a this is a perfectly valid way to send your statistics :) One other project you could look at is Graphite ( http://graphite.wikidot.com/) For the naming I usually go with [hostname, application_name, metric_name], the metric_name could itself include anything: [example.org, disk, sda1/write_operations]. Messagepack is great. But it's not very easy to parse in C (without dependencies) nothing is easy in C :p (I like the language but everything takes time with it) PS: Do you have a screenshot of jarred ? On 29 May 2012 21:50, Paul Colomiets p...@colomiets.name wrote: Hi Schmurfy, On Tue, May 29, 2012 at 3:35 PM, Schmurfy schmu...@gmail.com wrote: What I am trying to do is to have a common infrastructure for application (my applications) and system (disk io, memory used, disk used, ...) statistics, I shown what the collectd protocol could do but I am not entirely satisfied with it either, here are my problems with collectd: - I want to have statistics available as soon as possible: collectd has a cycle time (default is 10s) and plugins have no way to know when the cycle starts or end, they are notified when data is received or ready to be sent but that's all. One problem I had with my zeromq attempt (my original branch is on github: https://github.com/schmurfy/collectd/tree/zeromq but based on collectd 4.x) is that you cannot send one frame per cycle which is what I wanted to do with it (the original network plugins fills a buffer and when the buffer reach a threshold it sends the packet so your values can be sent now or at now+cycle time which means 10s later my a 10s cycle). Well I have no problem with cycle time. We actually don't even flush the statistics while viewing (which drops latency to about 15 minutes). However, we use collectd only for statistics and use separate monitoring system (nagios). And actually all monitoring/statistics systems I've seen have only bigger delay, not smaller. Making more tools which support libxs and using SURVEY sockets to get fresh data, may fix the problem. - I never used encryption, I consider it can be better handled at lower level and bonus point is that you don't cripple your code with it Ok. - I don't like the need to predefine the types used, it is a pain when using multiple collectd servers since they need to all have the same configuration file to understand each other, I prefer a more open way. Me too. That said you don't need to know the types definitions to actually parse the data stream, the number of parts is in the packet itself, you need the definitions to match each cell with its label. Yes. The most obvious way to implement that is use one packet per value. - I am not really fond of the plugin/plugin_instance/type/type_instance, most of the time I don't remember which one is supposed to be what and many existing collectd plugins do not use all fields so it is more annoying than anything else. Me too. But I'm not sure what format is best here. For simple values it's: example.org/cpu Then there are plugins: example.org/processes/zombie Then there is a namespace inside a plugin (plugin instances in terms of collectd): example.org/disk_usage/sda1/free Then there are complex values: example.org/disk/sda1/write.operations example.org/disk/sda1/write.bytes And there are also host-pair values: example.org-example.net/ping/round-trip-time frontend1.tld-www.h1.tld/http/requests frontend1.tld-www.h2.tld/http/requests frontend2.tld-www.h1.tld/http/requests It seems that collectd does things mostry right, except it puts type into a name of the value. This is one of the things I want to fix. Also pings usually should be viewed other way around (not by the host which
Re: [zeromq-dev] Statistics reporting using zeromq/crossroads
What I am trying to do is to have a common infrastructure for application (my applications) and system (disk io, memory used, disk used, ...) statistics, I shown what the collectd protocol could do but I am not entirely satisfied with it either, here are my problems with collectd: - I want to have statistics available as soon as possible: collectd has a cycle time (default is 10s) and plugins have no way to know when the cycle starts or end, they are notified when data is received or ready to be sent but that's all. One problem I had with my zeromq attempt (my original branch is on github: https://github.com/schmurfy/collectd/tree/zeromq but based on collectd 4.x) is that you cannot send one frame per cycle which is what I wanted to do with it (the original network plugins fills a buffer and when the buffer reach a threshold it sends the packet so your values can be sent now or at now+cycle time which means 10s later my a 10s cycle). - I never used encryption, I consider it can be better handled at lower level and bonus point is that you don't cripple your code with it - I don't like the need to predefine the types used, it is a pain when using multiple collectd servers since they need to all have the same configuration file to understand each other, I prefer a more open way. That said you don't need to know the types definitions to actually parse the data stream, the number of parts is in the packet itself, you need the definitions to match each cell with its label. - I am not really fond of the plugin/plugin_instance/type/type_instance, most of the time I don't remember which one is supposed to be what and many existing collectd plugins do not use all fields so it is more annoying than anything else. I don't really like text protocols, sure they may seem easier to parse (and debug since a human can read them) but for statistics which could be sent at a high rate you waste a lot of space (binary parsers are not that hard to write). For reference here is my ruby parser: https://github.com/schmurfy/rrd-grapher/blob/master/lib/rrd-grapher/notifier/parsers/ruby_parser.rb I was thinking about rewriting it in C but it works really well for my current load (rrd-grapher has a subpart handling notifications so my daemon receives the traffic from distant collectd nodes, raise/stop alerts and then sends the data to the local collectd which saves the data to the disk in rrd files). I am currently doing some experiments using a hash like structure serialized with messagepack and udp to transport, since messagepack support a lot of languages it virtually means any language could serialize/deserialize very easily the packets but I have not much to show currently since I am in the early phases of the project. I am currently using https://github.com/schmurfy/rrd-grapher to graph the data read directly from the rrd files (I have a rrdcached server to help keep my I/O from destroying the machine xD), unlike many tools out there the server do not generate any images but sends the data to the clients which then create the graph. I tested some of the existing solutions but I don't like the idea of generating images server side, today's browsers are more than capable of rendering one of more graphs and it allows to shift some work to the client which is a good thing for me since they don't do much ;) The good thing with collectd is the large number of existing plugins to collect nearly any statistic you could desire from your running servers, I even made an attempt to interface a ruby daemon directly with the plugins wich was not a real success ^^ (the plugins exports functions but also use exported functions from collectd itself so interfacing with them is a bit tricky) On 28 May 2012 21:03, Paul Colomiets p...@colomiets.name wrote: Hi Schmurfy, On Mon, May 28, 2012 at 4:00 PM, Schmurfy schmu...@gmail.com wrote: I am really skeptical about finding the graal of the statistic reporting protocol, I made many attempts and everytime there is some things that works in one case but not in another. Anyway I like the idea :) Please share use cases. I personally want to make nicer statistics for zerogw and implement statistics for paperjam, and add some statistics utility for zorro framework. We can also contribute to mongrel2 and to collectd. I also have few commercial projects that would migrate into the system when it's settled. The network protocol used by collectd is the cleanest thing I have seen until now and it is well described which is a really good thing: http://collectd.org/wiki/index.php/Binary_protocol Oh, thanks. I've not found this page. The coments are following. I have implemented parsers for it in maybe 3 languages and each time they were really easy to write. The collectd protocol allows: Will comment on the features: - multiple statistics per network packet - encryption Not particularly interesting. I'd say gathering and encryption of packets can be done
Re: [zeromq-dev] Statistics reporting using zeromq/crossroads
I am really skeptical about finding the graal of the statistic reporting protocol, I made many attempts and everytime there is some things that works in one case but not in another. Anyway I like the idea :) The network protocol used by collectd is the cleanest thing I have seen until now and it is well described which is a really good thing: http://collectd.org/wiki/index.php/Binary_protocol I have implemented parsers for it in maybe 3 languages and each time they were really easy to write. The collectd protocol allows: - multiple statistics per network packet - multiple values per statistics (uptime for example could have 3 for 1min, 5min, 15min) - support for doubles and integers (64bits) - extensible - transport of notifications (severity + string) - encryption I started working on zeromq collectd plugin which allowed external applications to send statistics to the daemon but never finished it, I think there is another one somewhere. For the graphing part there are some solutions based on rrd which can be written by collectd. Overall Collectd is the best thing I have seen to collect/store statistics, I am not fully satisfied with it but nothing else come closer to it, just look at the plugin list On 28 May 2012 10:17, Pieter Hintjens p...@imatix.com wrote: Hi Paul, This is a great idea. My advice is start simple and add functionality as needed over time. If you want an easy reusable legal template for protocols, see the rfc.zeromq.org site. -Pieter On Sun, May 27, 2012 at 4:38 PM, Paul Colomiets p...@colomiets.name wrote: Hi, Every time I develop something using zeromq or libxs I come up with a problem of how to report statistics. Pub/Sub pattern is suited for statistics reporting very well, but the format of messages is unclear. The result is ugly and diverse solutions for reporting real-time statistics. I think the community would benefit from having common format for reporting statistics, so that all open-source software could use the format. And various plugins can be developed for existing statistics and monitoring systems. The protocol should meet the following requirements: 1. Simple compact format. Probably textual or fixed-size binary. It should be easy to craft messages without any dependencies. 2. Each message should bring compact representation of the type information in itself (so that centralized system doesn't need to be reconfigured for each new reported value and each software upgrade on remote system) 3. Subscriber should be able to filter statistics data by pub/sub subscriptions 4. There should be solution to collect that statistics (draw graphs, etc.) So I have the following questions: 1. What formats are available to (re)use? 2. What protocols and software do you use for your projects? 3. Would you like to participate in standardization of the protocol? At our company we use collectd. And as zeromq plugin is not in the mainstream for collectd, and it has complex binary format, we report using some custom formats and a special application that translates that into collectd protocol for unix sockets (which in turn nice textual protocol). Same is done in zerogw (see zerogwstat utility). This kind of proxying, and the point #2 from the requirements makes me incomfortable. -- Paul ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] REQ/REP for possibly async comm?
be careful with setting identities, there are known problems with them, you may want to check messages about issue #325 fix. If you have connection issues between the two process one may end up with an auto generated identity instead of the one you wanted with latest code or in the released code you can get an assertion failure which will crash both your server and client ;) On 30 March 2012 18:24, Andrei Zmievski and...@zmievski.org wrote: ROUTER-ROUTER seemed to work. I just set the socket identity on process B to something known and then process A uses that to send the messages. The socket in process A is anonymous. -Andrei On Fri, Mar 30, 2012 at 8:27 AM, Michel Pelletier pelletier.mic...@gmail.com wrote: I suppose you could use PUB/SUB with subscriptions, I don't know enough about your topology or message patterns to be sure. The advantage would be if you needed to do 1 to N transmission or messages or N to 1 reception. -Michel On Wed, Mar 28, 2012 at 9:27 AM, Andrei Zmievski and...@zmievski.org wrote: Okay, will give that a try. Is there a reason why I couldn't use PUB/SUB sockets for this interaction? -Andrei On Tue, Mar 27, 2012 at 6:37 PM, Michel Pelletier pelletier.mic...@gmail.com wrote: If not getting a timely response is an exceptional condition, then you can throw away your REQ socket, create a new REQ socket, connect it and proceed from there as if it were new. If it's a common occurrence then you can do ROUTER-ROUTER and track the state of requests yourself. REQ and REP enforce a simple state, if you need a more complex state tracking, then you can implement it yourself with ROUTER. The way I think of ROUTER is exactly like the hardware device of the same name, you tell a router's send() where to send the data, when you recv(), it tells you where it came from. given that information you can track your own requests and timeouts for every given connected peer yourself in a zmq_poll() loop. -Michel On Tue, Mar 27, 2012 at 4:35 PM, Andrei Zmievski and...@zmievski.org wrote: Michel, Thanks for the example. However, from what I understand, this pattern does not allow process A to re-send the request in case it has not received the response after a certain amount of time, because REQ sockets don't allow multiple messages? -Andrei You can use non-blocking ROUTER on a poll loop and REQ or REP on the other end, that way you can handle requests and their responses as they come and as you compute them, respectively. Whether to use REQ or REP only matters for whichever end starts the conversation. The LRU pattern documented in the guide has a good implementation of this pattern with REQ-ROUTER-REP. It doesn't sound like you need the actual LRU part of the pattern, just the ROUTER-REQ part. See the code for an example how to use a ROUTER socket in non-blocking mode in a poll loop to handle out of order patterns like A then B requests but you respond B then A. Start here and read it a few times. :) http://zguide.zeromq.org/page:all#Least-Recently-Used-Routing-LRU-Pattern -Michel ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] REQ/REP for possibly async comm?
When you need asynchronous communication you better use ROUTER/DEALER than REQ/REP, the only time I may want to use a REQ socket is to write a simple client taking to a server using a ROUTER socket. Except that I am still wondering why would anyone use REP/REQ given its limitations for anything serious, I think it takes too many assumptions that everything will work as expected which rarely happens in a real use case. I never tried ROUTER-ROUTER though but that's one of the things I love with zeromq, you can really build interesting topologies without the need to concentrate on the low level implementation. On 28 March 2012 18:27, Andrei Zmievski and...@zmievski.org wrote: Okay, will give that a try. Is there a reason why I couldn't use PUB/SUB sockets for this interaction? -Andrei On Tue, Mar 27, 2012 at 6:37 PM, Michel Pelletier pelletier.mic...@gmail.com wrote: If not getting a timely response is an exceptional condition, then you can throw away your REQ socket, create a new REQ socket, connect it and proceed from there as if it were new. If it's a common occurrence then you can do ROUTER-ROUTER and track the state of requests yourself. REQ and REP enforce a simple state, if you need a more complex state tracking, then you can implement it yourself with ROUTER. The way I think of ROUTER is exactly like the hardware device of the same name, you tell a router's send() where to send the data, when you recv(), it tells you where it came from. given that information you can track your own requests and timeouts for every given connected peer yourself in a zmq_poll() loop. -Michel On Tue, Mar 27, 2012 at 4:35 PM, Andrei Zmievski and...@zmievski.org wrote: Michel, Thanks for the example. However, from what I understand, this pattern does not allow process A to re-send the request in case it has not received the response after a certain amount of time, because REQ sockets don't allow multiple messages? -Andrei You can use non-blocking ROUTER on a poll loop and REQ or REP on the other end, that way you can handle requests and their responses as they come and as you compute them, respectively. Whether to use REQ or REP only matters for whichever end starts the conversation. The LRU pattern documented in the guide has a good implementation of this pattern with REQ-ROUTER-REP. It doesn't sound like you need the actual LRU part of the pattern, just the ROUTER-REQ part. See the code for an example how to use a ROUTER socket in non-blocking mode in a poll loop to handle out of order patterns like A then B requests but you respond B then A. Start here and read it a few times. :) http://zguide.zeromq.org/page:all#Least-Recently-Used-Routing-LRU-Pattern -Michel ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Duplicate identities - #325
From what I understand it may take more time to fix it by closing the offender's connection and may cause other problems, if this is the case I'd rather have a working fix right now. I am working on a project where zeromq (or crosroads btw) would be of great help but I need to have the server publicly accessible and this bug is preventing me to even think of doing it currently (or at least was). I still fear there are remaining assertions which may explode in my face but at least that's one less ! On 24 March 2012 16:59, Chuck Remes li...@chuckremes.com wrote: On Mar 23, 2012, at 8:24 PM, Pieter Hintjens wrote: This is for routing only, we don't do durable sockets any longer in 3.1. So it should be OK. FYI, this patch was rejected by crossroads-io. They prefer it be handled by closing the offending connection rather than silently allowing it to connect but with an auto identity. cr ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Getting 0MQ/3.1 to stable status
If the bug I reported is confirmed by someone else It should be considered a blocker too, being able to crash any zeromq server is nothing less than critical for me. On 27 February 2012 22:09, Pieter Hintjens p...@imatix.com wrote: On Mon, Feb 27, 2012 at 2:51 PM, Ian Barber ian.bar...@gmail.com wrote: And Ben has fixed his own issue now too. What's our preferred standard - mark issues as resolved, or resolve and close on a fix? Probably pass to original reported just by making a comment, and mark closed when tested and confirmed. -Pieter ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Getting 0MQ/3.1 to stable status
Can anyone confirm https://zeromq.jira.com/browse/LIBZMQ-325 ? Just to be sure I am not the only one with this bug. On 21 February 2012 13:43, Calvin de Vries devries.cal...@gmail.com wrote: I had LIBZMQ-307 happen to me in production recently. To note, I am using 3.0.2 (which I know is no longer supported), but I think it should be made sure that this isn't occurring in 3.1. If I have time later today I can try to do a test with 3.1, but am not sure yet if that will be possible. On Tue, Feb 21, 2012 at 6:09 AM, Schmurfy schmu...@gmail.com wrote: Hi, I added a new issue I found on the 3.1 master branch: LIBZMQ-325 It allows to crash a server so I would consider it critical, two clients can connect with the same identity but bad things happen when one of them disconnects. On 20 February 2012 17:33, Emmanuel TAUREL tau...@esrf.fr wrote: Hello Peter, On 20/02/2012 17:21, Pieter Hintjens wrote: On Mon, Feb 20, 2012 at 9:24 AM, Pieter Hintjensp...@imatix.com wrote: On Mon, Feb 20, 2012 at 6:49 AM, Emmanuel TAURELtau...@esrf.fr wrote: - LIBZMQ-268 - LIBZMQ-283 These same to be the same problem? Not exactly. In LIBZMQ-268, the whole multipart message is lost (using PGM). In LIBZMQ-283, only the first part of the multipart message is lost using TCP. Thank's for your help Emmanuel ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Getting 0MQ/3.1 to stable status
Hi, I added a new issue I found on the 3.1 master branch: LIBZMQ-325 It allows to crash a server so I would consider it critical, two clients can connect with the same identity but bad things happen when one of them disconnects. On 20 February 2012 17:33, Emmanuel TAUREL tau...@esrf.fr wrote: Hello Peter, On 20/02/2012 17:21, Pieter Hintjens wrote: On Mon, Feb 20, 2012 at 9:24 AM, Pieter Hintjensp...@imatix.com wrote: On Mon, Feb 20, 2012 at 6:49 AM, Emmanuel TAURELtau...@esrf.fr wrote: - LIBZMQ-268 - LIBZMQ-283 These same to be the same problem? Not exactly. In LIBZMQ-268, the whole multipart message is lost (using PGM). In LIBZMQ-283, only the first part of the multipart message is lost using TCP. Thank's for your help Emmanuel ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Using ZMQ_FD with select
I mainly use ROUTER/DEALER for a simple reason: asynchronous communication, I consider REQ/REP to be fine for really simple client otherwise I stay away from them (but you can use a REQ in front of a ROUTER which is the old XREP). Working on the em-zeromq ruby library we have already found some really strange behavior linked to the way the returned descriptor worked but once you got them all it seems pretty reliable. On 19 February 2012 16:15, Yi Ding yi.s.d...@gmail.com wrote: On Sun, Feb 19, 2012 at 4:02 AM, Schmurfy schmu...@gmail.com wrote: Now I really feel stupid xD Thanks for pointing this out, not really intuitive. I am back to square one since it does not explain my problem then xD But I am curious what was your problem with REQ/REP if not this one because I also have problems with REQ/REP where the other sockets type works as intended. I just can't pinpoint what exactly is wrong and where. My problems with REQ/REP mainly came from the fact that it enforces this strict ordering of request, reply, request, reply. So it's really meant for synchronous communication, for both the client and the server. Trying to get it to work with asynchronous communication meant that I would have to create my own queues and monitor those, and make sure not to miss any incoming messages on the select, make sure I was resetting the edge triggering properly (which is doubly hard because calling recv too many times on a REQ or REP socket will trigger EFSM). Ultimately it turned into too much work. I've found a pair of PUSH/PULL sockets works well if the endpoints are all known in advance. Other people have suggested ROUTER/DEALER, but I haven't tried that yet. Also, if the endpoints aren't known in advance, PUB/SUB can work, although endpoint discovery is pretty tricky right now because there's no way to get the IP address off an inbound packet. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Using ZMQ_FD with select
Now I really feel stupid xD Thanks for pointing this out, not really intuitive. I am back to square one since it does not explain my problem then xD But I am curious what was your problem with REQ/REP if not this one because I also have problems with REQ/REP where the other sockets type works as intended. I just can't pinpoint what exactly is wrong and where. On 19 February 2012 00:42, Yi Ding yi.s.d...@gmail.com wrote: On Sat, Feb 18, 2012 at 5:32 PM, Yi Ding yi.s.d...@gmail.com wrote: On Sat, Feb 18, 2012 at 5:31 PM, Yi Ding yi.s.d...@gmail.com wrote: On Sat, Feb 18, 2012 at 8:59 AM, Schmurfy schmu...@gmail.com wrote: I have a strange problem related to using the file descriptor with an event loop which led me to do some testing with it in C but there is still something I cannot figure out. I created a simple client and server, the client (req.c) sends a request and wait for an answer while the server just sends back what was received, my problem is that the client (req.c) always timeout while doing a select on the file descriptor and I cannot figure out what is wrong in my code. Another thing that bother me is that the file descriptor is edge-triggered as specified in the documentation but what happens if I miss the change ? I am not sure if this is possible but let's say I was doing something else when the data was receiving, if I do a select after will it block forever ? Here is my code (I used czmq to make the code easier to write and read, I used zeromq 3.1 for this test): https://gist.github.com/1859646 Thanks for any answer I hate when I don't understand something ^^ I've run into this exact issue Julien. The advice I've gotten is to always call recv after send prior to calling select. I find it counterintuitive, but that's the way the devs have said it should be done. I don't think it's possible to miss a change, as long as you've called recv the exact right number of times. (don't call recv again after successfully recv'ing a message, unless you're looking to get a EFSM, no matter what people on the list might say) The edge triggering will remain valid as long as you don't select on it. Btw, this exact issue gave me so much trouble that I ended up just switching from REP-REQ to PUSH-PULL. OK, major foot in mouth. Should have looked at the code before e-mailing the list. Your problem is here: int ret = select(1, read_set, NULL, NULL, tv); It should instead be: int ret = select(fd + 1, read_set, NULL, NULL, tv); The creators of select wisely chose to confuse people by naming the parameter numfds instead of oneplusmaxfd which is what it really should be named. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Using ZMQ_FD with select
But the descriptor returned by ZMQ_FD is not the real descriptor right ? From my understanding it is just a way to use zeromq sockets with an external event loop and in general it works quite well. On 19 February 2012 14:06, john skaller skal...@users.sourceforge.netwrote: On 19/02/2012, at 9:02 PM, Schmurfy wrote: Now I really feel stupid xD Thanks for pointing this out, not really intuitive. I am back to square one since it does not explain my problem then xD But I am curious what was your problem with REQ/REP if not this one because I also have problems with REQ/REP where the other sockets type works as intended. I just can't pinpoint what exactly is wrong and where. Somehow my message didn't get posted. What actually happens is that ZMQ is doing a select itself, and it loads the message in a background thread so fast that by the time YOU do the select, there's nothing to read on the fd. ZMQ has already read it. You must use zmq_poll. You cannot use select yourself on a fd owned by ZMQ, that leads to a race:) -- john skaller skal...@users.sourceforge.net ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] Using ZMQ_FD with select
I have a strange problem related to using the file descriptor with an event loop which led me to do some testing with it in C but there is still something I cannot figure out. I created a simple client and server, the client (req.c) sends a request and wait for an answer while the server just sends back what was received, my problem is that the client (req.c) always timeout while doing a select on the file descriptor and I cannot figure out what is wrong in my code. Another thing that bother me is that the file descriptor is edge-triggered as specified in the documentation but what happens if I miss the change ? I am not sure if this is possible but let's say I was doing something else when the data was receiving, if I do a select after will it block forever ? Here is my code (I used czmq to make the code easier to write and read, I used zeromq 3.1 for this test): https://gist.github.com/1859646 Thanks for any answer I hate when I don't understand something ^^ Julien Ammous ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
[zeromq-dev] ZMQ_EVENTS / EMQ_FD
Hello, I wanted to know if there is any estimate as to when we can hope to see ZMQ_EVENTS / ZMQ_FD support in a stable release, I honestly don't understand clearly how the releases are done and when (and I never understood the need for an unstable release ;) ). I really love zeromq, it is a really nice library and want to try it on a work project but I want to integrates it with an event loop which is why I need these two features, I already got working code for the most part but working on an unstable or on HEAD is not really an option so here I comes. My concern is that it looks like a lot of things is happening on the master branch with XPUB, XSUB and other things so my question is: are all these features going to be released at once or will they be cut in parts and released by pieces in stable versions ? It would be great to have a minor release adding EVENTS/FD to be able to start to really use the library with an event loop efficiently. Julien Ammous. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] ZMQ_EVENTS / EMQ_FD
Thanks for the clarifications, I have nothing against helping :) I will try backporting it to maint, I noticed weird behaviors working with 2.1 and head but I cannot pinpoint if it is a bug or if I did something wrong. About the master branch, by what you said I suppose it is considered to be working ? I mean that half coded things will live in a separate branch before being merge to master. The reason for the question is that I already saw projects where head was not compiling half of the time and that is really annoying working that way. Julien Ammous On 13 February 2011 17:21, Martin Sustrik sust...@250bpm.com wrote: Hi Julien, I wanted to know if there is any estimate as to when we can hope to see ZMQ_EVENTS / ZMQ_FD support in a stable release It depends on how much testing is done by various parties. If you want to speed it up simply start using the 2.1 and report any bugs you may encounter. I honestly don't understand clearly how the releases are done and when (and I never understood the need for an unstable release ;) ). To have a common reference point, something you can report issues about etc. I really love zeromq, it is a really nice library and want to try it on a work project but I want to integrates it with an event loop which is why I need these two features, I already got working code for the most part but working on an unstable or on HEAD is not really an option so here I comes. Then you'll have to wait. That's the nature of collaborative development. You'll either help with what you want to see in the product or you just wait till others do it for you. My concern is that it looks like a lot of things is happening on the master branch with XPUB, XSUB and other things so my question is: are all these features going to be released at once or will they be cut in parts and released by pieces in stable versions ? XPUB and XSUB were added to the master to indicate that there will be such functionality in the future, however, all the non-trivial work on subscription forwarding is done outside of master, on sub-forward branch. It would be great to have a minor release adding EVENTS/FD to be able to start to really use the library with an event loop efficiently. If you want to backport EVENTS/FD to the maint branch, feel free to do so. Provide the patch in the standard way, just note it's a patch to maint instead of master: http://www.zeromq.org/docs:contributing Martin ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
