We haven't started tuning for UDP. That module was a PoC because all the other memcache modules were _incomplete_ in terms of feature set. We use namespaces for different things like threats, assets, users, containers, etc. The plan is to also use pools to scale the memory and allow for easier distributed update across all ingestion nodes.
I plan to start testing using UDP and dalli, but if dalli doesn't do the trick, we'll re-write with the java client lib. For the UDP lookups, a single key should be fine in 1400 bytes for common lookups like IPs. We can also consider jumboframes (9000) for increased payload. Multi packet sets may be necessary, especially if we do lookups on commonly exploited URI patterns with long URIs. With regard to testing, i'd like to PoC this with the goal of doing 1M events/s with 1M lookups/s for enrichment. On Sat, Mar 17, 2018 at 2:58 PM, dormando <[email protected]> wrote: > Hey, > > That is exactly the use case I would expect out of UDP! Thanks for > responding so quickly. > > In your example module I can't quickly figure out how UDP is configured > (it just seems to import dalli and use it?) > > Are you using multi-packet responses for UDP lookups? If you're running > sets via UDP they can't be more than 1400 bytes in size (more like > 1300ish). That's generally been okay? > > On Sat, 17 Mar 2018, Robert Gil wrote: > > > I still have a use case for it. We're looking to make the highest > performing enrichment for log ingestion possible. > > > > https://www.elastic.co/blog/elasticsearch-data-enrichment- > with-logstash-a-few-security-examples > > > > In the use case described above, we aim to enrich as close to real time > as possible for log ingestion. Small, light, lookups... non blocking > updates for > > threat and other security related feeds. > > > > We're not too concerned about misses since we need to back > enrich/compare against data that has already been ingested. > > > > Thoughts? > > > > Rob > > > > On Sat, Mar 17, 2018 at 2:22 PM, dormando <[email protected]> wrote: > > Not sure how active this list is anymore :P > > > > Are any of you still listening users of the UDP protocol? If so, > mind > > reaching out to me (here or privately) to explain your use case? > > > > I'm thinking around some long term options with it; one is to turn > it into > > a compile flag, and another is to keep it (and update it with > SO_REUSEPORT > > and *_mmsg under linux) but restrict responses to single packet. > It can > > still be useful for fire-and-forget cases (ie; spraying touch's or > > fetching flag keys asynchronously), but there isn't much use for it > > otherwise these days. > > > > Thanks, > > -Dormando > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, > send an email to [email protected]. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
