Hi Taylor, Good point. This is probably a better title now.
The problem geolocation runs into can be summarized as: "I have bolt that depends on external data that can fit in memory and that changes in batch. How can I update for a new batch without blocking the stream?" On 12/9/15 7:31 PM, "P. Taylor Goetz" <[email protected]> wrote: >Just a suggestion, but this thread has diverged from the original subject >line. Which is fine and happens all the time. > >When it does happen, one thing you can/should do is change the subject >line (e.g "FooBar (was re: Hello"). That makes it easier for subscribers >to notice the discussion branch, and also makes it easy to follow when >viewing the email archives. > >Just $.02 from a mentor. :) > >-Taylor > >> On Dec 9, 2015, at 8:14 PM, Bryan Taylor <[email protected]> wrote: >> >> Hi James, >> >> It seems the solution to "stop the world while we rebuild the cache" is >>to >> rebuild the new cache out of band in a separate thread/spout and then >> inject a reference to it once it is complete. >> >> Actually, I'm curious how updating the data works now with MySQL. The >> "Setting Up GeoLite Data" page describes the initial load using a LOAD >> DATA INFILE command in the MySQL shell, which would duplicate records if >> it was run a second time. >> >> I suppose you could delete and load all the new rows in one big atomic >> transaction, but there could be cute issues if you commit as you go >>during >> the table data rebuild, since queries that happen along the way would be >> hitting a blend of the two datasets. I've used partition swapping for >>this >> with some databases, but I'm not sure if MySQL supports that feature. A >> similar idea is to load the new data into a completely new table, but >> expose it via a view and then recompile the view when the new table is >> done. This has to get a lock on the view, which will stop the world >>while >> the view definition changes, but that's a pretty short duration. >> >> Bryan >> >> >> >>> On 12/9/15 6:20 PM, "James Sirota" <[email protected]> wrote: >>> >>> So the nature of the problem was that as we were processing ~1.3 >>>million >>> of messages per second the time it took for the in-memory DB to update >>> caused the Storm tuples to back up to a point where this would bring >>>down >>> the topology. We also had problems during initialization. I don’t >>>know >>> if this feature exists now, but at the time we couldn’t figure out a >>>way >>> to have the topology deploy and wait for all the instances of Geo bolt >>>to >>> finish reading their data and signal back that they were ready. So at >>> initialization they would get blasted with tuples and fall over. We >>> solved that problem at the time by delaying our ingest 30 seconds to >>>give >>> the topology a chance to fully come up. But eventually we decided we >>> needed to simplify things so we abandoned the in-memory route. >>> >>> Thanks, >>> James >>> >>> >>> >>>> On 12/9/15, 5:35 PM, "Bryan Taylor" <[email protected]> wrote: >>>> >>>> >>>> The GeoLite site says they update once a month, so I assume something >>>>can >>>> check for this and grab the new file. It seems like a fun problem to >>>>have >>>> this also trigger a rebuild of the in-memory cache and swap it out >>>>live. >>>> This seems like it would be a useful streaming enrichment pattern, >>>>where >>>> the configuration data for the enrichment changes. >>>> >>>> This does raise another interesting question about what do we expect >>>>the >>>> memory profile of the steam processing to be. 70Mb + 40Mb isn't that >>>>big >>>> by itself, but when is it worth it? and how do operators take >>>>advantage >>>> of >>>> more system memory if they have it. >>>> >>>> >>>>> On 12/9/15 4:32 PM, "James Sirota" <[email protected]> wrote: >>>>> >>>>> Hi Bryan, >>>>> >>>>> We had HSQLDB at one point, but we were struggling to make these >>>>>bolts >>>>> reliable. Also, the geo data needs to be periodically updated and >>>>>it¹s >>>>> easier to do when it¹s decoupled. >>>>> >>>>> Thanks, >>>>> James >>>>> >>>>> >>>>> >>>>>>
