Hi All, I'll be working on implementing Spooled Data structures. The overall proposal is the following:
Overall design proposal: Spooled Data structures will be designed to work ontop of a key value store. Spooled data structures would be used as a component inside of an operator. The way the component would function is as followed: StateFactory: The StateFactory is an interface. The interface will have the setup, beginWindow, endWindow and teardown callbacks which an operator would have to call. The StateFactory is used to create handles to spooled DataStructures. The three types of spooled data structures it can create are Lists, Maps, and MultivaluedMaps Like this. StateFactory myFactory = new ManagedStateStateFactoryImpl(); List mySpooledList = myFactory.createNewList(byte[] keyPrefix, long bucket); Map mySpooledMap = myFactory.createNewMap(byte[] keyPrefix, long bucket); Map mySpooledMultiValuesMap = myFactory.createNewMultiValuedMap(byte[] keyPrefix, long bucket); The StateFactory can have specific implementations for different backing stores, and can be set as a property on operators. More complex spooled datastructures can simply be created from spooled lists and maps. More details are on the ticket https://issues.apache.org/jira/browse/APEXMALHAR-2026 Thanks, Tim
