The cache manager https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java is a cache used only for DTFile. DTFile is a TFileReader(Writer) with cache which provides much faster IO compared to native hadoop TFile but keeps the format
On Tue, Sep 1, 2015 at 2:58 PM, Thomas Weise <[email protected]> wrote: > An operator can use any technology to implement the cache. Here is an > example how to structure a cache from the Malhar library: > > > https://github.com/apache/incubator-apex-malhar/blob/v3.0.0/library/src/main/java/com/datatorrent/lib/db/cache/AbstractDBLookupCacheBackedOperator.java > > > Please note that the resulting caching is local to each operator instance > (every partition has its own cache). The idea is that access to shared > state is minimized for more efficient processing. > > > On Mon, Aug 3, 2015 at 8:21 AM, Amol Kekre <[email protected]> wrote: > > > > > Chiranjeevi, > > We do have guava ( > > https://code.google.com/p/guava-libraries/wiki/CachesExplained) > available > > as a Malhar operator > > > > > > > https://github.com/DataTorrent/Malhar/blob/master/contrib/src/main/java/org/apache/hadoop/io/file/tfile/CacheManager.java > > > > Do take a look. > > > > Thks, > > Amol > > > > > > On Mon, Aug 3, 2015 at 1:51 AM, ravi <[email protected]> wrote: > > > >> Hi Team, > >> > >> Do we have caching available in Datatorrent? If yes, Please let me know > >> how to implement. > >> > >> -many thanks > >> Chiranjeevi > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Malhar" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected]. > >> To post to this group, send email to [email protected]. > >> Visit this group at http://groups.google.com/group/malhar-users. > >> For more options, visit https://groups.google.com/d/optout. > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Malhar" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To post to this group, send email to [email protected]. > > Visit this group at http://groups.google.com/group/malhar-users. > > For more options, visit https://groups.google.com/d/optout. > > >
