Never mind about the first question. The problem becomes simpler (but still couldn't find the solution) after talking to Chen Luo. Thanks.
Let's say the inverted index search module Blah1 (Actually SearchResult and FixedSizeElementInvertedListCursor in case you are interested in). For this Blah1 module to generate intermediate and final result work properly, the caller needs to pass three parameters (frameSize, how to allocate/deallocate buffers, and the maximum number of frames that Blah1 can have). Blah1 can be called in two ways. 1) Search using HyracksTaskCotext (class name: Task) 2) Search initiated by merge (class name: AsynchronousScheduler) 1) and 2) are independent processes so it looks like hard to associate 1) in 2) or 2) in 1). So, for Blah1, it needs to consider 1) as default and 2) as a special case or vice versa. Any suggestions? Again, for B+ tree, R tree, this problem doesn't exist since there is no intermediate result calculation. Best, Taewoo On Tue, Oct 24, 2017 at 9:21 AM, Taewoo Kim <wangs...@gmail.com> wrote: > Hello All, > > Here is the background. I'm currently fixing an inverted-index search OOM > issue. This could happen because the core inverted-index search module does > not have any budget limit. One might ask that B+-tree and R-tree search > doesn't have any budget limit either. However, a difference is that an > inverted-index has an inverted list (list of primary keys per secondary > key). That is, for a single PK, there can be multiple secondary keys. So, > dealing with this requires additional frames. So, let's assume that we need > to have multiple frames to answer a text query that utilizes an inverted > index. > > Here is my issue. We will add one more parameter to the config (e.g., > compiler.textsearchmemory = 32MB) to control this behavior. The > index-search core module needs to receive and conform to this budget > information. For inverted-index search query case, it's relatively easy. I > can use HyracksTaskContext to pass that since there's a pipeline from > compiler to the search module (IndexSearchOperatorNodePushable). However, > I found an issue regarding the merge case. Merge of an LSM index happens in > Hyracks codebase and index merge code doesn't know about > HyracksTaskContext. So, I talked to Abdullah and he shared a hint about > ServiceContext. In the ServiceContext, there is AppConfig interface and > this contains all config information. And when creating an LSM index > instance, this ServiceContext is accessible. > > So, my final question is: is it OK to access AppConfig from Hyracks code > by specifying the exact parameter (e.g., compiler.textsearchmemory) to get > some information? Technically, it's doable since the code exists in > Hyracks. However, most similar parameters are processed in Asterix > codebase, not in Hyracks codebase. If not, what could be a way to ensure > this? > > Thanks for reading! > > Best, > Taewoo >
