The ini-style config should be fine to access from Hyracks-level, it's modular. Some of the properties are already accessed from within Hyracks for the most part (e.g. IO devices, etc).
On Tue, Oct 24, 2017 at 1:20 PM, Taewoo Kim <[email protected]> wrote: > Never mind about the first question. The problem becomes simpler (but still > couldn't find the solution) after talking to Chen Luo. Thanks. > > Let's say the inverted index search module Blah1 (Actually SearchResult and > FixedSizeElementInvertedListCursor in case you are interested in). For this > Blah1 module to generate intermediate and final result work properly, the > caller needs to pass three parameters (frameSize, how to > allocate/deallocate buffers, and the maximum number of frames that Blah1 > can have). Blah1 can be called in two ways. > > 1) Search using HyracksTaskCotext (class name: Task) > 2) Search initiated by merge (class name: AsynchronousScheduler) > > 1) and 2) are independent processes so it looks like hard to associate 1) > in 2) or 2) in 1). So, for Blah1, it needs to consider 1) as default and 2) > as a special case or vice versa. Any suggestions? Again, for B+ tree, R > tree, this problem doesn't exist since there is no intermediate result > calculation. > > > > Best, > Taewoo > > On Tue, Oct 24, 2017 at 9:21 AM, Taewoo Kim <[email protected]> wrote: > >> Hello All, >> >> Here is the background. I'm currently fixing an inverted-index search OOM >> issue. This could happen because the core inverted-index search module does >> not have any budget limit. One might ask that B+-tree and R-tree search >> doesn't have any budget limit either. However, a difference is that an >> inverted-index has an inverted list (list of primary keys per secondary >> key). That is, for a single PK, there can be multiple secondary keys. So, >> dealing with this requires additional frames. So, let's assume that we need >> to have multiple frames to answer a text query that utilizes an inverted >> index. >> >> Here is my issue. We will add one more parameter to the config (e.g., >> compiler.textsearchmemory = 32MB) to control this behavior. The >> index-search core module needs to receive and conform to this budget >> information. For inverted-index search query case, it's relatively easy. I >> can use HyracksTaskContext to pass that since there's a pipeline from >> compiler to the search module (IndexSearchOperatorNodePushable). However, >> I found an issue regarding the merge case. Merge of an LSM index happens in >> Hyracks codebase and index merge code doesn't know about >> HyracksTaskContext. So, I talked to Abdullah and he shared a hint about >> ServiceContext. In the ServiceContext, there is AppConfig interface and >> this contains all config information. And when creating an LSM index >> instance, this ServiceContext is accessible. >> >> So, my final question is: is it OK to access AppConfig from Hyracks code >> by specifying the exact parameter (e.g., compiler.textsearchmemory) to get >> some information? Technically, it's doable since the code exists in >> Hyracks. However, most similar parameters are processed in Asterix >> codebase, not in Hyracks codebase. If not, what could be a way to ensure >> this? >> >> Thanks for reading! >> >> Best, >> Taewoo >>
