Hello All, Here is the background. I'm currently fixing an inverted-index search OOM issue. This could happen because the core inverted-index search module does not have any budget limit. One might ask that B+-tree and R-tree search doesn't have any budget limit either. However, a difference is that an inverted-index has an inverted list (list of primary keys per secondary key). That is, for a single PK, there can be multiple secondary keys. So, dealing with this requires additional frames. So, let's assume that we need to have multiple frames to answer a text query that utilizes an inverted index.
Here is my issue. We will add one more parameter to the config (e.g., compiler.textsearchmemory = 32MB) to control this behavior. The index-search core module needs to receive and conform to this budget information. For inverted-index search query case, it's relatively easy. I can use HyracksTaskContext to pass that since there's a pipeline from compiler to the search module (IndexSearchOperatorNodePushable). However, I found an issue regarding the merge case. Merge of an LSM index happens in Hyracks codebase and index merge code doesn't know about HyracksTaskContext. So, I talked to Abdullah and he shared a hint about ServiceContext. In the ServiceContext, there is AppConfig interface and this contains all config information. And when creating an LSM index instance, this ServiceContext is accessible. So, my final question is: is it OK to access AppConfig from Hyracks code by specifying the exact parameter (e.g., compiler.textsearchmemory) to get some information? Technically, it's doable since the code exists in Hyracks. However, most similar parameters are processed in Asterix codebase, not in Hyracks codebase. If not, what could be a way to ensure this? Thanks for reading! Best, Taewoo
