Hussain Towaileb created ASTERIXDB-3247:
-------------------------------------------
Summary: Apply filter evaluator at each returned batch of files
list
Key: ASTERIXDB-3247
URL: https://issues.apache.org/jira/browse/ASTERIXDB-3247
Project: Apache AsterixDB
Issue Type: Task
Components: EXT - External data
Affects Versions: 0.9.9
Reporter: Hussain Towaileb
Assignee: Hussain Towaileb
Fix For: 0.9.9
External store SDKs return the list of objects in batches, for example, AWS's
S3 list returns 1000 at a time. Currently, we collect all the files, then apply
the evaluator on them to filter them out.
We could optimize this by pushing the filter to perform the
evaluation/filtration on each incoming batch to avoid keeping references to
objects that might eventually get filtered out.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
