Yida Wu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16264
Change subject: WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1 ...................................................................... WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1 WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1 Major Features: 1) Local files as buffers for spilling to S3. 2) Async Upload and Sync Fetching of remote files. 3) Sync remote files deletion after query ends. 4) Local buffer files management. 5) Compatibility of spilling to local and remote. 6) All the errors from hdfs/s3 should terminate the query. Implementation Details: 1) An new enum type is added to specify the function of local files. LocalFileMode::BUFFER and LocalFileMode::FILE. LocalFileMode::BUFFER indicates that the local file is used as a buffer for remote operations. LocalFileMode::FILE indicates the local file is used for spilling to local. Also, startup option "remote_tmp_file_local_buff_mode" is added to specify the implementation of the reading pages from the remote. If set to true, the whole file would be fetched to the local buffer during reading. If set to false, only a page is read for each reading. 2) Two disk queues have been added to do the file operation jobs. Queue name: RemoteS3DiskFileOper/RemoteDfsDiskFileOper File operations on the remote disk like upload and fetch should be done in these queues. The purpose of the queues is to seperate long run operations with short ones, and also to have a more accurate control on the thread number working on these file operation jobs, sometimes we might don't want too many upload and fetch jobs working in the same time. RemoteOperRange is the new type to carry the file operation jobs. Previously,we have request types of READ and WRITE. Now FETCH/UPLOAD/EVICT have been added. 3) The tmp files are deleted when the tmp file group is deconstructing. 4) The local buffer files management is to control the total size of local buffer files and evict files if needed. There are basically five status of a remote tmp file, IN_WRITING/DUMPED/IN_DUMPING/REMOTE/TO_DELETE. A local buffer file can be evicted only if it is in status REMOTE. An EVICT job is sent to the local disk queue if a file is decided to be evicted. There are two modes to decide the sequence of choosing files to be evicted. Default is LIFO, the other is FIFO. It can be decided by startup option "remote_tmp_files_avail_pool_lifo". 5) Spilling to local has higher priority than spilling to remote. If no local scratch space is available, temporary data will be spilled to remote. Remote scratch space uses the highest priority local scratch dir as its buffer. If no local scratch space or only one has been configured, a default local buffer should be used. The purpose of the design is to simplify the implementation in milestone 1 with less changes on the configuration. Limitations: * Only one remote scratch dir is supported. * The highest priority local scratch dir is used for the buffer of remote scratch space if remote scratch dir exists. TODO: - Testcases - Refine the naming of the remote scratch dir and files. - Upper and lower bounds of new options related to size. - More accurate error codes and error handling. - Preserve memory buffer for block buffers on file upload and fetch. - Jobs cancelling for new disk queues. - Some metrics might need to be added. - Efficiency issue when mixing local and remote scratch space. Change-Id: Ia5aa4036b4c72656b4297f9fbe42e21d2796a495 --- M be/src/runtime/hdfs-fs-cache.cc M be/src/runtime/io/CMakeLists.txt M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h A be/src/runtime/io/file-writer.h M be/src/runtime/io/hdfs-file-reader.cc A be/src/runtime/io/hdfs-file-writer.cc A be/src/runtime/io/hdfs-file-writer.h M be/src/runtime/io/local-file-system.cc M be/src/runtime/io/local-file-system.h A be/src/runtime/io/local-file-writer.cc A be/src/runtime/io/local-file-writer.h M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M common/thrift/metrics.json 22 files changed, 2,065 insertions(+), 211 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/16264/1 -- To view, visit http://gerrit.cloudera.org:8080/16264 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia5aa4036b4c72656b4297f9fbe42e21d2796a495 Gerrit-Change-Number: 16264 Gerrit-PatchSet: 1 Gerrit-Owner: Yida Wu <[email protected]>
