[
https://issues.apache.org/jira/browse/DRILL-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16822830#comment-16822830
]
ASF GitHub Bot commented on DRILL-7191:
---------------------------------------
HanumathRao commented on pull request #1762: DRILL-7191 Distributed state
persistence and Integration of Distributed queue configuration with Planner
URL: https://github.com/apache/drill/pull/1762
This PR contains changes for the support of RM Framework both on execution
and planning side.
1) Refactoring existing ZK based queue to accommodate new Distributed queue
for RM. Moved QueryResourceAllocators memory allocation code to utility classes
like ZKQueueMemoryAllocationUtilities and DefaultMemoryAllocationUtilities.
Refactored the Parallelizer code to accommodate the memory adjustment for the
operators during parallelization phase. There are 3 different implementation of
SimpleParallelizer such as ZKQueueParallelizer, DistributedQueueParallelizer
and DefaultParallelizer which will be used by ZK based RM, Distributed RM and
Non RM configuration.
2) Planner integration with RM to select queue and reduce query level memory
to be within queue limits. Changes to handle scenarios where buffered operator
are at least getting minimum required memory allocation. Based on the
calculated memory for each operator within each fragment it’s initial and
maximum memory allocation is set which is later consumed by execution layer to
enforce memory limits.
3) Introduced new DrillNode class to deal with issues when DrillbitEndpoint
is searched in a map using some of it’s field.
4) Changes to support storing UUID for each Drillbit Service Instance
locally to be used by planner and execution layer. This UUID is used to
uniquely identify a Drillbit and register Drillbit information in the RM
StateBlobs. Introduced a PersistentStore named
ZookeeperTransactionalPersistenceStore with Transactional capabilities using
Zookeeper Transactional API’s. This is used for updating RM State blobs as all
the updates need to happen in transactional manner. Added RMStateBlobs
definition and support for serde to Zookeeper. Implementation for DistributedRM
and its corresponding QueryRM apis.
5) Updated the state management of Query in Foreman so that same Foreman
object can be submitted multiple times. Also introduced concept of 2 maps
keeping track of waiting and running queries. These were done to support for
async admit protocol which will be needed with Distributed RM.
6) Support for serde of optimalMemoryAllocation for each operator in each
minor fragment in QueryProfile. This is needed to verify the optimalMemory
calculated by planner is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Distributed state persistence and Integration of Distributed queue
> configuration with Planner
> ---------------------------------------------------------------------------------------------
>
> Key: DRILL-7191
> URL: https://issues.apache.org/jira/browse/DRILL-7191
> Project: Apache Drill
> Issue Type: Sub-task
> Components: Server, Query Planning & Optimization
> Affects Versions: 1.17.0
> Reporter: Hanumath Rao Maduri
> Priority: Major
> Fix For: 1.17.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
