GitHub user cwestin opened a pull request:
https://github.com/apache/drill/pull/132
Alloc nolocks
DRILL-1942-hygiene (another preparatory patch):
- add AutoCloseable to some classes
- minor fixes
- formatting
Unit tests pass
Regression suite passed except for some window function queries (this
branch needs to get rebased to pick up that code), and the known fragment
distribution timeout.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cwestin/incubator-drill alloc-nolocks
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/132.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #132
----
commit 51558954489e9015f604a0d017c519664eb7d55a
Author: Chris Westin <[email protected]>
Date: 2015-02-13T23:30:24Z
DRILL-1942: Improve off-heap memory usage tracking
This is a reimplementation of the allocators to make tracking better. The
re-implementation includes support for JMX MBeans for on-the-fly memory
statistics inspection. The new classes are BaseAllocator, RootAllocator, and
ChildAllocator. The BufferAllocator interface has been modified a bit.
In addition to that, the new allocator flattens the use of buffers in
slices,
so that a DrillBuf is always referring to an UnsafeDirectLittleEndian, and
never to another DrillBuf; this simplifies figuring out what's going on when
something goes wrong. The new allocator (BaseAllocator, and its derived
classes RootAllocator and ChildAllocator) treats sharing buffers in the same
way as slicing buffers, and sharing and ownership transfer of buffers and
their slices both work.
Record batch and value vector classes previously had a mandatory cleanup()
method. These classes have been made to include AutoCloseable as a base
class,
and cleanup() has been renamed to close(). This was done to get additional
help from tools to find places that we need to close() things.
This allocator detects a lot more problems than the old one did; fixed many
leaks in various locations.
AbstractRecordBatch:
- cleanup() -> close()
- hygiene
AbstractSingleRecordBatch:
- cleanup() -> close()
- hygiene
Accounter:
- hygiene
- TODOs
AllocatorLedger:
- created private interface for child allocators to communicate with parents
AllocationReservation:
- created
AtomicRemainder:
- TODOs
BaseAllocator:
- created common base class for buffer allocators
BaseDataValueVector
- released the data buffer in close()
BitVector
- fixed a bug in splitAndTransferTo(), which was missing a retain() call
after
slicing a buffer for the transfer target.
BootStrapContext:
- hygiene
BroadcastSenderRootExec:
- cleanup() => close()
- hygiene
BufferAllocator:
- replace preallocator with createReservation()
- Closeable -> AutoCloseable, and add throws clause
- required minor changes to several other files
- created DrillCloseables and DrillAutoCloseables to make this easy
- renamed the second form of BufferAllocator.takeOwnership to shareOwnership
- clean up interface definition - no abstract needed
- hygiene
- some javadoc
- TODOs
ChildAllocator
- created
DataServer
- handle failed data transfers due to cancelled closed fragments
DrillAutoCloseables:
- add comment that null argument is supported
- fix a typo in a comment
Drillbit:
- sychhronize ShutdownThread.run()'s action against a static to avoid
concurrent shutdown problems with metrics by serializing the shutdown
of multiple drillbits running in the same JVM.
DrillBuf:
- add some finals
- minor formatting
- fix spelling mixtake
- TODOs
DrillConfig:
- hygiene
DrillRuleSets:
- can't reuse query context from past queries
ExecConstants
- debugging property for memory to ExecConstants
- added ALLOCATION_POLICY property name
ExternalSortBatch:
- only get allocator once, outside loop body
- release buffers obtained from getBuffers()
FakeAllocator:
- switch from preallocator to AllocationReservation
- private logger
- TODOs
FilterRecordBatch:
- made some variables private
FragmentContext:
- getAllocator() shouldn't be @Deprecated
- made logger private
- TODOs
ForemanException:
- removed unused fromThrowable()
- commented out unused logger
- formatting
FragmentSetupException:
- leave out the finals
HashAggTemplate:
- cleanup() -> close()
- hygiene
MappifyUtility:
- remove unused imports
MergeJoinBatchBuilder:
- replace preallocator with AllocationReservation
- implement close() to clean up reservation
MSortTemplate:
- made some variables private
NestedConfig:
- hygiene
OperatorContext:
- removed unused applyFragmentLimits member
- made logger private
- TODOs
OutOfMemoryException:
- add serialVersionUID to silence warning
- minor formatting
OutOfMemoryRuntimeException:
- add serialVersionUID to silence warning
- minor formatting
PartitionSenderRootExec:
- cleanup() -> close()
- hygiene
PooledByteBufAllocatorL:
- finals
- formatting
- make logger private (and then comment out because it is unused)
PrintingResultsListener:
- TODOs
PriorityQueueCopierTemplate:
- removed unnecessary use of the pre-allocator
- made some members private
PriorityQueueTemplate:
- removed unnecessary use of preallocator
- made some members private
RecordBatch:
- interface now extends AutoCloseable instead of declaring it's own
cleanup()
- hygiene
- TODOs
RootAllocator:
- created replacement for TopLevelAllocator
ScreenCreator:
- clean() => close()
- hygiene
SimpleRootExec:
- hygiene
- fixup RecordBatch close()
SingleSenderCreator:
- cleanup() => close()
- hygiene
SortRecordBatchBuilder:
- replace preallocation with AllocationReservation
- implements AutoCloseable
- implement close() to close reservation
- clear batches list after closing items on it
- formatting
- hygiene
StackTrace:
- DRILL-2624: fixed numeric formatting for line numbers
TestBaseAllocator:
- created new test for allocators, unit tests them in isolation
TopLevelAllocator:
- replace preallocator with AllocationReservation implementation
- eliminated @Deprecated constructors
- changed many other files to use remaining constructor
- eliminate private constructor
- add finals, private
- formatting
- TODOs
UnionAllRecordBatch:
- added missing null check to clearCurrentRecordBatch()
VectorAccessible:
- hygiene (mostly comments)
ExternalSortBatch:
ExpressionInterpreterTest:
FilterRecordBatch:
HashAggBatch:
HashJoinBatch:
IteratorValidatorBatchIterator:
LimitRecordBatch:
MergeJoinBatch:
MergingRecordBatch:
OrderedPartitionRecordBatch:
ProducerConsumerBatch:
RemovingRecordBatch:
ScanBatch:
SortBatch:
StreamingAggBatch:
StreamingWindowFrameRecordBatch:
TopNBatch:
TraceRecordBatch:
UnionAllRecordBatch:
UnorderedReceiverBatch:
WriterRecordBatch:
- cleanup() -> close()
- close builders that use AllocationReservation
- hygiene
TestAllocators
TestSimpleFragmentRun
TestSimpleFunctions
TestStringFunctions
TestLoad
TestRepeated
TestStringFunctions
... many others...
- modify to use new allocator
- some hygiene
ValueVector and RecordWriter templates:
- added clauses to release buffers before references are overwritten
- eliminated common subexpressions and repeated getAccessor()/getMutator()
calls in loops
- hygiene
commit 169ef67ee3156cd7a3983ecc6cde5c72c86b69e5
Author: Chris Westin <[email protected]>
Date: 2015-07-29T20:16:16Z
DRILL-1942: application of non-locking patch to rebased (20150729) locking
implementation (cwestin/alloc).
- Some manual edits to handle hunks that did not apply cleanly
- inability to allocate needs to throw an exception
- makes TestAllocators run again
- ensure preallocated space tracking is fully protected/transacted under
debug mode in non-locking patch
commit e81b6f4acd914d66501657e558b6e39e41ac1ecc
Author: Chris Westin <[email protected]>
Date: 2015-08-19T16:35:16Z
DRILL-1942-no-locks-retry-stats
- collect stats on retries and report on them at shutdown
commit c6df22d361d79d3db0172f4d11fa98711f97d834
Author: Chris Westin <[email protected]>
Date: 2015-08-25T23:58:08Z
DRILL-1942-no-locks-chunking:
- add secondary chunking to allocation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
