Tim Armstrong created IMPALA-5833:
-------------------------------------
Summary: Add automatic validation of memory ownership and transfer
Key: IMPALA-5833
URL: https://issues.apache.org/jira/browse/IMPALA-5833
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Tim Armstrong
Assignee: Tim Armstrong
Currently we have no reliable way to directly detect violations of our memory
ownership or transfer, e.g. if something is attached too early or is freed or
unpinned too early. We can catch many bugs indirectly because they cause
crashes or incorrect results, and some others using tools like
AddressSanitizer, but it would be much better to have a direct way of detecting
the violations that can provide better diagnostics.
The idea is to come up with a way to validate that a RowBatch is only
referencing memory that is valid and owned by the right entity. Currently it's
valid for a RowBatch to reference the following things:
* Any memory directly attached to the batch
* Any memory attached to a batch that is returned from a subsequent GetNext()
call
* Any memory owned by an ExecNode at or below the current node in the plan in
one of the following forms:
* * Buffers/pages
* * MemPools
* * Disk I/O buffers in a ScannerContext.
When validating a RowBatch we could potentially create a map of all valid
memory ranges that the RowBatch is allowed to reference, then traverse the
whole RowBatch and check that all pointers are contained in one of those
ranges. That would be expensive so we would probably have to treat it in the
same way as a tool like AddressSanitizer and run a separate build with it
enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
