GitHub user dsbos opened a pull request:
https://github.com/apache/drill/pull/245
DRILL-2288: Fix ScanBatch violation of IterOutcome protocol and downstream
chain of bugs
Increments:
2288: Pt. 1 Core: Added unit test.
[Drill2288GetColumnsMetadataWhenNoRowsTest, empty.json]
2288: Pt. 1 Core: Changed HBase test table #1's # of regions from 1 to 2.
[HBaseTestsSuite]
Also added TODO(DRILL-3954) comment about # of regions.
2288: Pt. 2 Core: Documented IterOutcome much more clearly. [RecordBatch]
Also edited some related Javadoc.
2288: Pt. 2 Hyg.: Edited doc., added @Override, etc.
[AbstractRecordBatch, RecordBatch]
Purged unused SetupOutcome.
Added @Override.
Edited comments.
Fix some comments to doc. comments.
2288: Pt. 3 Core&Hyg.: Added validation of IterOutcome sequence.
[IteratorValidatorBatchIterator]
Also:
Renamed internal members for clarity.
Added comments.
2288: Pt. 4 Core: Fixed a NONE -> OK_NEW_SCHEMA in ScanBatch.next().
[ScanBatch]
(With nearby comments.)
2288: Pt. 4 Hyg.: Edited comments, reordered, whitespace. [ScanBatch]
Reordered
Added comments.
Aligned.
2288: Pt. 4 Core+: Fixed UnionAllRecordBatch to receive IterOutcome
sequence right. (3659) [UnionAllRecordBatch]
2288: Pt. 5 Core: Fixed ScanBatch.Mutator.isNewSchema() to stop spurious
"new schema" reports (fix short-circuit OR, to call resetting method right).
[ScanBatch]
2288: Pt. 5 Hyg.: Renamed, edited comments, reordered. [ScanBatch,
SchemaChangeCallBack, AbstractSingleRecordBatch]
Renamed getSchemaChange -> getSchemaChangedAndReset.
Renamed schemaChange -> schemaChanged.
Added doc. comments.
Aligned.
2288: Pt. 6 Core: Avoided dummy Null.IntVec. column in JsonReader when
not needed (MapWriter.isEmptyMap()). [JsonReader, 3 vector files]
2288: Pt. 6 Hyg.: Edited comments, message. Fixed message formatting.
[RecordReader, JSONFormatPlugin, JSONRecordReader, AbstractMapVector,
JsonReader]
Fixed message formatting.
Edited comments.
Edited message.
Fixed spurious line break.
2288: Pt. 7 Core: Added column families in HBaseRecordReader* to avoid
dummy Null.IntVec. clash. [HBaseRecordReader]
2288: Pt. 8 Core.1: Cleared recordCount in
OrderedPartitionRecordBatch.innerNext(). [OrderedPartitionRecordBatch]
2288: Pt. 8 Core.2: Cleared recordCount in ProjectRecordBatch.innerNext.
[ProjectRecordBatch]
2288: Pt. 8 Core.3: Cleared recordCount in TopNBatch.innerNext.
[TopNBatch]
2288: Pt. 9 Core: Had UnorderedReceiverBatch reset RecordBatchLoader's
record count. [UnorderedReceiverBatch, RecordBatchLoader]
2288: Pt. 9 Hyg.: Added comments. [RecordBatchLoader]
2288: Pt. 10 Core: Worked around mismatched map child vectors in
MapVector.getObject(). [MapVector]
2288: Pt. 11 Core: Added OK_NEW_SCHEMA schema comparison for HashAgg.
[HashAggTemplate]
2288: Pt. 12 Core: Fixed memory leak in BaseTestQuery's printing.
Fixed bad skipping of RecordBatchLoader.clear(...) and
QueryDataBatch.load(...) for zero-row batches in printResult(...).
Also, dropped suppression of call to
VectorUtil.showVectorAccessibleContent(...) (so zero-row batches are
as visible as others).
2288: Pt. 13 Core: Fixed test that used unhandled periods in column alias
identifiers.
2288: Misc.: Added # of rows to showVectorAccessibleContent's output.
[VectorUtil]
2288: Misc.: Added simple/partial toString() [VectorContainer,
AbstractRecordReader, JSONRecordReader, BaseValueVector, FieldSelection,
AbstractBaseWriter]
2288: Misc. Hyg.: Added doc. comments to VectorContainer.
[VectorContainer]
2288: Misc. Hyg.: Edited comment. [DrillStringUtils]
2288: Misc. Hyg.: Clarified message for unhandled identifier containing
period.
2288: Pt. 3 Core&Hyg. Upd.: Added schema comparison result to logging.
[IteratorValidatorBatchIterator]
2288: Pt. 7 Core Upd.: Handled HBase columns too re NullableIntVectors.
[HBaseRecordReader, TestTableGenerator, TestHBaseFilterPushDown]
Created map-child vectors for requested columns.
Added unit test method testDummyColumnsAreAvoided, adding new row to test
table,
updated some row counts.
2288: Pt. 7 Hyg. Upd.: Edited comment. [HBaseRecordReader]
2288: Pt. 11 Core Upd.: REVERTED all of bad OK_NEW_SCHEMA schema
comparison for HashAgg. [HashAggTemplate]
This reverts commit 0939660f4620c03da97f4e1bf25a27514e6d0b81.
2288: Pt. 6 Core Upd.: Added isEmptyMap override in new (just-rebased-in)
PromotableWriter. [PromotableWriter]
Adjusted definition and default implementation of isEmptyMap (to handle
MongoDB
storage plugin's use of JsonReader).
2288: Pt. 6 Hyg. Upd.: Purged old atLeastOneWrite flag. [JsonReader]
2288: Pt. 14: Disabled newly dying test testNestedFlatten().
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dsbos/incubator-drill
bugs/drill-2288_etc_SQUASHED
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/245.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #245
----
commit 3d39f7aa3a991b3c9d1d3a7646cc2a9f144005b0
Author: dbarclay <[email protected]>
Date: 2015-10-28T02:25:25Z
DRILL-2288: Fix ScanBatch violation of IterOutcome protocol and downstream
chain of bugs.
Increments:
2288: Pt. 1 Core: Added unit test.
[Drill2288GetColumnsMetadataWhenNoRowsTest, empty.json]
2288: Pt. 1 Core: Changed HBase test table #1's # of regions from 1 to 2.
[HBaseTestsSuite]
Also added TODO(DRILL-3954) comment about # of regions.
2288: Pt. 2 Core: Documented IterOutcome much more clearly. [RecordBatch]
Also edited some related Javadoc.
2288: Pt. 2 Hyg.: Edited doc., added @Override, etc.
[AbstractRecordBatch, RecordBatch]
Purged unused SetupOutcome.
Added @Override.
Edited comments.
Fix some comments to doc. comments.
2288: Pt. 3 Core&Hyg.: Added validation of IterOutcome sequence.
[IteratorValidatorBatchIterator]
Also:
Renamed internal members for clarity.
Added comments.
2288: Pt. 4 Core: Fixed a NONE -> OK_NEW_SCHEMA in ScanBatch.next().
[ScanBatch]
(With nearby comments.)
2288: Pt. 4 Hyg.: Edited comments, reordered, whitespace. [ScanBatch]
Reordered
Added comments.
Aligned.
2288: Pt. 4 Core+: Fixed UnionAllRecordBatch to receive IterOutcome
sequence right. (3659) [UnionAllRecordBatch]
2288: Pt. 5 Core: Fixed ScanBatch.Mutator.isNewSchema() to stop spurious
"new schema" reports (fix short-circuit OR, to call resetting method right).
[ScanBatch]
2288: Pt. 5 Hyg.: Renamed, edited comments, reordered. [ScanBatch,
SchemaChangeCallBack, AbstractSingleRecordBatch]
Renamed getSchemaChange -> getSchemaChangedAndReset.
Renamed schemaChange -> schemaChanged.
Added doc. comments.
Aligned.
2288: Pt. 6 Core: Avoided dummy Null.IntVec. column in JsonReader when
not needed (MapWriter.isEmptyMap()). [JsonReader, 3 vector files]
2288: Pt. 6 Hyg.: Edited comments, message. Fixed message formatting.
[RecordReader, JSONFormatPlugin, JSONRecordReader, AbstractMapVector,
JsonReader]
Fixed message formatting.
Edited comments.
Edited message.
Fixed spurious line break.
2288: Pt. 7 Core: Added column families in HBaseRecordReader* to avoid
dummy Null.IntVec. clash. [HBaseRecordReader]
2288: Pt. 8 Core.1: Cleared recordCount in
OrderedPartitionRecordBatch.innerNext(). [OrderedPartitionRecordBatch]
2288: Pt. 8 Core.2: Cleared recordCount in ProjectRecordBatch.innerNext.
[ProjectRecordBatch]
2288: Pt. 8 Core.3: Cleared recordCount in TopNBatch.innerNext.
[TopNBatch]
2288: Pt. 9 Core: Had UnorderedReceiverBatch reset RecordBatchLoader's
record count. [UnorderedReceiverBatch, RecordBatchLoader]
2288: Pt. 9 Hyg.: Added comments. [RecordBatchLoader]
2288: Pt. 10 Core: Worked around mismatched map child vectors in
MapVector.getObject(). [MapVector]
2288: Pt. 11 Core: Added OK_NEW_SCHEMA schema comparison for HashAgg.
[HashAggTemplate]
2288: Pt. 12 Core: Fixed memory leak in BaseTestQuery's printing.
Fixed bad skipping of RecordBatchLoader.clear(...) and
QueryDataBatch.load(...) for zero-row batches in printResult(...).
Also, dropped suppression of call to
VectorUtil.showVectorAccessibleContent(...) (so zero-row batches are
as visible as others).
2288: Pt. 13 Core: Fixed test that used unhandled periods in column alias
identifiers.
2288: Misc.: Added # of rows to showVectorAccessibleContent's output.
[VectorUtil]
2288: Misc.: Added simple/partial toString() [VectorContainer,
AbstractRecordReader, JSONRecordReader, BaseValueVector, FieldSelection,
AbstractBaseWriter]
2288: Misc. Hyg.: Added doc. comments to VectorContainer.
[VectorContainer]
2288: Misc. Hyg.: Edited comment. [DrillStringUtils]
2288: Misc. Hyg.: Clarified message for unhandled identifier containing
period.
2288: Pt. 3 Core&Hyg. Upd.: Added schema comparison result to logging.
[IteratorValidatorBatchIterator]
2288: Pt. 7 Core Upd.: Handled HBase columns too re NullableIntVectors.
[HBaseRecordReader, TestTableGenerator, TestHBaseFilterPushDown]
Created map-child vectors for requested columns.
Added unit test method testDummyColumnsAreAvoided, adding new row to test
table,
updated some row counts.
2288: Pt. 7 Hyg. Upd.: Edited comment. [HBaseRecordReader]
2288: Pt. 11 Core Upd.: REVERTED all of bad OK_NEW_SCHEMA schema
comparison for HashAgg. [HashAggTemplate]
This reverts commit 0939660f4620c03da97f4e1bf25a27514e6d0b81.
2288: Pt. 6 Core Upd.: Added isEmptyMap override in new (just-rebased-in)
PromotableWriter. [PromotableWriter]
Adjusted definition and default implementation of isEmptyMap (to handle
MongoDB
storage plugin's use of JsonReader).
2288: Pt. 6 Hyg. Upd.: Purged old atLeastOneWrite flag. [JsonReader]
2288: Pt. 14: Disabled newly dying test testNestedFlatten().
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
