jean-claude created DRILL-6796:
----------------------------------
Summary: Record reader with batch size larger than 0x4000 fail
with an assertion error
Key: DRILL-6796
URL: https://issues.apache.org/jira/browse/DRILL-6796
Project: Apache Drill
Issue Type: Bug
Reporter: jean-claude
@Test
public void testArrayOfArrayJson() throws Exception {
try (OutputStreamWriter w = new OutputStreamWriter(new FileOutputStream(new
File(testDir, "test.json")))) {
w.write("\{\"arrayOfArray\":[[1],[1,2]]}\n");
for (int i = 0; i < JSONRecordReader.DEFAULT_ROWS_PER_BATCH; i++) {
w.write("\{\"anInt\":1}\n");
}
}
LogFixtureBuilder logBuilder = LogFixture.builder()
// Log to the console for debugging convenience
.toConsole().logger("org.apache.drill.exec", Level.TRACE);
try (LogFixture logs = logBuilder.build()) {
String sql = "select root.arrayOfArray[0][0] as w from
`dfs.data`.`test.json` as root";
rowSetIterator = client.queryBuilder().sql(sql).rowSetIterator();
schemaBuilder = new SchemaBuilder();
schemaBuilder.add("w", TypeProtos.MinorType.BIGINT, DataMode.OPTIONAL);
expectedSchema = schemaBuilder.buildSchema();
DirectRowSet batch1 = nextRowSet();
rowSetBuilder = newRowSetBuilder();
rowSetBuilder.addRow(1L);
for (int i = 0; i < JSONRecordReader.DEFAULT_ROWS_PER_BATCH - 1; i++) {
rowSetBuilder.addRow(new Object[] \{ null });
}
verify(rowSetBuilder.build(), batch1);
DirectRowSet batch2 = nextRowSet();
rowSetBuilder = newRowSetBuilder();
rowSetBuilder.addRow(new Object[] \{ null });
verify(rowSetBuilder.build(), batch2);
}
}
The test passes. Then I change the JsonRecordReader batch size
public static final long DEFAULT_ROWS_PER_BATCH =
BaseValueVector.INITIAL_VALUE_ALLOCATION ;
to be
public static final long DEFAULT_ROWS_PER_BATCH =
BaseValueVector.INITIAL_VALUE_ALLOCATION + 1;
and the test case fails.
This problem was noticed when writing a msgpack reader. If I use a size larger
than 0x4000 I would get the same error as with the JSON reader.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
