Hi Paul,
I fixed the issue with the unit test, updated the PR but now Tavis is timing
out when building Kudu or something not related to my PR. Is there anything I
can do about that?
The job exceeded the maximum time limit for jobs, and has been terminated.
> On Nov 11, 2018, at 14:32, Paul Rogers <par0...@yahoo.com.INVALID> wrote:
>
> Hi Charles,
>
> This error is Drill's long-winded way of saying you have a direct memory
> (value vector) memory leak: you allocated a vector (or buffer) which was
> never passed downstream or freed. Looks like quite a few leaked so should be
> fairly easy to track down.
>
> On your last batch, did you allocate some vectors that ended up not being
> used, maybe?
>
> Thanks,
> - Paul
>
>
>
> On Sunday, November 11, 2018, 6:49:21 AM PST, Charles Givre
> <cgi...@gmail.com> wrote:
>
> Hi Paul,
> Thanks for this. I updated the test and I think I’m close but how I’m
> getting different errors. It looks like it isn’t closing the cluster
> properly, but I’m not quite sure what to do about this.
> — C
>
>
>
> Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.817 sec <<<
> FAILURE! - in org.apache.drill.exec.store.syslog.TestSyslogFormat
> org.apache.drill.exec.store.syslog.TestSyslogFormat Time elapsed: 1.817 sec
> <<< ERROR!
> java.lang.RuntimeException: Exception while closing
> at
> org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:46)
> at org.apache.drill.exec.client.DrillClient.close(DrillClient.java:475)
> at org.apache.drill.test.ClientFixture.close(ClientFixture.java:242)
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
> at org.apache.drill.test.ClusterTest.shutdown(ClusterTest.java:89)
> Caused by: java.lang.IllegalStateException: Allocator[ROOT] closed with
> outstanding buffers allocated (1).
> Allocator(ROOT) 0/2048/6176/4294967296 (res/actual/peak/limit)
> child allocators: 0
> ledgers: 1
> ledger[251] allocator: ROOT), isOwning: true, size: 2048, references: 33,
> life: 811185821181057..0, allocatorManager: [177, life: 811185821154333..0]
> holds 59 buffers.
> DrillBuf[447], udle: [178 1229..1237]
> DrillBuf[470], udle: [178 1366..1367]
> DrillBuf[457], udle: [178 1282..1290]
> DrillBuf[442], udle: [178 1207..1215]
> DrillBuf[315], udle: [178 11..1126]
> DrillBuf[443], udle: [178 1215..1228]
> DrillBuf[465], udle: [178 1350..1351]
> DrillBuf[456], udle: [178 1282..1319]
> DrillBuf[480], udle: [178 1402..1403]
> DrillBuf[452], udle: [178 1262..1270]
> DrillBuf[453], udle: [178 1270..1281]
> DrillBuf[460], udle: [178 1319..1320]
> DrillBuf[473], udle: [178 1375..1380]
> DrillBuf[485], udle: [178 1416..1417]
> DrillBuf[471], udle: [178 1367..1380]
> DrillBuf[463], udle: [178 1328..1350]
> DrillBuf[474], udle: [178 1380..1402]
> DrillBuf[459], udle: [178 1319..1350]
> DrillBuf[461], udle: [178 1320..1350]
> DrillBuf[468], udle: [178 1359..1366]
> DrillBuf[441], udle: [178 1207..1228]
> DrillBuf[488], udle: [178 1425..1426]
> DrillBuf[313], udle: [178 0..2048]
> DrillBuf[433], udle: [178 1129..1426]
> DrillBuf[486], udle: [178 1417..1426]
> DrillBuf[436], udle: [178 1130..1206]
> DrillBuf[464], udle: [178 1350..1366]
> DrillBuf[444], udle: [178 1228..1261]
> DrillBuf[466], udle: [178 1351..1366]
> DrillBuf[469], udle: [178 1366..1380]
> DrillBuf[435], udle: [178 1129..1130]
> DrillBuf[449], udle: [178 1261..1281]
> DrillBuf[458], udle: [178 1290..1319]
> DrillBuf[484], udle: [178 1416..1426]
> DrillBuf[454], udle: [178 1281..1319]
> DrillBuf[434], udle: [178 1129..1206]
> DrillBuf[467], udle: [178 1351..1359]
> DrillBuf[317], udle: [178 1129..1426]
> DrillBuf[438], udle: [178 1138..1206]
> DrillBuf[448], udle: [178 1237..1261]
> DrillBuf[462], udle: [178 1320..1328]
> DrillBuf[450], udle: [178 1261..1262]
> DrillBuf[476], udle: [178 1381..1402]
> DrillBuf[478], udle: [178 1389..1402]
> DrillBuf[479], udle: [178 1402..1416]
> DrillBuf[455], udle: [178 1281..1282]
> DrillBuf[475], udle: [178 1380..1381]
> DrillBuf[439], udle: [178 1206..1228]
> DrillBuf[440], udle: [178 1206..1207]
> DrillBuf[477], udle: [178 1381..1389]
> DrillBuf[437], udle: [178 1130..1138]
> DrillBuf[446], udle: [178 1229..1261]
> DrillBuf[451], udle: [178 1262..1281]
> DrillBuf[487], udle: [178 1417..1425]
> DrillBuf[472], udle: [178 1367..1375]
> DrillBuf[482], udle: [178 1403..1411]
> DrillBuf[483], udle: [178 1411..1416]
> DrillBuf[481], udle: [178 1403..1416]
> DrillBuf[445], udle: [178 1228..1229]
> reservations: 0
>
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:503)
> at
> org.apache.drill.common.DrillAutoCloseables.closeNoChecked(DrillAutoCloseables.java:44)
> at org.apache.drill.exec.client.DrillClient.close(DrillClient.java:475)
> at org.apache.drill.test.ClientFixture.close(ClientFixture.java:242)
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81)
> at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69)
> at org.apache.drill.test.ClusterTest.shutdown(ClusterTest.java:89)
>
>
> Results :
>
> Tests in error:
> TestSyslogFormat>ClusterTest.shutdown:89 » Runtime Exception while closing
>
>
>> On Nov 9, 2018, at 16:09, Paul Rogers <par0...@yahoo.com.INVALID> wrote:
>>
>> Hi Charles,
>>
>> Thanks for the PR. Two suggestions for your test. First, use TupleSchema:
>>
>> TupleSchema schema = new SchemaBuilder() ... .buildSchema().
>>
>> BatchSchema has some limitations that TupleSchema overcomes.
>>
>> Second, when I did a PR that added unions, I normalized the "buildFoo()"
>> methods. They are now called .resumeSchema(). Checkout the test
>> TestSchemaBuilder for an example.
>>
>> Just checked, the example in the SchemaBuilder class shows the old names.
>> Please file a JIRA ticket for this and I'll submit a PR with the fix.
>>
>> Thanks,
>> - Paul
>>
>>
>>
>> On Friday, November 9, 2018, 12:52:56 PM PST, Charles Givre
>> <cgi...@gmail.com> wrote:
>>
>> Hello everyone,
>> I just submitted a PR for a format plugin for Syslog formatted data. I’m
>> really stuck on writing a unit test however and could use some assistance.
>> Basically I’m stuck writing a unit test where the data returns a map. I
>> attempted to follow the sample code shown below, however, Java doesn’t seem
>> to be finding the build() or buildMap() functions and hence the unit test
>> doesn’t work.
>>
>> <p>
>> * Code:<pre><code>
>> * BatchSchema batchSchema = new SchemaBuilder()
>> * .add("c", MinorType.INT)
>> * .addMap("a")
>> * .addNullable("b", MinorType.VARCHAR)
>> * .add("d", MinorType.INT)
>> * .addMap("e") // or .addMapArray("e")
>> * .add("f", MinorType.VARCHAR)
>> * .buildMap()
>> * .add("g", MinorType.INT)
>> * .buildMap()
>> * .addUnion("h") // or .addList("h")
>> * .addType(MinorType.INT)
>> * .addMap()
>> * .add("h1", MinorType.INT)
>> * .buildNested()
>> * .addList()
>> * .addType(MinorType.BIGINT)
>> * .buildNested()
>> * .build()
>> * .addArray("i", MinorType.BIGINT)
>> * .addRepeatedList("j")
>> * .addDimension()
>> * .addArray(MinorType.VARCHAR)
>> * .endDimension()
>> * .build()
>> * .build();
>> * </code</pre>
>>
>> However, it would not work. The test file can be found here:
>> https://github.com/apache/drill/blob/a2365042ee0baa73698f1bb9dd183bb4623bc254/contrib/format-syslog/src/test/java/org/apache/drill/exec/store/syslog/TestSyslogFormat.java
>>
>> <https://github.com/apache/drill/blob/a2365042ee0baa73698f1bb9dd183bb4623bc254/contrib/format-syslog/src/test/java/org/apache/drill/exec/store/syslog/TestSyslogFormat.java>.
>> I commented out the non-working test. Any help would be greatly
>> appreciated.
>> Thanks!
>> —C
>>
