Here you go… Thanks for your help!
SELECT
event_date,severity_code,facility_code,severity,facility,ip,app_name,process_id,message_id,structured_data_text,structured_data_UserAgent,structured_data_UserHostAddress,structured_data_BrowserSession,structured_data_Realm,structured_data_Appliance,structured_data_Company,structured_data_UserID,structured_data_PEN,structured_data_HostName,structured_data_Category,structured_data_Priority,message
FROM cp.`syslog/test.syslog1`
18:16:50.061 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0 B(1
B), h: 21.6 MiB(130.9 MiB), nh: 2.0 MiB(82.2 MiB)):
testExplicitFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat)
java.lang.NullPointerException: null
at
org.apache.drill.test.rowSet.RowSetComparison.verifyAndClearAll(RowSetComparison.java:276)
~[drill-java-exec-1.16.0-SNAPSHOT-tests.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.store.syslog.TestSyslogFormat.testExplicitFlattenedStructuredDataQuery(TestSyslogFormat.java:310)
~[test-classes/:na]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
18:16:50.571 [main] ERROR org.apache.drill.TestReporter - Test Failed (d: 0 B(1
B), h: 20.5 MiB(209.9 MiB), nh: 411.1 KiB(85.4 MiB)):
testStarFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat)
java.lang.NullPointerException: null
at
org.apache.drill.test.rowSet.RowSetComparison.verifyAndClearAll(RowSetComparison.java:276)
~[drill-java-exec-1.16.0-SNAPSHOT-tests.jar:1.16.0-SNAPSHOT]
at
org.apache.drill.exec.store.syslog.TestSyslogFormat.testStarFlattenedStructuredDataQuery(TestSyslogFormat.java:248)
~[test-classes/:na]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
[ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 3.173 s
<<< FAILURE! - in org.apache.drill.exec.store.syslog.TestSyslogFormat
[ERROR]
testExplicitFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat)
Time elapsed: 0.234 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.drill.exec.store.syslog.TestSyslogFormat.testExplicitFlattenedStructuredDataQuery(TestSyslogFormat.java:310)
[ERROR]
testStarFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat)
Time elapsed: 0.125 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.drill.exec.store.syslog.TestSyslogFormat.testStarFlattenedStructuredDataQuery(TestSyslogFormat.java:248)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TestSyslogFormat.testExplicitFlattenedStructuredDataQuery:310 »
NullPointer
[ERROR] TestSyslogFormat.testStarFlattenedStructuredDataQuery:248 »
NullPointer
[INFO]
[ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33.919 s
[INFO] Finished at: 2019-01-24T18:16:51-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M2:test (default-test) on
project drill-format-syslog: There are test failures.
[ERROR]
[ERROR] Please refer to
/Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports
for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump,
[date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M2:test (default-test) on
project drill-format-syslog: There are test failures.
Please refer to
/Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports
for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump
and [date].dumpstream.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:81)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch
(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main
(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test
failures.
Please refer to
/Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports
for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump
and [date].dumpstream.
at org.apache.maven.plugin.surefire.SurefireHelper.throwException
(SurefireHelper.java:271)
at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution
(SurefireHelper.java:159)
at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary
(SurefirePlugin.java:373)
at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked
(AbstractSurefireMojo.java:1016)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute
(AbstractSurefireMojo.java:846)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
(DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
(LifecycleModuleBuilder.java:81)
at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
(SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch
(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main
(Launcher.java:356)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
If you run the query in sqline, here is the result:
jdbc:drill:zk=local> select * from dfs.test.`test.syslog1`;
Error: DATA_READ ERROR: Error parsing file
DATA_READ ERROR: Maximum Error Threshold Exceeded:
Line: 1
DATA_WRITE ERROR: null
Could not write string:
index: 0, length: 342 (expected: range(0, 256))
[Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ]
[Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ]
Fragment 0:0
[Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010]
(state=,code=0)
java.sql.SQLException: DATA_READ ERROR: Error parsing file
DATA_READ ERROR: Maximum Error Threshold Exceeded:
Line: 1
DATA_WRITE ERROR: null
Could not write string:
index: 0, length: 342 (expected: range(0, 256))
[Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ]
[Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ]
Fragment 0:0
[Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010]
at
org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
at
org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:608)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1288)
at
org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
at
org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
at
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1107)
at
org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1118)
at
org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
at
org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200)
at
org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
at
org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217)
at sqlline.Commands.execute(Commands.java:938)
at sqlline.Commands.sql(Commands.java:882)
at sqlline.SqlLine.dispatch(SqlLine.java:725)
at sqlline.SqlLine.begin(SqlLine.java:540)
at sqlline.SqlLine.start(SqlLine.java:264)
at sqlline.SqlLine.main(SqlLine.java:195)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: DATA_READ
ERROR: Error parsing file
DATA_READ ERROR: Maximum Error Threshold Exceeded:
Line: 1
DATA_WRITE ERROR: null
Could not write string:
index: 0, length: 342 (expected: range(0, 256))
[Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ]
[Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ]
Fragment 0:0
[Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010]
at
org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
at
org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: DATA_READ ERROR: Error parsing file
DATA_READ ERROR: Maximum Error Threshold Exceeded:
Line: 1
DATA_WRITE ERROR: null
Could not write string:
index: 0, length: 342 (expected: range(0, 256))
[Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ]
[Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ]
Fragment 0:0
[Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
at
org.apache.drill.exec.store.syslog.SyslogRecordReader.next(SyslogRecordReader.java:169)
at
org.apache.drill.exec.physical.impl.ScanBatch.internalNext(ScanBatch.java:223)
at
org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:271)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
at
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
at
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:143)
at
org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
at
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
at
org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:297)
at
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:284)
at .......(:0)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:284)
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
at .......(:0)
If you run a query w/o long fields it works fine:
dbc:drill:zk=local> select * from dfs.test.`logs.syslog1`;
+--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+
| event_date | severity_code | facility_code | severity |
facility | ip | app_name | message_id |
message | process_id |
structured_data_text |
structured_data |
+--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+
| 2003-10-11 22:14:15.003 | 2 | 4 | CRIT | AUTH
| mymachine.example.com | su | ID47 | BOM'su root' failed
for lonvick on /dev/pts/8 | null | null
| {}
|
| 1985-04-12 23:20:50.52 | 2 | 4 | CRIT | AUTH
| mymachine.example.com | su | ID47 | BOM'su root' failed
for lonvick on /dev/pts/8 | null | null
| {}
|
| 1985-04-12 23:20:50.52 | 2 | 4 | CRIT | AUTH
| mymachine.example.com | su | ID47 | BOM'su root' failed
for lonvick on /dev/pts/8 | null | null
| {}
|
| 2003-10-11 22:14:15.003 | 2 | 4 | CRIT | AUTH
| mymachine.example.com | su | ID47 | BOM'su root' failed
for lonvick on /dev/pts/8 | null | null
| {}
|
| 2003-08-24 12:14:15.0 | 2 | 4 | CRIT | AUTH
| mymachine.example.com | su | ID47 | BOM'su root' failed
for lonvick on /dev/pts/8 | null | null
| {}
|
| 2003-08-24 12:14:15.0 | 5 | 20 | NOTICE |
LOCAL4 | 192.0.2.1 | myproc | null | %% It's time to
make the do-nuts. | 8710 | null
| {}
|
| 2003-10-11 22:14:15.003 | 5 | 20 | NOTICE |
LOCAL4 | mymachine.example.com | evntslog | ID47 | null
| null |
{examplePriority@32473=[class=high], exampleSDID@32473=[iut=3,
eventSource=Application, eventID=1011]} |
{"class":"high","iut":"3","eventSource":"Application","eventID":"1011"} |
| 2003-10-11 22:14:15.003 | 5 | 20 | NOTICE |
LOCAL4 | mymachine.example.com | evntslog | ID47 | - and thats a
wrap! | null |
{examplePriority@32473=[class=high], exampleSDID@32473=[iut=3,
eventSource=Application, eventID=1011]} |
{"class":"high","iut":"3","eventSource":"Application","eventID":"1011"} |
+--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+
8 rows selected (1.702 seconds)
> On Jan 24, 2019, at 17:02, Karthikeyan Manivannan <[email protected]>
> wrote:
>
> Hi Charles,
>
> Can you please provide the stack trace.
>
> Thanks
>
> Karthik
>
> On Tue, Jan 22, 2019 at 9:18 PM Charles Givre <[email protected]> wrote:
>
>> Hello all,
>> I’m working on a format plugin to read syslog data, and have encountered
>> what seems to be a bit of a regression (maybe). The code below is a
>> helper function which writes strings from the data. As of Drill 1.16, the
>> varchar holder seems to throw an error if the string you are trying to
>> write is > 256 characters. Is there a workaround?
>>
>> Thanks!
>>
>>
>> //Helper function to map strings
>> private void mapStringField(String name, String value,
>> BaseWriter.MapWriter map) {
>> if (value == null) {
>> return;
>> }
>> try {
>> byte[] bytes = value.getBytes("UTF-8");
>> int stringLength = bytes.length;
>> this.buffer.setBytes(0, bytes, 0, stringLength);
>> map.varChar(name).writeVarChar(0, stringLength, buffer);
>> } catch (Exception e) {
>> throw UserException
>> .dataWriteError()
>> .addContext("Could not write string: ")
>> .addContext(e.getMessage())
>> .build(logger);
>> }
>> }
>>
>>
>>
>>
>>
