Your buffer is allocated as
this.buffer = context.getManagedBuffer();
and that boils down to
BufferManagerImpl.java
@Override
public DrillBuf getManagedBuffer() {
return getManagedBuffer(256);
}
This maybe the root of the problem.
Thanks.
Karthik
On Fri, Jan 25, 2019 at 3:02 PM Karthikeyan Manivannan <[email protected]>
wrote:
> Hi Charles,
>
> In the code that you had pasted
>
> this.buffer.setBytes(0, bytes, 0, stringLength);
>
> what guarantees that this.buffer has enough space for stringLength chars?
>
> Karthik
>
> On Fri, Jan 25, 2019 at 2:32 PM Karthikeyan Manivannan <
> [email protected]> wrote:
>
>> Charles,
>>
>> Does this work on 1.15 ?
>>
>> Drill 1.16 is able to correctly read CSV files with > 256 char strings,
>> so I guess the problem might be in the Syslog plugin code.
>> Can you share your format plugin code ?
>>
>> Thanks.
>>
>> Karthik
>>
>>
>> On Thu, Jan 24, 2019 at 4:24 PM Charles Givre <[email protected]> wrote:
>>
>>> Here you go… Thanks for your help!
>>>
>>>
>>> SELECT
>>> event_date,severity_code,facility_code,severity,facility,ip,app_name,process_id,message_id,structured_data_text,structured_data_UserAgent,structured_data_UserHostAddress,structured_data_BrowserSession,structured_data_Realm,structured_data_Appliance,structured_data_Company,structured_data_UserID,structured_data_PEN,structured_data_HostName,structured_data_Category,structured_data_Priority,message
>>> FROM cp.`syslog/test.syslog1`
>>> 18:16:50.061 [main] ERROR org.apache.drill.TestReporter - Test Failed
>>> (d: 0 B(1 B), h: 21.6 MiB(130.9 MiB), nh: 2.0 MiB(82.2 MiB)):
>>> testExplicitFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat)
>>> java.lang.NullPointerException: null
>>> at
>>> org.apache.drill.test.rowSet.RowSetComparison.verifyAndClearAll(RowSetComparison.java:276)
>>> ~[drill-java-exec-1.16.0-SNAPSHOT-tests.jar:1.16.0-SNAPSHOT]
>>> at
>>> org.apache.drill.exec.store.syslog.TestSyslogFormat.testExplicitFlattenedStructuredDataQuery(TestSyslogFormat.java:310)
>>> ~[test-classes/:na]
>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
>>> 18:16:50.571 [main] ERROR org.apache.drill.TestReporter - Test Failed
>>> (d: 0 B(1 B), h: 20.5 MiB(209.9 MiB), nh: 411.1 KiB(85.4 MiB)):
>>> testStarFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat)
>>> java.lang.NullPointerException: null
>>> at
>>> org.apache.drill.test.rowSet.RowSetComparison.verifyAndClearAll(RowSetComparison.java:276)
>>> ~[drill-java-exec-1.16.0-SNAPSHOT-tests.jar:1.16.0-SNAPSHOT]
>>> at
>>> org.apache.drill.exec.store.syslog.TestSyslogFormat.testStarFlattenedStructuredDataQuery(TestSyslogFormat.java:248)
>>> ~[test-classes/:na]
>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
>>> [ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed:
>>> 3.173 s <<< FAILURE! - in
>>> org.apache.drill.exec.store.syslog.TestSyslogFormat
>>> [ERROR]
>>> testExplicitFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat)
>>> Time elapsed: 0.234 s <<< ERROR!
>>> java.lang.NullPointerException
>>> at
>>> org.apache.drill.exec.store.syslog.TestSyslogFormat.testExplicitFlattenedStructuredDataQuery(TestSyslogFormat.java:310)
>>>
>>> [ERROR]
>>> testStarFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat)
>>> Time elapsed: 0.125 s <<< ERROR!
>>> java.lang.NullPointerException
>>> at
>>> org.apache.drill.exec.store.syslog.TestSyslogFormat.testStarFlattenedStructuredDataQuery(TestSyslogFormat.java:248)
>>>
>>> [INFO]
>>> [INFO] Results:
>>> [INFO]
>>> [ERROR] Errors:
>>> [ERROR] TestSyslogFormat.testExplicitFlattenedStructuredDataQuery:310
>>> » NullPointer
>>> [ERROR] TestSyslogFormat.testStarFlattenedStructuredDataQuery:248 »
>>> NullPointer
>>> [INFO]
>>> [ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0
>>> [INFO]
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 33.919 s
>>> [INFO] Finished at: 2019-01-24T18:16:51-05:00
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M2:test (default-test)
>>> on project drill-format-syslog: There are test failures.
>>> [ERROR]
>>> [ERROR] Please refer to
>>> /Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports
>>> for the individual test results.
>>> [ERROR] Please refer to dump files (if any exist) [date].dump,
>>> [date]-jvmRun[N].dump and [date].dumpstream.
>>> [ERROR] -> [Help 1]
>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
>>> execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M2:test
>>> (default-test) on project drill-format-syslog: There are test failures.
>>>
>>> Please refer to
>>> /Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports
>>> for the individual test results.
>>> Please refer to dump files (if any exist) [date].dump,
>>> [date]-jvmRun[N].dump and [date].dumpstream.
>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>>> (MojoExecutor.java:215)
>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>>> (MojoExecutor.java:156)
>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>>> (MojoExecutor.java:148)
>>> at
>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
>>> (LifecycleModuleBuilder.java:117)
>>> at
>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
>>> (LifecycleModuleBuilder.java:81)
>>> at
>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>>> (SingleThreadedBuilder.java:56)
>>> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
>>> (LifecycleStarter.java:128)
>>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
>>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
>>> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
>>> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
>>> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
>>> at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke
>>> (NativeMethodAccessorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>>> (DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke (Method.java:497)
>>> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
>>> (Launcher.java:289)
>>> at org.codehaus.plexus.classworlds.launcher.Launcher.launch
>>> (Launcher.java:229)
>>> at
>>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
>>> (Launcher.java:415)
>>> at org.codehaus.plexus.classworlds.launcher.Launcher.main
>>> (Launcher.java:356)
>>> Caused by: org.apache.maven.plugin.MojoFailureException: There are test
>>> failures.
>>>
>>> Please refer to
>>> /Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports
>>> for the individual test results.
>>> Please refer to dump files (if any exist) [date].dump,
>>> [date]-jvmRun[N].dump and [date].dumpstream.
>>> at org.apache.maven.plugin.surefire.SurefireHelper.throwException
>>> (SurefireHelper.java:271)
>>> at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution
>>> (SurefireHelper.java:159)
>>> at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary
>>> (SurefirePlugin.java:373)
>>> at
>>> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked
>>> (AbstractSurefireMojo.java:1016)
>>> at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute
>>> (AbstractSurefireMojo.java:846)
>>> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo
>>> (DefaultBuildPluginManager.java:137)
>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>>> (MojoExecutor.java:210)
>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>>> (MojoExecutor.java:156)
>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
>>> (MojoExecutor.java:148)
>>> at
>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
>>> (LifecycleModuleBuilder.java:117)
>>> at
>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject
>>> (LifecycleModuleBuilder.java:81)
>>> at
>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>>> (SingleThreadedBuilder.java:56)
>>> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute
>>> (LifecycleStarter.java:128)
>>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
>>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
>>> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
>>> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
>>> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
>>> at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke
>>> (NativeMethodAccessorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke
>>> (DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke (Method.java:497)
>>> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced
>>> (Launcher.java:289)
>>> at org.codehaus.plexus.classworlds.launcher.Launcher.launch
>>> (Launcher.java:229)
>>> at
>>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode
>>> (Launcher.java:415)
>>> at org.codehaus.plexus.classworlds.launcher.Launcher.main
>>> (Launcher.java:356)
>>> [ERROR]
>>> [ERROR]
>>> [ERROR] For more information about the errors and possible solutions,
>>> please read the following articles:
>>> [ERROR] [Help 1]
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__cwiki.apache.org_confluence_display_MAVEN_MojoFailureException&d=DwIFaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=HlugibuI4IVjs-VMnFvNTcaBtEaDDqE4Ya96cugWqJ8&m=FYH3iU2DaKxg6aW__bi0KbNTl6lqB4g5vyB4-lVHbPk&s=dp6wxwSeKJ-a87PHGpnBjn7aOhg4DNJucCXguyddfMQ&e=
>>>
>>>
>>> If you run the query in sqline, here is the result:
>>>
>>>
>>> jdbc:drill:zk=local> select * from dfs.test.`test.syslog1`;
>>> Error: DATA_READ ERROR: Error parsing file
>>>
>>> DATA_READ ERROR: Maximum Error Threshold Exceeded:
>>>
>>> Line: 1
>>> DATA_WRITE ERROR: null
>>>
>>> Could not write string:
>>> index: 0, length: 342 (expected: range(0, 256))
>>>
>>> [Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ]
>>>
>>> [Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ]
>>> Fragment 0:0
>>>
>>> [Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010]
>>> (state=,code=0)
>>> java.sql.SQLException: DATA_READ ERROR: Error parsing file
>>>
>>> DATA_READ ERROR: Maximum Error Threshold Exceeded:
>>>
>>> Line: 1
>>> DATA_WRITE ERROR: null
>>>
>>> Could not write string:
>>> index: 0, length: 342 (expected: range(0, 256))
>>>
>>> [Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ]
>>>
>>> [Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ]
>>> Fragment 0:0
>>>
>>> [Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010]
>>> at
>>> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536)
>>> at
>>> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:608)
>>> at
>>> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1288)
>>> at
>>> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61)
>>> at
>>> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
>>> at
>>> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1107)
>>> at
>>> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1118)
>>> at
>>> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
>>> at
>>> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200)
>>> at
>>> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
>>> at
>>> org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217)
>>> at sqlline.Commands.execute(Commands.java:938)
>>> at sqlline.Commands.sql(Commands.java:882)
>>> at sqlline.SqlLine.dispatch(SqlLine.java:725)
>>> at sqlline.SqlLine.begin(SqlLine.java:540)
>>> at sqlline.SqlLine.start(SqlLine.java:264)
>>> at sqlline.SqlLine.main(SqlLine.java:195)
>>> Caused by: org.apache.drill.common.exceptions.UserRemoteException:
>>> DATA_READ ERROR: Error parsing file
>>>
>>> DATA_READ ERROR: Maximum Error Threshold Exceeded:
>>>
>>> Line: 1
>>> DATA_WRITE ERROR: null
>>>
>>> Could not write string:
>>> index: 0, length: 342 (expected: range(0, 256))
>>>
>>> [Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ]
>>>
>>> [Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ]
>>> Fragment 0:0
>>>
>>> [Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010]
>>> at
>>> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)
>>> at
>>> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
>>> at
>>> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
>>> at
>>> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
>>> at
>>> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
>>> at
>>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>>> at
>>> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>>> at
>>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>>> at
>>> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
>>> at
>>> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>>> at
>>> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
>>> at
>>> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
>>> at
>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
>>> at
>>> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
>>> at
>>> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
>>> at
>>> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
>>> at
>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.lang.Exception: DATA_READ ERROR: Error parsing file
>>>
>>> DATA_READ ERROR: Maximum Error Threshold Exceeded:
>>>
>>> Line: 1
>>> DATA_WRITE ERROR: null
>>>
>>> Could not write string:
>>> index: 0, length: 342 (expected: range(0, 256))
>>>
>>> [Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ]
>>>
>>> [Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ]
>>> Fragment 0:0
>>>
>>> [Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010]
>>> at
>>> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633)
>>> at
>>> org.apache.drill.exec.store.syslog.SyslogRecordReader.next(SyslogRecordReader.java:169)
>>> at
>>> org.apache.drill.exec.physical.impl.ScanBatch.internalNext(ScanBatch.java:223)
>>> at
>>> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:271)
>>> at
>>> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126)
>>> at
>>> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116)
>>> at
>>> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63)
>>> at
>>> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:143)
>>> at
>>> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186)
>>> at
>>> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
>>> at
>>> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
>>> at
>>> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
>>> at org.apache.drill.exec.work
>>> .fragment.FragmentExecutor$1.run(FragmentExecutor.java:297)
>>> at org.apache.drill.exec.work
>>> .fragment.FragmentExecutor$1.run(FragmentExecutor.java:284)
>>> at .......(:0)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
>>> at org.apache.drill.exec.work
>>> .fragment.FragmentExecutor.run(FragmentExecutor.java:284)
>>> at
>>> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
>>> at .......(:0)
>>>
>>>
>>> If you run a query w/o long fields it works fine:
>>> dbc:drill:zk=local> select * from dfs.test.`logs.syslog1`;
>>>
>>> +--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+
>>> | event_date | severity_code | facility_code | severity
>>> | facility | ip | app_name | message_id |
>>> message | process_id |
>>> structured_data_text |
>>> structured_data |
>>>
>>> +--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+
>>> | 2003-10-11 22:14:15.003 | 2 | 4 | CRIT
>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su
>>> root' failed for lonvick on /dev/pts/8 | null | null
>>> | {}
>>> |
>>> | 1985-04-12 23:20:50.52 | 2 | 4 | CRIT
>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su
>>> root' failed for lonvick on /dev/pts/8 | null | null
>>> | {}
>>> |
>>> | 1985-04-12 23:20:50.52 | 2 | 4 | CRIT
>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su
>>> root' failed for lonvick on /dev/pts/8 | null | null
>>> | {}
>>> |
>>> | 2003-10-11 22:14:15.003 | 2 | 4 | CRIT
>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su
>>> root' failed for lonvick on /dev/pts/8 | null | null
>>> | {}
>>> |
>>> | 2003-08-24 12:14:15.0 | 2 | 4 | CRIT
>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su
>>> root' failed for lonvick on /dev/pts/8 | null | null
>>> | {}
>>> |
>>> | 2003-08-24 12:14:15.0 | 5 | 20 | NOTICE
>>> | LOCAL4 | 192.0.2.1 | myproc | null | %% It's
>>> time to make the do-nuts. | 8710 | null
>>> | {}
>>> |
>>> | 2003-10-11 22:14:15.003 | 5 | 20 | NOTICE
>>> | LOCAL4 | mymachine.example.com | evntslog | ID47 | null
>>> | null |
>>> {examplePriority@32473=[class=high], exampleSDID@32473=[iut=3,
>>> eventSource=Application, eventID=1011]} |
>>> {"class":"high","iut":"3","eventSource":"Application","eventID":"1011"} |
>>> | 2003-10-11 22:14:15.003 | 5 | 20 | NOTICE
>>> | LOCAL4 | mymachine.example.com | evntslog | ID47 | - and
>>> thats a wrap! | null |
>>> {examplePriority@32473=[class=high], exampleSDID@32473=[iut=3,
>>> eventSource=Application, eventID=1011]} |
>>> {"class":"high","iut":"3","eventSource":"Application","eventID":"1011"} |
>>>
>>> +--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+
>>> 8 rows selected (1.702 seconds)
>>>
>>>
>>> > On Jan 24, 2019, at 17:02, Karthikeyan Manivannan <
>>> [email protected]> wrote:
>>> >
>>> > Hi Charles,
>>> >
>>> > Can you please provide the stack trace.
>>> >
>>> > Thanks
>>> >
>>> > Karthik
>>> >
>>> > On Tue, Jan 22, 2019 at 9:18 PM Charles Givre <[email protected]>
>>> wrote:
>>> >
>>> >> Hello all,
>>> >> I’m working on a format plugin to read syslog data, and have
>>> encountered
>>> >> what seems to be a bit of a regression (maybe). The code below is a
>>> >> helper function which writes strings from the data. As of Drill
>>> 1.16, the
>>> >> varchar holder seems to throw an error if the string you are trying to
>>> >> write is > 256 characters. Is there a workaround?
>>> >>
>>> >> Thanks!
>>> >>
>>> >>
>>> >> //Helper function to map strings
>>> >> private void mapStringField(String name, String value,
>>> >> BaseWriter.MapWriter map) {
>>> >> if (value == null) {
>>> >> return;
>>> >> }
>>> >> try {
>>> >> byte[] bytes = value.getBytes("UTF-8");
>>> >> int stringLength = bytes.length;
>>> >> this.buffer.setBytes(0, bytes, 0, stringLength);
>>> >> map.varChar(name).writeVarChar(0, stringLength, buffer);
>>> >> } catch (Exception e) {
>>> >> throw UserException
>>> >> .dataWriteError()
>>> >> .addContext("Could not write string: ")
>>> >> .addContext(e.getMessage())
>>> >> .build(logger);
>>> >> }
>>> >> }
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>>
>>>
