HI Karthik, Thanks for your response. This was working in Drill 1.15 when I submitted the original PR. I updated the code for Drill 1.16 and suddenly the unit tests started failing and I couldn’t figure out why. After some digging, I found that the buffer situation was the issue. I was under the impression that the getManagedBuffer() function would allocate and/or reallocate space as needed but I guess that is not the case. Did something change with the BufferManagerImpl class for Drill 1.16?
In any event, what would the best practice be? Would you suggest just setting the buffer size to a large value and hope for the best? Thanks! — C > On Jan 25, 2019, at 20:17, Karthikeyan Manivannan <[email protected]> > wrote: > > Your buffer is allocated as > > this.buffer = context.getManagedBuffer(); > > and that boils down to > > BufferManagerImpl.java > > @Override > public DrillBuf getManagedBuffer() { > return getManagedBuffer(256); > } > > This maybe the root of the problem. > > Thanks. > > Karthik > > On Fri, Jan 25, 2019 at 3:02 PM Karthikeyan Manivannan <[email protected]> > wrote: > >> Hi Charles, >> >> In the code that you had pasted >> >> this.buffer.setBytes(0, bytes, 0, stringLength); >> >> what guarantees that this.buffer has enough space for stringLength chars? >> >> Karthik >> >> On Fri, Jan 25, 2019 at 2:32 PM Karthikeyan Manivannan < >> [email protected]> wrote: >> >>> Charles, >>> >>> Does this work on 1.15 ? >>> >>> Drill 1.16 is able to correctly read CSV files with > 256 char strings, >>> so I guess the problem might be in the Syslog plugin code. >>> Can you share your format plugin code ? >>> >>> Thanks. >>> >>> Karthik >>> >>> >>> On Thu, Jan 24, 2019 at 4:24 PM Charles Givre <[email protected]> wrote: >>> >>>> Here you go… Thanks for your help! >>>> >>>> >>>> SELECT >>>> event_date,severity_code,facility_code,severity,facility,ip,app_name,process_id,message_id,structured_data_text,structured_data_UserAgent,structured_data_UserHostAddress,structured_data_BrowserSession,structured_data_Realm,structured_data_Appliance,structured_data_Company,structured_data_UserID,structured_data_PEN,structured_data_HostName,structured_data_Category,structured_data_Priority,message >>>> FROM cp.`syslog/test.syslog1` >>>> 18:16:50.061 [main] ERROR org.apache.drill.TestReporter - Test Failed >>>> (d: 0 B(1 B), h: 21.6 MiB(130.9 MiB), nh: 2.0 MiB(82.2 MiB)): >>>> testExplicitFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat) >>>> java.lang.NullPointerException: null >>>> at >>>> org.apache.drill.test.rowSet.RowSetComparison.verifyAndClearAll(RowSetComparison.java:276) >>>> ~[drill-java-exec-1.16.0-SNAPSHOT-tests.jar:1.16.0-SNAPSHOT] >>>> at >>>> org.apache.drill.exec.store.syslog.TestSyslogFormat.testExplicitFlattenedStructuredDataQuery(TestSyslogFormat.java:310) >>>> ~[test-classes/:na] >>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] >>>> 18:16:50.571 [main] ERROR org.apache.drill.TestReporter - Test Failed >>>> (d: 0 B(1 B), h: 20.5 MiB(209.9 MiB), nh: 411.1 KiB(85.4 MiB)): >>>> testStarFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat) >>>> java.lang.NullPointerException: null >>>> at >>>> org.apache.drill.test.rowSet.RowSetComparison.verifyAndClearAll(RowSetComparison.java:276) >>>> ~[drill-java-exec-1.16.0-SNAPSHOT-tests.jar:1.16.0-SNAPSHOT] >>>> at >>>> org.apache.drill.exec.store.syslog.TestSyslogFormat.testStarFlattenedStructuredDataQuery(TestSyslogFormat.java:248) >>>> ~[test-classes/:na] >>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] >>>> [ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: >>>> 3.173 s <<< FAILURE! - in >>>> org.apache.drill.exec.store.syslog.TestSyslogFormat >>>> [ERROR] >>>> testExplicitFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat) >>>> Time elapsed: 0.234 s <<< ERROR! >>>> java.lang.NullPointerException >>>> at >>>> org.apache.drill.exec.store.syslog.TestSyslogFormat.testExplicitFlattenedStructuredDataQuery(TestSyslogFormat.java:310) >>>> >>>> [ERROR] >>>> testStarFlattenedStructuredDataQuery(org.apache.drill.exec.store.syslog.TestSyslogFormat) >>>> Time elapsed: 0.125 s <<< ERROR! >>>> java.lang.NullPointerException >>>> at >>>> org.apache.drill.exec.store.syslog.TestSyslogFormat.testStarFlattenedStructuredDataQuery(TestSyslogFormat.java:248) >>>> >>>> [INFO] >>>> [INFO] Results: >>>> [INFO] >>>> [ERROR] Errors: >>>> [ERROR] TestSyslogFormat.testExplicitFlattenedStructuredDataQuery:310 >>>> » NullPointer >>>> [ERROR] TestSyslogFormat.testStarFlattenedStructuredDataQuery:248 » >>>> NullPointer >>>> [INFO] >>>> [ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0 >>>> [INFO] >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] BUILD FAILURE >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [INFO] Total time: 33.919 s >>>> [INFO] Finished at: 2019-01-24T18:16:51-05:00 >>>> [INFO] >>>> ------------------------------------------------------------------------ >>>> [ERROR] Failed to execute goal >>>> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M2:test (default-test) >>>> on project drill-format-syslog: There are test failures. >>>> [ERROR] >>>> [ERROR] Please refer to >>>> /Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports >>>> for the individual test results. >>>> [ERROR] Please refer to dump files (if any exist) [date].dump, >>>> [date]-jvmRun[N].dump and [date].dumpstream. >>>> [ERROR] -> [Help 1] >>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to >>>> execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M2:test >>>> (default-test) on project drill-format-syslog: There are test failures. >>>> >>>> Please refer to >>>> /Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports >>>> for the individual test results. >>>> Please refer to dump files (if any exist) [date].dump, >>>> [date]-jvmRun[N].dump and [date].dumpstream. >>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute >>>> (MojoExecutor.java:215) >>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute >>>> (MojoExecutor.java:156) >>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute >>>> (MojoExecutor.java:148) >>>> at >>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject >>>> (LifecycleModuleBuilder.java:117) >>>> at >>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject >>>> (LifecycleModuleBuilder.java:81) >>>> at >>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build >>>> (SingleThreadedBuilder.java:56) >>>> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute >>>> (LifecycleStarter.java:128) >>>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) >>>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) >>>> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) >>>> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) >>>> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) >>>> at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke >>>> (NativeMethodAccessorImpl.java:62) >>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke >>>> (DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke (Method.java:497) >>>> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced >>>> (Launcher.java:289) >>>> at org.codehaus.plexus.classworlds.launcher.Launcher.launch >>>> (Launcher.java:229) >>>> at >>>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode >>>> (Launcher.java:415) >>>> at org.codehaus.plexus.classworlds.launcher.Launcher.main >>>> (Launcher.java:356) >>>> Caused by: org.apache.maven.plugin.MojoFailureException: There are test >>>> failures. >>>> >>>> Please refer to >>>> /Users/cgivre/github/drill-dev/drill/contrib/format-syslog/target/surefire-reports >>>> for the individual test results. >>>> Please refer to dump files (if any exist) [date].dump, >>>> [date]-jvmRun[N].dump and [date].dumpstream. >>>> at org.apache.maven.plugin.surefire.SurefireHelper.throwException >>>> (SurefireHelper.java:271) >>>> at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution >>>> (SurefireHelper.java:159) >>>> at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary >>>> (SurefirePlugin.java:373) >>>> at >>>> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked >>>> (AbstractSurefireMojo.java:1016) >>>> at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute >>>> (AbstractSurefireMojo.java:846) >>>> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo >>>> (DefaultBuildPluginManager.java:137) >>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute >>>> (MojoExecutor.java:210) >>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute >>>> (MojoExecutor.java:156) >>>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute >>>> (MojoExecutor.java:148) >>>> at >>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject >>>> (LifecycleModuleBuilder.java:117) >>>> at >>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject >>>> (LifecycleModuleBuilder.java:81) >>>> at >>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build >>>> (SingleThreadedBuilder.java:56) >>>> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute >>>> (LifecycleStarter.java:128) >>>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) >>>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) >>>> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) >>>> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) >>>> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) >>>> at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke >>>> (NativeMethodAccessorImpl.java:62) >>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke >>>> (DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke (Method.java:497) >>>> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced >>>> (Launcher.java:289) >>>> at org.codehaus.plexus.classworlds.launcher.Launcher.launch >>>> (Launcher.java:229) >>>> at >>>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode >>>> (Launcher.java:415) >>>> at org.codehaus.plexus.classworlds.launcher.Launcher.main >>>> (Launcher.java:356) >>>> [ERROR] >>>> [ERROR] >>>> [ERROR] For more information about the errors and possible solutions, >>>> please read the following articles: >>>> [ERROR] [Help 1] >>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__cwiki.apache.org_confluence_display_MAVEN_MojoFailureException&d=DwIFaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=HlugibuI4IVjs-VMnFvNTcaBtEaDDqE4Ya96cugWqJ8&m=FYH3iU2DaKxg6aW__bi0KbNTl6lqB4g5vyB4-lVHbPk&s=dp6wxwSeKJ-a87PHGpnBjn7aOhg4DNJucCXguyddfMQ&e= >>>> >>>> >>>> If you run the query in sqline, here is the result: >>>> >>>> >>>> jdbc:drill:zk=local> select * from dfs.test.`test.syslog1`; >>>> Error: DATA_READ ERROR: Error parsing file >>>> >>>> DATA_READ ERROR: Maximum Error Threshold Exceeded: >>>> >>>> Line: 1 >>>> DATA_WRITE ERROR: null >>>> >>>> Could not write string: >>>> index: 0, length: 342 (expected: range(0, 256)) >>>> >>>> [Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ] >>>> >>>> [Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ] >>>> Fragment 0:0 >>>> >>>> [Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010] >>>> (state=,code=0) >>>> java.sql.SQLException: DATA_READ ERROR: Error parsing file >>>> >>>> DATA_READ ERROR: Maximum Error Threshold Exceeded: >>>> >>>> Line: 1 >>>> DATA_WRITE ERROR: null >>>> >>>> Could not write string: >>>> index: 0, length: 342 (expected: range(0, 256)) >>>> >>>> [Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ] >>>> >>>> [Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ] >>>> Fragment 0:0 >>>> >>>> [Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010] >>>> at >>>> org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:536) >>>> at >>>> org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:608) >>>> at >>>> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1288) >>>> at >>>> org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:61) >>>> at >>>> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667) >>>> at >>>> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1107) >>>> at >>>> org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1118) >>>> at >>>> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) >>>> at >>>> org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200) >>>> at >>>> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) >>>> at >>>> org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217) >>>> at sqlline.Commands.execute(Commands.java:938) >>>> at sqlline.Commands.sql(Commands.java:882) >>>> at sqlline.SqlLine.dispatch(SqlLine.java:725) >>>> at sqlline.SqlLine.begin(SqlLine.java:540) >>>> at sqlline.SqlLine.start(SqlLine.java:264) >>>> at sqlline.SqlLine.main(SqlLine.java:195) >>>> Caused by: org.apache.drill.common.exceptions.UserRemoteException: >>>> DATA_READ ERROR: Error parsing file >>>> >>>> DATA_READ ERROR: Maximum Error Threshold Exceeded: >>>> >>>> Line: 1 >>>> DATA_WRITE ERROR: null >>>> >>>> Could not write string: >>>> index: 0, length: 342 (expected: range(0, 256)) >>>> >>>> [Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ] >>>> >>>> [Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ] >>>> Fragment 0:0 >>>> >>>> [Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010] >>>> at >>>> org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123) >>>> at >>>> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422) >>>> at >>>> org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96) >>>> at >>>> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273) >>>> at >>>> org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243) >>>> at >>>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >>>> at >>>> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >>>> at >>>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >>>> at >>>> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312) >>>> at >>>> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >>>> at >>>> io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) >>>> at >>>> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) >>>> at >>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) >>>> at >>>> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) >>>> at >>>> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) >>>> at >>>> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) >>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) >>>> at >>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.lang.Exception: DATA_READ ERROR: Error parsing file >>>> >>>> DATA_READ ERROR: Maximum Error Threshold Exceeded: >>>> >>>> Line: 1 >>>> DATA_WRITE ERROR: null >>>> >>>> Could not write string: >>>> index: 0, length: 342 (expected: range(0, 256)) >>>> >>>> [Error Id: 39ac3038-63b7-4190-bcc7-e2bdda924021 ] >>>> >>>> [Error Id: 84807daf-ae51-4a2b-ba2a-1729facdb891 ] >>>> Fragment 0:0 >>>> >>>> [Error Id: 6054aa6b-91df-43b5-8157-a45b4e316e7f on 192.168.1.21:31010] >>>> at >>>> org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) >>>> at >>>> org.apache.drill.exec.store.syslog.SyslogRecordReader.next(SyslogRecordReader.java:169) >>>> at >>>> org.apache.drill.exec.physical.impl.ScanBatch.internalNext(ScanBatch.java:223) >>>> at >>>> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:271) >>>> at >>>> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:126) >>>> at >>>> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:116) >>>> at >>>> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:63) >>>> at >>>> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:143) >>>> at >>>> org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:186) >>>> at >>>> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) >>>> at >>>> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83) >>>> at >>>> org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) >>>> at org.apache.drill.exec.work >>>> .fragment.FragmentExecutor$1.run(FragmentExecutor.java:297) >>>> at org.apache.drill.exec.work >>>> .fragment.FragmentExecutor$1.run(FragmentExecutor.java:284) >>>> at .......(:0) >>>> at >>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) >>>> at org.apache.drill.exec.work >>>> .fragment.FragmentExecutor.run(FragmentExecutor.java:284) >>>> at >>>> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) >>>> at .......(:0) >>>> >>>> >>>> If you run a query w/o long fields it works fine: >>>> dbc:drill:zk=local> select * from dfs.test.`logs.syslog1`; >>>> >>>> +--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+ >>>> | event_date | severity_code | facility_code | severity >>>> | facility | ip | app_name | message_id | >>>> message | process_id | >>>> structured_data_text | >>>> structured_data | >>>> >>>> +--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+ >>>> | 2003-10-11 22:14:15.003 | 2 | 4 | CRIT >>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su >>>> root' failed for lonvick on /dev/pts/8 | null | null >>>> | {} >>>> | >>>> | 1985-04-12 23:20:50.52 | 2 | 4 | CRIT >>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su >>>> root' failed for lonvick on /dev/pts/8 | null | null >>>> | {} >>>> | >>>> | 1985-04-12 23:20:50.52 | 2 | 4 | CRIT >>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su >>>> root' failed for lonvick on /dev/pts/8 | null | null >>>> | {} >>>> | >>>> | 2003-10-11 22:14:15.003 | 2 | 4 | CRIT >>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su >>>> root' failed for lonvick on /dev/pts/8 | null | null >>>> | {} >>>> | >>>> | 2003-08-24 12:14:15.0 | 2 | 4 | CRIT >>>> | AUTH | mymachine.example.com | su | ID47 | BOM'su >>>> root' failed for lonvick on /dev/pts/8 | null | null >>>> | {} >>>> | >>>> | 2003-08-24 12:14:15.0 | 5 | 20 | NOTICE >>>> | LOCAL4 | 192.0.2.1 | myproc | null | %% It's >>>> time to make the do-nuts. | 8710 | null >>>> | {} >>>> | >>>> | 2003-10-11 22:14:15.003 | 5 | 20 | NOTICE >>>> | LOCAL4 | mymachine.example.com | evntslog | ID47 | null >>>> | null | >>>> {examplePriority@32473=[class=high], exampleSDID@32473=[iut=3, >>>> eventSource=Application, eventID=1011]} | >>>> {"class":"high","iut":"3","eventSource":"Application","eventID":"1011"} | >>>> | 2003-10-11 22:14:15.003 | 5 | 20 | NOTICE >>>> | LOCAL4 | mymachine.example.com | evntslog | ID47 | - and >>>> thats a wrap! | null | >>>> {examplePriority@32473=[class=high], exampleSDID@32473=[iut=3, >>>> eventSource=Application, eventID=1011]} | >>>> {"class":"high","iut":"3","eventSource":"Application","eventID":"1011"} | >>>> >>>> +--------------------------+----------------+----------------+-----------+-----------+------------------------+-----------+-------------+------------------------------------------------+-------------+----------------------------------------------------------------------------------+--------------------------------------------------------------------------+ >>>> 8 rows selected (1.702 seconds) >>>> >>>> >>>>> On Jan 24, 2019, at 17:02, Karthikeyan Manivannan < >>>> [email protected]> wrote: >>>>> >>>>> Hi Charles, >>>>> >>>>> Can you please provide the stack trace. >>>>> >>>>> Thanks >>>>> >>>>> Karthik >>>>> >>>>> On Tue, Jan 22, 2019 at 9:18 PM Charles Givre <[email protected]> >>>> wrote: >>>>> >>>>>> Hello all, >>>>>> I’m working on a format plugin to read syslog data, and have >>>> encountered >>>>>> what seems to be a bit of a regression (maybe). The code below is a >>>>>> helper function which writes strings from the data. As of Drill >>>> 1.16, the >>>>>> varchar holder seems to throw an error if the string you are trying to >>>>>> write is > 256 characters. Is there a workaround? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> //Helper function to map strings >>>>>> private void mapStringField(String name, String value, >>>>>> BaseWriter.MapWriter map) { >>>>>> if (value == null) { >>>>>> return; >>>>>> } >>>>>> try { >>>>>> byte[] bytes = value.getBytes("UTF-8"); >>>>>> int stringLength = bytes.length; >>>>>> this.buffer.setBytes(0, bytes, 0, stringLength); >>>>>> map.varChar(name).writeVarChar(0, stringLength, buffer); >>>>>> } catch (Exception e) { >>>>>> throw UserException >>>>>> .dataWriteError() >>>>>> .addContext("Could not write string: ") >>>>>> .addContext(e.getMessage()) >>>>>> .build(logger); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>>
