Hi John, I don' think problem is in your data. I guess the problem in select with options. I managed to reproduce similar error locally by querying two csv files: First file (2 records): key_header, value_header key_1,value_1
Second file (500000 records): key_header, value_header key_1,value_1 ....... key_499999,value_499999 Select with options returned the same error: select * from table(dfs.root.`/home/arina/files/ver/*.csv`(type => 'text',extractHeader => true, fieldDelimiter => ',')) limit 10; I have logged Jira for this issue (DRILL-5166) but the root cause should be investigated. Meanwhile, I suggest you to add "extractHeader": true to your storage plugin format and select from file without options. Kind regards Arina On Thu, Dec 8, 2016 at 7:29 PM, John Omernik <[email protected]> wrote: > Hey all, I am trying to do a select with options on a CSV file. select > columns[0], columns[1] already works for this data. Ideally, I am trying > to do a > > select * from table(dfs.root.`path/to/data.csc'(type => 'text', > extractHeader => true, fieldDelimiter => ',') limit 10 and have it work and > read the column names... however, when I do that I get this odd NPE... (The > NPE happens for mytable.*.csv or mytable/myfile.csv) > > This is Drill 1.8. Thanks for any help! > > John > > > > select * from table(`mytable/*.csv`(type => 'text', extractHeader => > true, fieldDelimiter => ',')) limit 10; > > Error: SYSTEM ERROR: NullPointerException > > > Fragment 1:0 > > > [Error Id: dc3f675d-6cca-4ede-8364-8b4bcd0a7c42 on mynode:20045] > > > (com.fasterxml.jackson.databind.JsonMappingException) Instantiation of > [simple type, class org.apache.drill.exec.store.dfs.easy.EasySubScan] > value > failed (java.lang.NullPointerException): null > > at [Source: { > > "pop" : "single-sender", > > "@id" : 0, > > "receiver-major-fragment" : 0, > > "receiver-minor-fragment" : 0, > > "child" : { > > "pop" : "selection-vector-remover", > > "@id" : 1, > > "child" : { > > "pop" : "limit", > > "@id" : 2, > > "child" : { > > "pop" : "fs-sub-scan", > > "@id" : 3, > > "userName" : "jomernik", > > "files" : [ { > > "start" : 268435456, > > "length" : 117805309, > > "path" : "maprfs:/data/datatest/mytable/load1.csv" > > } ], > > "storage" : { > > "type" : "file", > > "enabled" : true, > > "connection" : "maprfs:///", > > "config" : null, > > "workspaces" : { > > "root" : { > > "location" : "/", > > "writable" : false, > > "defaultInputFormat" : null > > }, > > "datatest" : { > > "location" : "/data/datatest", > > "writable" : true, > > "defaultInputFormat" : null > > } > > }, > > "formats" : { > > "psv" : { > > "type" : "text", > > "extensions" : [ "tbl" ], > > "delimiter" : "|" > > }, > > "csv" : { > > "type" : "text", > > "extensions" : [ "csv" ], > > "delimiter" : "," > > }, > > "tsv" : { > > "type" : "text", > > "extensions" : [ "tsv" ], > > "delimiter" : "\t" > > }, > > "parquet" : { > > "type" : "parquet" > > }, > > "json" : { > > "type" : "json", > > "extensions" : [ "json" ] > > }, > > "maprdb" : { > > "type" : "maprdb" > > } > > } > > }, > > "format" : { > > "type" : "named", > > "name" : "text" > > }, > > "columns" : [ "`*`" ], > > "selectionRoot" : "maprfs:/data/datatest/mytable", > > "initialAllocation" : 1000000, > > "maxAllocation" : 10000000000, > > "cost" : 0.0 > > }, > > "first" : 0, > > "last" : 10, > > "initialAllocation" : 1000000, > > "maxAllocation" : 10000000000, > > "cost" : 10.0 > > }, > > "initialAllocation" : 1000000, > > "maxAllocation" : 10000000000, > > "cost" : 10.0 > > }, > > "destination" : "CiVnaXN6ZXRhYWdlbnQwMi5lbGFiLm > JhbmtvZmFtZXJpY2EuY29tEM2cARjOnAEgz5wB", > > "initialAllocation" : 1000000, > > "maxAllocation" : 10000000000, > > "cost" : 10.0 > > }; line: 75, column: 7] (through reference chain: org.apache.drill.exec. > physical.config.SingleSender["child"]->org.apache.drill. > exec.physical.config.SelectionVectorRemover["child" > ]->org.apache.drill.exec. > physical.config.Limit["child"]) > > com.fasterxml.jackson.databind.JsonMappingException.from():223 > > com.fasterxml.jackson.databind.deser.std.StdValueInstantiator. > wrapAsJsonMappingException():445 > > com.fasterxml.jackson.databind.deser.std.StdValueInstantiator. > rewrapCtorProblem():464 > > com.fasterxml.jackson.databind.deser.std.StdValueInstantiator. > createFromObjectWith():258 > > com.fasterxml.jackson.databind.deser.impl. > PropertyBasedCreator.build(): > 135 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeUsingPropertyBased():444 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeFromObjectUsingNonDefault():1123 > > com.fasterxml.jackson.databind.deser.BeanDeserializer. > deserializeFromObject():298 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeWithObjectId():1094 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeOther():166 > > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize() > :135 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer._deserializeTypedForId():120 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer. > deserializeTypedFromObject():91 > > com.fasterxml.jackson.databind.deser.AbstractDeserializer. > deserializeWithType():142 > > com.fasterxml.jackson.databind.deser.SettableBeanProperty. > deserialize():488 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeWithErrorWrapping():465 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeUsingPropertyBased():380 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeFromObjectUsingNonDefault():1123 > > com.fasterxml.jackson.databind.deser.BeanDeserializer. > deserializeFromObject():298 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeWithObjectId():1094 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeOther():166 > > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize() > :135 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer._deserializeTypedForId():120 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer. > deserializeTypedFromObject():91 > > com.fasterxml.jackson.databind.deser.AbstractDeserializer. > deserializeWithType():142 > > com.fasterxml.jackson.databind.deser.SettableBeanProperty. > deserialize():488 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeWithErrorWrapping():465 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeUsingPropertyBased():380 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeFromObjectUsingNonDefault():1123 > > com.fasterxml.jackson.databind.deser.BeanDeserializer. > deserializeFromObject():298 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeWithObjectId():1094 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeOther():166 > > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize() > :135 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer._deserializeTypedForId():120 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer. > deserializeTypedFromObject():91 > > com.fasterxml.jackson.databind.deser.AbstractDeserializer. > deserializeWithType():142 > > com.fasterxml.jackson.databind.deser.SettableBeanProperty. > deserialize():488 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeWithErrorWrapping():465 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeUsingPropertyBased():380 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeFromObjectUsingNonDefault():1123 > > com.fasterxml.jackson.databind.deser.BeanDeserializer. > deserializeFromObject():298 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeWithObjectId():1094 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeOther():166 > > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize() > :135 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer._deserializeTypedForId():120 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer. > deserializeTypedFromObject():91 > > com.fasterxml.jackson.databind.deser.AbstractDeserializer. > deserializeWithType():142 > > com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer. > deserialize():42 > > com.fasterxml.jackson.databind.ObjectReader._bindAndClose():1575 > > com.fasterxml.jackson.databind.ObjectReader.readValue():1200 > > org.apache.drill.exec.planner.PhysicalPlanReader. > readFragmentOperator():94 > > org.apache.drill.exec.work.fragment.FragmentExecutor.run():203 > > org.apache.drill.common.SelfCleaningRunnable.run():38 > > java.util.concurrent.ThreadPoolExecutor.runWorker():1142 > > java.util.concurrent.ThreadPoolExecutor$Worker.run():617 > > java.lang.Thread.run():745 > > Caused By (java.lang.NullPointerException) null > > com.google.common.base.Preconditions.checkNotNull():210 > > org.apache.drill.exec.store.dfs.easy.EasySubScan.<init>():60 > > sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2 > > sun.reflect.NativeConstructorAccessorImpl.newInstance():62 > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45 > > java.lang.reflect.Constructor.newInstance():423 > > com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(): > 114 > > com.fasterxml.jackson.databind.deser.std.StdValueInstantiator. > createFromObjectWith():256 > > com.fasterxml.jackson.databind.deser.impl. > PropertyBasedCreator.build(): > 135 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeUsingPropertyBased():444 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeFromObjectUsingNonDefault():1123 > > com.fasterxml.jackson.databind.deser.BeanDeserializer. > deserializeFromObject():298 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeWithObjectId():1094 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeOther():166 > > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize() > :135 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer._deserializeTypedForId():120 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer. > deserializeTypedFromObject():91 > > com.fasterxml.jackson.databind.deser.AbstractDeserializer. > deserializeWithType():142 > > com.fasterxml.jackson.databind.deser.SettableBeanProperty. > deserialize():488 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeWithErrorWrapping():465 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeUsingPropertyBased():380 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeFromObjectUsingNonDefault():1123 > > com.fasterxml.jackson.databind.deser.BeanDeserializer. > deserializeFromObject():298 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeWithObjectId():1094 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeOther():166 > > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize() > :135 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer._deserializeTypedForId():120 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer. > deserializeTypedFromObject():91 > > com.fasterxml.jackson.databind.deser.AbstractDeserializer. > deserializeWithType():142 > > com.fasterxml.jackson.databind.deser.SettableBeanProperty. > deserialize():488 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeWithErrorWrapping():465 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeUsingPropertyBased():380 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeFromObjectUsingNonDefault():1123 > > com.fasterxml.jackson.databind.deser.BeanDeserializer. > deserializeFromObject():298 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeWithObjectId():1094 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeOther():166 > > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize() > :135 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer._deserializeTypedForId():120 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer. > deserializeTypedFromObject():91 > > com.fasterxml.jackson.databind.deser.AbstractDeserializer. > deserializeWithType():142 > > com.fasterxml.jackson.databind.deser.SettableBeanProperty. > deserialize():488 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeWithErrorWrapping():465 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeUsingPropertyBased():380 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeFromObjectUsingNonDefault():1123 > > com.fasterxml.jackson.databind.deser.BeanDeserializer. > deserializeFromObject():298 > > com.fasterxml.jackson.databind.deser.BeanDeserializerBase. > deserializeWithObjectId():1094 > > com.fasterxml.jackson.databind.deser.BeanDeserializer._ > deserializeOther():166 > > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize() > :135 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer._deserializeTypedForId():120 > > com.fasterxml.jackson.databind.jsontype.impl. > AsPropertyTypeDeserializer. > deserializeTypedFromObject():91 > > com.fasterxml.jackson.databind.deser.AbstractDeserializer. > deserializeWithType():142 > > com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer. > deserialize():42 > > com.fasterxml.jackson.databind.ObjectReader._bindAndClose():1575 > > > com.fasterxml.jackson.databind.ObjectReader.readValue():1200 > > org.apache.drill.exec.planner.PhysicalPlanReader. > readFragmentOperator():94 > > org.apache.drill.exec.work.fragment.FragmentExecutor.run():203 > > org.apache.drill.common.SelfCleaningRunnable.run():38 > > java.util.concurrent.ThreadPoolExecutor.runWorker():1142 > > java.util.concurrent.ThreadPoolExecutor$Worker.run():617 > > java.lang.Thread.run():745 (state=,code=0) >
