[
https://issues.apache.org/jira/browse/DRILL-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485619#comment-14485619
]
Jason Altekruse commented on DRILL-1460:
----------------------------------------
[~dragoncurve] I'm sorry I haven't gotten back to this for so long. I actually
put up a discussion on the mailing list a few months ago to just change the
default to reading all values as double by default, no configuration needed.
These datasets are far too common to have anything else as a default. We do
appreciate the patch, but I really think there is very little value in the
current behavior, that is until Drill supports the concept of changing type
(mentioned in the discussion thread linked below). I am going to make a simpler
patch that just changes the reader to always read the values as doubles and
post it here for review (this will simplify the JSON reader as well as avoid
further cluttering the config options). For anyone with requirements to read
the numbers as integers or bigints for precision the option to use
all_text_mode with casts will still be there.
Again thanks for the work, this is something we have been needing to solve for
a while. Please do check back to review the new patch.
http://mail-archives.apache.org/mod_mbox/drill-dev/201501.mbox/%3CCAMpYv7Bd4sScGdefFsGyZq8bU%3DQH4Tg%2BvYMc8m-BUOz09pp2Dg%40mail.gmail.com%3E
> JsonReader fails reading files with decimal numbers and integers in the same
> field
> ----------------------------------------------------------------------------------
>
> Key: DRILL-1460
> URL: https://issues.apache.org/jira/browse/DRILL-1460
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JSON
> Affects Versions: 0.6.0, 0.7.0
> Reporter: Bhallamudi Venkata Siva Kamesh
> Assignee: Jason Altekruse
> Fix For: 0.9.0
>
> Attachments: DRILL-1460.1.patch.txt, DRILL-1460.2.patch.txt
>
>
> Used the following dataset :
> http://thecodebarbarian.wordpress.com/2014/03/28/plugging-usda-nutrition-data-into-mongodb
> Executed the following query
> {noformat}select t.nutrients from dfs.usda.`usda.json` t limit 1;{noformat}
> and it failed with following exception
> {noformat}
> 2014-09-27 17:48:39,421 [b9dfbb9b-29a9-425d-801c-2e418533525f:frag:0:0] ERROR
> o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error
> 0568d90a-d7df-4a5d-87e9-8b9f718dffa4: Screen received stop request sent.
> java.lang.IllegalArgumentException: You tried to write a BigInt type when you
> are using a ValueWriter of type NullableFloat8WriterImpl.
> at
> org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:513)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:145)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.impl.NullableFloat8WriterImpl.write(NullableFloat8WriterImpl.java:88)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:257)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:310)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:204)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:134)
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReaderWithState.write(JsonReaderWithState.java:65)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.store.easy.json.JSONRecordReader2.next(JSONRecordReader2.java:111)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> {noformat}
> {noformat}select t.nutrients[0].units from dfs.usda.`usda.json` t limit
> 1;{noformat}
> and it failed with following exception
> {noformat}
> 2014-09-27 17:50:04,394 [9ee8a529-17fd-492f-9cba-2d1f5842eae1:frag:0:0] ERROR
> o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error
> c4c6bffd-b62b-4878-af1e-58db64453307: Screen received stop request sent.
> java.lang.IllegalArgumentException: You tried to write a BigInt type when you
> are using a ValueWriter of type NullableFloat8WriterImpl.
> at
> org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:513)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:145)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.impl.NullableFloat8WriterImpl.write(NullableFloat8WriterImpl.java:88)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:257)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:310)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:204)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:134)
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReaderWithState.write(JsonReaderWithState.java:65)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.store.easy.json.JSONRecordReader2.next(JSONRecordReader2.java:111)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:158)
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
