[
https://issues.apache.org/jira/browse/DRILL-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311836#comment-14311836
]
Adam Gilmore commented on DRILL-1460:
-------------------------------------
I agree this is a difficult one; however, I think the solution is really quite
simple. Rather than forcing a user to cast, which can be painful (e.g. in my
use case, I want to convert JSON to Parquet without knowing the JSON schema at
query time), we have an "allNumbersAsDouble" option like the "allTextMode"
option.
This gives us the best of both worlds. If users want to cast from text and have
complete control, they can. If users want to treat all numbers as doubles,
they can. If they want the default functionality (that potentially errors),
they can.
I'm happy to investigate implementing this functionality if we agree this would
be useful. [~jaltekruse] can you comment on this?
> JsonReader fails reading files with decimal numbers and integers in the same
> field
> ----------------------------------------------------------------------------------
>
> Key: DRILL-1460
> URL: https://issues.apache.org/jira/browse/DRILL-1460
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JSON
> Affects Versions: 0.6.0
> Reporter: Bhallamudi Venkata Siva Kamesh
> Assignee: Jason Altekruse
> Fix For: 0.8.0
>
>
> Used the following dataset :
> http://thecodebarbarian.wordpress.com/2014/03/28/plugging-usda-nutrition-data-into-mongodb
> Executed the following query
> {noformat}select t.nutrients from dfs.usda.`usda.json` t limit 1;{noformat}
> and it failed with following exception
> {noformat}
> 2014-09-27 17:48:39,421 [b9dfbb9b-29a9-425d-801c-2e418533525f:frag:0:0] ERROR
> o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error
> 0568d90a-d7df-4a5d-87e9-8b9f718dffa4: Screen received stop request sent.
> java.lang.IllegalArgumentException: You tried to write a BigInt type when you
> are using a ValueWriter of type NullableFloat8WriterImpl.
> at
> org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:513)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:145)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.impl.NullableFloat8WriterImpl.write(NullableFloat8WriterImpl.java:88)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:257)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:310)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:204)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:134)
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReaderWithState.write(JsonReaderWithState.java:65)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.store.easy.json.JSONRecordReader2.next(JSONRecordReader2.java:111)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> {noformat}
> {noformat}select t.nutrients[0].units from dfs.usda.`usda.json` t limit
> 1;{noformat}
> and it failed with following exception
> {noformat}
> 2014-09-27 17:50:04,394 [9ee8a529-17fd-492f-9cba-2d1f5842eae1:frag:0:0] ERROR
> o.a.d.e.p.i.ScreenCreator$ScreenRoot - Error
> c4c6bffd-b62b-4878-af1e-58db64453307: Screen received stop request sent.
> java.lang.IllegalArgumentException: You tried to write a BigInt type when you
> are using a ValueWriter of type NullableFloat8WriterImpl.
> at
> org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.fail(AbstractFieldWriter.java:513)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.impl.AbstractFieldWriter.write(AbstractFieldWriter.java:145)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.impl.NullableFloat8WriterImpl.write(NullableFloat8WriterImpl.java:88)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:257)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:310)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.writeData(JsonReader.java:204)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReader.write(JsonReader.java:134)
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.vector.complex.fn.JsonReaderWithState.write(JsonReaderWithState.java:65)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.store.easy.json.JSONRecordReader2.next(JSONRecordReader2.java:111)
>
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> at
> org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:158)
> ~[drill-java-exec-0.6.0-incubating-SNAPSHOT-rebuffed.jar:0.6.0-incubating-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
