[
https://issues.apache.org/jira/browse/DRILL-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15188349#comment-15188349
]
Aman Sinha commented on DRILL-4479:
-----------------------------------
My patch above will handle the case where we are doing the group-by on 1 column
which happens to have all null values in the first batch (with all_text_mode
enabled). However, if there are 2 or more group-by columns, the JSON reader
succeeds but the HashAggregate fails with a schema change exception:
{noformat}
alter session set `store.json.all_text_mode` = true;
select a, c, count(*) as cnt from mostlynulls.json group by a, c;
Exception (no rows returned):
org.apache.drill.common.exceptions.UserRemoteException: UNSUPPORTED_OPERATION
ERROR: Hash aggregate does not support schema changes
{noformat}
The reason for this is JsonReader.ensureAtLeastOneField() method creates a
single output vector regardless of whether there were multiple columns with all
nulls in the first batch. So, initially HashAggregate thinks there is 1 column
to group-by and subsequently gets 2 columns when the JsonReader sends the next
batch. This causes a SchemaChangeException.
In order to address this, I have changed ensureAtLeastOneField() to produce
vectors corresponding to the number of columns; the purpose of this method is
to ensure 'at least 1 field' but should not prevent creating more fields if
necessary. I will update the PR after running more tests.
> JsonReader should pick a less restrictive type when creating the default
> column
> -------------------------------------------------------------------------------
>
> Key: DRILL-4479
> URL: https://issues.apache.org/jira/browse/DRILL-4479
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - JSON
> Affects Versions: 1.5.0
> Reporter: Aman Sinha
> Attachments: mostlynulls.json
>
>
> This JIRA is related to DRILL-3806 but has a narrower scope, so I decided to
> create separate one.
> The JsonReader has the method ensureAtLeastOneField() (see
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/vector/complex/fn/JsonReader.java#L91)
> that ensures that when no columns are found, create an empty one and it
> chooses to create a nullable int column. One consequence is that queries of
> the following type fail:
> {noformat}
> select c1 from dfs.`mostlynulls.json`;
> ...
> ...
> | null |
> | null |
> Error: DATA_READ ERROR: Error parsing JSON - You tried to write a VarChar
> type when you are using a ValueWriter of type NullableIntWriterImpl.
> File /Users/asinha/data/mostlynulls.json
> Record 4097
> {noformat}
> In this file the first 4096 rows have NULL values for c1 followed by rows
> that have a valid string.
> It would be useful for the Json reader to choose a less restrictive type such
> as varchar in order to allow more types of queries to run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
