[
https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380119#comment-15380119
]
ASF GitHub Bot commented on DRILL-4653:
---------------------------------------
Github user jaltekruse commented on the issue:
https://github.com/apache/drill/pull/518
I don't think we should merge this without a mechanism to return a warning
to the user to tell them at least that some data was ignored, and ideally some
indication of how much data was discarded. While I do understand this is not
the default behavior, I think there is still too high of a risk that an admin
could set this at a global level and users would be unaware of some of their
data being discarded.
I am willing to discuss the benefits of merging this before such a system
exists, but until this issue has been thoroughly evaluated I am -1 on the
change.
One improvement you could make to the current implementation is moving the
option to the format plugin instead of the system/session list. This enables
users to include setting the option in there query with the "table with
options" syntax that was added last fall. We already have a JIRA open for
moving the all_text_mode and read_numbers_as_double options to this location,
because it doesn't really make sense to change query results based on session
state. Unfortunately this change does not completely remove my initial concern,
because not all users can modify or see the storage plugins in the case when
web UI security is enabled. Non-admin users in these cases could be surprised
by this behavior.
For examples of how this is done, you can look at the text plugin config,
you would just need to add these options as properties to the json config which
is currently mostly empty.
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/json/JSONFormatPlugin.java#L93
https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/easy/text/TextFormatPlugin.java#L135
Select with options: https://issues.apache.org/jira/browse/DRILL-4047
Jira for moving the existing options:
https://issues.apache.org/jira/browse/DRILL-4206
> Malformed JSON should not stop the entire query from progressing
> ----------------------------------------------------------------
>
> Key: DRILL-4653
> URL: https://issues.apache.org/jira/browse/DRILL-4653
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - JSON
> Affects Versions: 1.6.0
> Reporter: subbu srinivasan
> Fix For: Future
>
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something
> similar to a setting of (ignore.malformed.json) would help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
