jorisvandenbossche commented on issue #12885: URL: https://github.com/apache/arrow/issues/12885#issuecomment-1102439997
It might be useful to include a small actual example with some dummy data and explain how you would want to read that. Now, in general, I don't think we have much support for mixed types in JSON. In theory, those could also be read as "union" types (although for writing to parquet as a next step that's not so useful, since parquet doesn't support this) > Is it possible to still keep the default inference behaviour while also ignoring or casting values that don't conform? It's not fully clear to me what you exactly meaning with "ignoring" in this case. Do you mean ignoring certain values in a column, or ignoring a full column? And if ignoring certain values, what should they be replaced with? (null?) > Or, casting I mean casting to higher class of the json data type, for example int casts to str in this case. Maybe we could enable this by letting the user specify an explicit schema (as str in this case), although that currently will also error if then a value doesn't match the specified type. From a quick search, someone else reported a similar JIRA about this: https://issues.apache.org/jira/browse/ARROW-11978 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
