[jira] [Commented] (ORC-54) Evolve schemas based on field name rather than index
[ https://issues.apache.org/jira/browse/ORC-54?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15334675#comment-15334675 ] Mark Wagner commented on ORC-54: I'm reading through your patch on HIVE-13974. Wasn't aware of that work. Still need to read through it some more to grok, but I agree that there's overlap in that both of the patches are making the distinction around the included array explicit. I don't see a reason Hive shouldn't be able to read these datasets as long as the OrcSerde declares the reader schema in the Options. We use Hive for reading Orc, so that's definitely something I'm paying attention to. > Evolve schemas based on field name rather than index > > > Key: ORC-54 > URL: https://issues.apache.org/jira/browse/ORC-54 > Project: Orc > Issue Type: Improvement >Reporter: Mark Wagner >Assignee: Mark Wagner > > Schema evolution as it stands today allows adding fields to the end of > schemas or removing them from the end. However, because it is based on the > index of the column, you can only ever add or remove -- not both. > ORC files have the full schema information of their contents, so there's > actually enough metadata to support changing columns anywhere in the schema. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ORC-54) Evolve schemas based on field name rather than index
[ https://issues.apache.org/jira/browse/ORC-54?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333251#comment-15333251 ] Mark Wagner commented on ORC-54: [~mmccline], I've just posted a PR. Let me know your thoughts. Thanks! > Evolve schemas based on field name rather than index > > > Key: ORC-54 > URL: https://issues.apache.org/jira/browse/ORC-54 > Project: Orc > Issue Type: Improvement >Reporter: Mark Wagner >Assignee: Mark Wagner > > Schema evolution as it stands today allows adding fields to the end of > schemas or removing them from the end. However, because it is based on the > index of the column, you can only ever add or remove -- not both. > ORC files have the full schema information of their contents, so there's > actually enough metadata to support changing columns anywhere in the schema. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (ORC-53) OrcMap does not support complex types as keys
[ https://issues.apache.org/jira/browse/ORC-53?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294412#comment-15294412 ] Mark Wagner commented on ORC-53: [~owen.omalley], was there a requirement for OrcMap to maintain ordering? This is fixed by extending HashMap. All tests pass then, but it is an incompatible change. > OrcMap does not support complex types as keys > - > > Key: ORC-53 > URL: https://issues.apache.org/jira/browse/ORC-53 > Project: Orc > Issue Type: Bug >Reporter: Mark Wagner > > The current implementation of OrcMap extends TreeMap, which requires that the > keys implement Comparable. However, none of OrcStruct, OrcList, OrcUnion, or > OrcMap do so. Other write paths such as through the OrcSerde seem to work > okay. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
