[jira] [Commented] (ORC-120) Create a backwards compatibility mode of ignoring names for evolution

2016-12-18 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/ORC-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759654#comment-15759654
 ] 

Lefty Leverenz commented on ORC-120:


The new configuration parameter in OrcConf.java is 
*orc.force.positional.evolution*.

> Create a backwards compatibility mode of ignoring names for evolution
> -
>
> Key: ORC-120
> URL: https://issues.apache.org/jira/browse/ORC-120
> Project: Orc
>  Issue Type: Task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 1.3.0
>
>
> ORC's schema evolution uses the column names when they are available. Hive 
> 2.1 uses a positional schema, so ORC should support a backward compatibility 
> mode for Hive users during the transition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (ORC-120) Create a backwards compatibility mode of ignoring names for evolution

2016-12-18 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/ORC-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759643#comment-15759643
 ] 

Lefty Leverenz commented on ORC-120:


This will need documentation, as discussed in some Dec. 12 messages on the 
dev@orc email thread, archived here:

* 
https://mail-archives.apache.org/mod_mbox/orc-dev/201612.mbox/%3ced4820a4-8ee1-462a-8658-551d54323...@iq80.com%3e

It's strange that the messages don't appear here in the JIRA.

Synopsis:

Dain Sundstrom -- Is "ORC's schema evolution uses the column names when they 
are availableā€ documented somewhere?

Owen O'Malley -- No, unfortunately, but it needs to be. The basic rules from 
SchemaEvolution.java look like:

structs (including the top row):
   if field names are available (post HIVE-4243), use name matching
   otherwise use positional matching

lists, maps, unions:
  children must match

Many primitives can convert to each other, but this list needs to be cleaned up:
  boolean, byte, short, int, long, float, double, decimal -> boolean, byte,
short, int, long, float, double, decimal, string, char, varchar, timestamp
  string, char, varchar -> all
  timestamp -> boolean, byte, short, int, long, float, double, decimal,
string, char, varchar, date
  date -> string, char, varchar, timestamp
  binary -> string, char, varchar, date

Dain Sundstrom -- So, rename column is not expected to work anymore?

Owen O'Malley -- ORC-120 will add an option to force positional mapping.

Dain Sundstrom -- Oh, I see this is an ORC feature like the Parquet schema 
evolution stuff.  We implemented support for ordering by the top level struct 
names in Presto a while back.

> Create a backwards compatibility mode of ignoring names for evolution
> -
>
> Key: ORC-120
> URL: https://issues.apache.org/jira/browse/ORC-120
> Project: Orc
>  Issue Type: Task
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Fix For: 1.3.0
>
>
> ORC's schema evolution uses the column names when they are available. Hive 
> 2.1 uses a positional schema, so ORC should support a backward compatibility 
> mode for Hive users during the transition.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)