Andrew Otto created SPARK-23890:

             Summary: Hive ALTER TABLE CHANGE COLUMN for struct type no longer 
                 Key: SPARK-23890
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0
            Reporter: Andrew Otto

As part of SPARK-14118, Spark SQL removed support for sending ALTER TABLE 
CHANGE COLUMN commands to Hive.  This was expanded in 
[] to allow for those commands if 
they only change the column comment.

Wikimedia has been evolving Parquet backed Hive tables with data originally 
come from JSON data by adding newly found columns to the Hive table schema, via 
a Spark job we call 'Refine'.  We do this by recursively merging an input 
DataFrame schema with a Hive table DataFrame schema, finding new fields, and 
then issuing an ALTER TABLE statement to add the columns.  However, because we 
allow for nested data types in the incoming JSON data, we make extensive use of 
struct type fields.  In order to add newly detected fields in a nested data 
type, we must alter the struct column and append the nested struct field.  This 
requires CHANGE COLUMN that alters the column type.  In reality, the 'type' of 
the column is not changing, it just just a new field being added to the struct, 
but to SQL, this looks like a type change.

We were about to upgrade to Spark 2 but this new restriction in SQL DDL that 
can be sent to Hive will block us.  I believe this is fixable by adding an 
exception in 
 to allow ALTER TABLE CHANGE COLUMN with a new type, if the original type and 
destination type are both struct types, and the destination type only adds new 



This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to