[
https://issues.apache.org/jira/browse/SPARK-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen updated SPARK-9342:
-----------------------------
Assignee: Xiao Li
> Spark SQL views don't work
> --------------------------
>
> Key: SPARK-9342
> URL: https://issues.apache.org/jira/browse/SPARK-9342
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.3.1
> Environment: Ubuntu on AWS
> Reporter: Simeon Simeonov
> Assignee: Xiao Li
> Labels: sql, views
> Fix For: 2.0.0
>
>
> The Spark SQL documentation's section on Hive support claims that views are
> supported. However, even basic view operations fail with exceptions related
> to column resolution.
> For example,
> {code}
> // The test table has columns category & num
> ctx.sql("create view view1 as select * from test")
> ctx.table("view1").printSchema
> {code}
> generates
> {code}
> org.apache.spark.sql.AnalysisException: cannot resolve 'test.col' given input
> columns category, num; line 1 pos 7
> at
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> ...
> {code}
> You can see a standalone reproducible example with full spark-shell output
> demonstrating the problem at
> [https://gist.github.com/ssimeonov/57164f9d6b928ba0cfde]
> The problem is that {{ctx.sql("create view view1 as select * from test")}}
> puts the following in the metastore including {{cols:[FieldSchema(name:col,
> type:string, comment:null)]}} even though the {{test}} table has {{category}}
> and {{num}} columns:
> {code}
> 15/07/26 15:47:28 INFO HiveMetaStore: 0: create_table: Table(tableName:view1,
> dbName:default, owner:ubuntu, createTime:1437925648, lastAccessTime:0,
> retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:string,
> comment:null)], location:null,
> inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:null, parameters:{}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{},
> viewOriginalText:select * from test, viewExpandedText:select `test`.`col`
> from `default`.`test`, tableType:VIRTUAL_VIEW)
> 15/07/26 15:47:28 INFO audit: ugi=ubuntu ip=unknown-ip-addr
> cmd=create_table: Table(tableName:view1, dbName:default, owner:ubuntu,
> createTime:1437925648, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:col, type:string, comment:null)],
> location:null, inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:null, parameters:{}), bucketCols:[], sortCols:[],
> parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
> skewedColValueLocationMaps:{})), partitionKeys:[], parameters:{},
> viewOriginalText:select * from test, viewExpandedText:select `test`.`col`
> from `default`.`test`, tableType:VIRTUAL_VIEW)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
