GitHub user scwf opened a pull request:
https://github.com/apache/spark/pull/3352
[SQL] set spark.sql.hive.convertMetastoreParquet false by default
Set ```spark.sql.hive.convertMetastoreParquet``` true now leads to
1 run
```create table test_parquet(key int, value string) stored as parquet;```
``` select * from test_parquet;```
get error as follow
java.lang.IllegalArgumentException: Could not find Parquet metadata at path
file:/user/hive/warehouse/test_parquet
at
org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$readMetaData$4.apply(ParquetTypes.scala:459)
at
org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$readMetaData$4.apply(ParquetTypes.scala:459)
at scala.Option.getOrElse(Option.scala:120)
at
org.apache.spark.sql.parquet.ParquetTypesConverter$.readMetaData(ParquetTypes.sc
2 run
```create table test_parquet(key int, value string) stored as parquet;```
```insert into table test_parquet select * from src;```
``` select * from test_parquet;```
get result as follow
...
282 [B@38fda3b8
138 [B@1407a243
238 [B@12de6fb
419 [B@6c976957
15 [B@48850673
118 [B@156a8d37
72 [B@65d20dd
90 [B@4c18906e
307 [B@60b24cc9
19 [B@59cf51b7
435 [B@39fdf371
10 [B@4f799d75
277 [B@39509516
273 [B@596bf4bf
306 [B@3e915576
224 [B@3781d611
309 [B@2d0d128f
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/scwf/spark patch-11
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3352.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3352
----
commit 734e5ef202001c53c5e00795a4afef56222c24e9
Author: wangfei <[email protected]>
Date: 2014-11-19T01:02:31Z
set spark.sql.hive.convertMetastoreParquet false by defauly
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
