GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/14430
[SPARK-16825] [SQL] Replace hive.default.fileformat by
spark.sql.default.fileformat
### What changes were proposed in this pull request?
Currently, we are using `hive.default.fileformat`, if users do not specify
the format in the CREATE TABLE SQL statement. Multiple issues exist:
- This parameter value is not from `hive-site.xml`. Thus, even if users
change the hive.default.fileformat in `hive-site.xml`, Spark will ignore it. To
change the parameter values, users have to use Spark interface, (e.g., by a SET
command or API).
- This parameter is not documented.
- This parameter value will not be sent to Hive metastore. It is being used
by Spark internals when processing CREATE TABLE statement.
- This parameter is case sensitive.
Since this is being used by Spark only, it does not make sense to use a
parameter starting from `hive`. we might follow the other Hive-related
parameters and introduce a new Spark parameter here. It should be public. Thus,
this PR is to replace `hive.default.fileformat` by
`spark.sql.default.fileformat`. It also makes it case insensitive.
## How was this patch tested?
Added test cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark setHive
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14430.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14430
----
commit 336bd11a25b1eceaa41128d59923d511e65f0642
Author: gatorsmile <[email protected]>
Date: 2016-07-31T15:08:07Z
remove useless set statements.
commit 340e46325f6cca25168aadee81aafa624f622856
Author: gatorsmile <[email protected]>
Date: 2016-07-31T20:46:33Z
change hive.default.fileformat to spark.sql.default.fileformat
commit 0dc7cdaa60ee09847349d577017c8cba5983f456
Author: gatorsmile <[email protected]>
Date: 2016-07-31T20:53:38Z
typo
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
