Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14883#discussion_r77290783
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala
---
@@ -184,4 +184,17 @@ abstract class ExternalCatalog {
def listFunctions(db: String, pattern: String): Seq[String]
+ //
--------------------------------------------------------------------------
+ // Resources
+ //
--------------------------------------------------------------------------
+
+ /**
+ * Add a JAR resource to the underlying external catalog for DDL (e.g.
CREATE TABLE) and DML
+ * (e.g., LOAD TABLE) operations.
+ *
+ * For example, when users create a Hive serde table, they can specify a
custom
+ * Serializer-Deserializer (SerDe) class. When Hive metastore is unable
to access the custom SerDe
+ * JAR (e.g., not on the Hive classpath), the JAR file must be added at
runtime using this API.
+ */
+ def addJar(path: String): Unit
--- End diff --
Let me copy and paste the reasons why users need the customization
capabilities:
- When to add a new File Format?
> User has files with special file formats not supported by Hive yet, and
users donât want to convert the files before loading into Hive.
> User has a more efficient way of storing data on disk.
- When to add a new File Format?
- When to add a new SerDe?
> User has data with special serialized format not supported by Hive yet,
and users donât want to convert the data before loading into Hive.
> User has a more efficient way of serializing the data on disk.
Also, the existing UDF, UDAF and UDTF can be reused without any rewriting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
