[
https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-19494:
-----------------------------------------
Description:
Hive Streaming API jars are sometimes shaded with a different prefix when used
in environments where another version of hive already exists (spark for
example). In most cases, shading is done with rename of classes with some
prefix. If an uber/assembly jar is generated with renamed prefix, Hive
Streaming API will not work as Hive Streaming API will reflectively instantiate
outputformat class using FQCN string provided by metastore table storage
descriptor object.
For example:
RecordWriter will create instance of OutputFormat using string
"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with
renamed class references are used, this class will not be found by the
classloader.
We can optionally accept a shade prefix from user via config which will be
tried (as fallback) when ClassNotFoundException is thrown.
was:
Hive Streaming API jars are sometimes shaded with a different prefix when used
in environments where another version of hive already exists (spark for
example). In most cases, shading is done with rename of classes with some
prefix. If an uber/assembly jar is generated with renamed prefix, Hive
Streaming API will not work as Hive Streaming API will reflectively instantiate
outputformat class using FQCN string provided by metastore table storage
descriptor object.
For example:
RecordWriter will create instance of OutputFormat using string
"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with
renamed class references are used, this class will not be found by the
classloader.
We can optionally accept a shade prefix from user which will be tried (as
fallback) when ClassNotFoundException is thrown.
> Accept shade prefix during reflective instantiation of output format
> --------------------------------------------------------------------
>
> Key: HIVE-19494
> URL: https://issues.apache.org/jira/browse/HIVE-19494
> Project: Hive
> Issue Type: Sub-task
> Components: Streaming
> Affects Versions: 3.0.0, 3.1.0
> Reporter: Prasanth Jayachandran
> Assignee: Prasanth Jayachandran
> Priority: Major
>
> Hive Streaming API jars are sometimes shaded with a different prefix when
> used in environments where another version of hive already exists (spark for
> example). In most cases, shading is done with rename of classes with some
> prefix. If an uber/assembly jar is generated with renamed prefix, Hive
> Streaming API will not work as Hive Streaming API will reflectively
> instantiate outputformat class using FQCN string provided by metastore table
> storage descriptor object.
> For example:
> RecordWriter will create instance of OutputFormat using string
> "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat". When a shaded jar with
> renamed class references are used, this class will not be found by the
> classloader.
> We can optionally accept a shade prefix from user via config which will be
> tried (as fallback) when ClassNotFoundException is thrown.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
