[
https://issues.apache.org/jira/browse/HIVE-27737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-27737:
--------------------------------
Description:
[HIVE-17574|https://github.com/apache/hive/commit/26753ade2a130339940119c950b9c9af53e3d024]
was about an optimization, where HDFS-based resources optionally were
localized directly from the "original" hdfs folder instead of a tez session
dir. This reduced the HDFS overhead, by introducing
hive.resource.use.hdfs.location, so there are 2 cases:
1. hive.resource.use.hdfs.location=true
a) collect "HDFS temp files" and optimize their access: added files, added jars
b) collect local temp files and use the non-optimized session-based approach:
added files, added jars, aux jars, reloadable aux jars
{code}
// reference HDFS based resource directly, to use distribute cache
efficiently.
addHdfsResource(conf, tmpResources, LocalResourceType.FILE,
getHdfsTempFilesFromConf(conf));
// local resources are session based.
tmpResources.addAll(
addTempResources(conf, hdfsDirPathStr, LocalResourceType.FILE,
getLocalTempFilesFromConf(conf), null).values()
);
{code}
2. hive.resource.use.hdfs.location=false
a) original behavior: collect all jars in hs2's scope (added files, added jars,
aux jars, reloadable aux jars) and put it to a session based directory
{code}
// all resources including HDFS are session based.
tmpResources.addAll(
addTempResources(conf, hdfsDirPathStr, LocalResourceType.FILE,
getTempFilesFromConf(conf), null).values()
);
{code}
my proposal is related to 1)
let's say user is about to load an aux jar from hdfs and have it set in
hive.aux.jars.path:
{code}
hive.aux.jars.path=file:///opt/some_local_jar.jar,hdfs:///tmp/some_distributed.jar
{code}
in this case: we can distinguish between file:// scheme resources and hdfs://
scheme resources:
- file scheme resources should fall into 1b), still be used from session dir
- hdfs scheme resources should fall into 1a), simply used by addHdfsResource
> Consider extending HIVE-17574 to aux jars
> -----------------------------------------
>
> Key: HIVE-27737
> URL: https://issues.apache.org/jira/browse/HIVE-27737
> Project: Hive
> Issue Type: Improvement
> Reporter: László Bodor
> Priority: Major
>
> [HIVE-17574|https://github.com/apache/hive/commit/26753ade2a130339940119c950b9c9af53e3d024]
> was about an optimization, where HDFS-based resources optionally were
> localized directly from the "original" hdfs folder instead of a tez session
> dir. This reduced the HDFS overhead, by introducing
> hive.resource.use.hdfs.location, so there are 2 cases:
> 1. hive.resource.use.hdfs.location=true
> a) collect "HDFS temp files" and optimize their access: added files, added
> jars
> b) collect local temp files and use the non-optimized session-based approach:
> added files, added jars, aux jars, reloadable aux jars
> {code}
> // reference HDFS based resource directly, to use distribute cache
> efficiently.
> addHdfsResource(conf, tmpResources, LocalResourceType.FILE,
> getHdfsTempFilesFromConf(conf));
> // local resources are session based.
> tmpResources.addAll(
> addTempResources(conf, hdfsDirPathStr, LocalResourceType.FILE,
> getLocalTempFilesFromConf(conf), null).values()
> );
> {code}
> 2. hive.resource.use.hdfs.location=false
> a) original behavior: collect all jars in hs2's scope (added files, added
> jars, aux jars, reloadable aux jars) and put it to a session based directory
> {code}
> // all resources including HDFS are session based.
> tmpResources.addAll(
> addTempResources(conf, hdfsDirPathStr, LocalResourceType.FILE,
> getTempFilesFromConf(conf), null).values()
> );
> {code}
> my proposal is related to 1)
> let's say user is about to load an aux jar from hdfs and have it set in
> hive.aux.jars.path:
> {code}
> hive.aux.jars.path=file:///opt/some_local_jar.jar,hdfs:///tmp/some_distributed.jar
> {code}
> in this case: we can distinguish between file:// scheme resources and hdfs://
> scheme resources:
> - file scheme resources should fall into 1b), still be used from session dir
> - hdfs scheme resources should fall into 1a), simply used by addHdfsResource
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
