[
https://issues.apache.org/jira/browse/PHOENIX-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani reassigned PHOENIX-7953:
-------------------------------------
Assignee: Ujjawal Kumar
> Add addPhoenixDependencyJars utility method to ship phoenix-core-client
> runtime deps to YARN distributed cache
> --------------------------------------------------------------------------------------------------------------
>
> Key: PHOENIX-7953
> URL: https://issues.apache.org/jira/browse/PHOENIX-7953
> Project: Phoenix
> Issue Type: Improvement
> Reporter: Ujjawal Kumar
> Assignee: Ujjawal Kumar
> Priority: Minor
>
>
> Phoenix-based MapReduce jobs (IndexTool, UpdateStatisticsTool,
> TransformTool, and external consumers like HGrate) must manually enumerate
> classes from phoenix-core-client's dependency tree in
> TableMapReduceUtil.addDependencyJarsForClasses() to ship jars to YARN
> containers via the distributed cache (tmpjars). There is no centralized
> Phoenix equivalent of HBase's TableMapReduceUtil.addHBaseDependencyJars(conf).
>
> {*}Problem:{*}{*}{*}
>
> Each consumer maintains its own stale, incomplete list. When Phoenix
> introduces new runtime dependencies (e.g., json-path, bson for JSON type
> support, or phoenix-hbase-compat refactoring in 5.3), every consumer hits
> ClassNotFoundException / NoClassDefFoundError in production and must
> independently discover and add the missing jars. Phoenix's own
> UpdateStatisticsTool currently only lists 5 classes and is already missing
> deps needed since
> the JSON type was added.
>
> Example failures after Phoenix 5.3 upgrade:
> - NoClassDefFoundError: com/jayway/jsonpath/spi/json/JsonProvider (PJson →
> JsonDataFormatFactory → json-path)
> - ClassNotFoundException: org.bson.io.BsonInput (JsonDataFormatFactory →
> BsonDataFormat → bson)
> - ClassNotFoundException:
> org.apache.phoenix.compat.hbase.CompatPagingFilter (ScanRanges.<clinit> →
> phoenix-hbase-compat-2.5.4)
>
> All three jars are in the tarball/classpath on the submitting host, but
> never added to tmpjars so they're absent on YARN containers.
>
> {*}Proposed fix:{*}{*}{*}
>
> Add a static method
> PhoenixMapReduceUtil.addPhoenixDependencyJars(Configuration conf) in
> phoenix-core-server (where the existing PhoenixMapReduceUtil class lives)
> that registers one representative class from each runtime
> dependency of phoenix-core-client. This mirrors HBase's
> addHBaseDependencyJars pattern.
>
> Jars to include (direct runtime deps of phoenix-core-client that aren't
> covered by TableMapReduceUtil.addDependencyJars):
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
