Ujjawal Kumar created PHOENIX-7953:
--------------------------------------
Summary: Add addPhoenixDependencyJars utility method to ship
phoenix-core-client runtime deps to YARN distributed cache
Key: PHOENIX-7953
URL: https://issues.apache.org/jira/browse/PHOENIX-7953
Project: Phoenix
Issue Type: Improvement
Reporter: Ujjawal Kumar
Phoenix-based MapReduce jobs (IndexTool, UpdateStatisticsTool, TransformTool,
and external consumers like HGrate) must manually enumerate classes from
phoenix-core-client's dependency tree in
TableMapReduceUtil.addDependencyJarsForClasses() to ship jars to YARN
containers via the distributed cache (tmpjars). There is no centralized Phoenix
equivalent of HBase's TableMapReduceUtil.addHBaseDependencyJars(conf).
{*}Problem:{*}{*}{*}
Each consumer maintains its own stale, incomplete list. When Phoenix
introduces new runtime dependencies (e.g., json-path, bson for JSON type
support, or phoenix-hbase-compat refactoring in 5.3), every consumer hits
ClassNotFoundException / NoClassDefFoundError in production and must
independently discover and add the missing jars. Phoenix's own
UpdateStatisticsTool currently only lists 5 classes and is already missing deps
needed since
the JSON type was added.
Example failures after Phoenix 5.3 upgrade:
- NoClassDefFoundError: com/jayway/jsonpath/spi/json/JsonProvider (PJson →
JsonDataFormatFactory → json-path)
- ClassNotFoundException: org.bson.io.BsonInput (JsonDataFormatFactory →
BsonDataFormat → bson)
- ClassNotFoundException: org.apache.phoenix.compat.hbase.CompatPagingFilter
(ScanRanges.<clinit> → phoenix-hbase-compat-2.5.4)
All three jars are in the tarball/classpath on the submitting host, but never
added to tmpjars so they're absent on YARN containers.
{*}Proposed fix:{*}{*}{*}
Add a static method
PhoenixMapReduceUtil.addPhoenixDependencyJars(Configuration conf) in
phoenix-core-server (where the existing PhoenixMapReduceUtil class lives) that
registers one representative class from each runtime
dependency of phoenix-core-client. This mirrors HBase's
addHBaseDependencyJars pattern.
Jars to include (direct runtime deps of phoenix-core-client that aren't
covered by TableMapReduceUtil.addDependencyJars):
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
