shmilygkd opened a new pull request, #6755:
URL: https://github.com/apache/hive/pull/6755
### What changes were proposed in this pull request?
Move the `--add-opens` injection for MR jobs from a private method in
`ExecDriver` into `JavaVersionUtils.addOpensFlags(Configuration)` and call it
from every code path that submits an MR job through `JobClient`:
- `ExecDriver.execute` (unchanged behaviour, now delegates to the helper)
- `MergeFileTask.execute` (`ALTER TABLE ... CONCATENATE` on the MR engine)
- `ColumnTruncateTask.execute` (`TRUNCATE TABLE ... COLUMNS` on RCFile
tables)
- `MRCompactor.launchCompactionJob` (MR-based ACID compaction launched by
the metastore Worker)
The helper appends the flags to `mapreduce.map.java.opts`,
`mapreduce.reduce.java.opts` and `yarn.app.mapreduce.am.command-opts`, keeping
whatever is already configured. `TestJavaVersionUtils` is added to cover the
helper.
### Why are the changes needed?
HIVE-28869 only injects the flags in `ExecDriver`. The three paths above
submit MR jobs directly, so on a JDK 17 cluster whose `mapred-site.xml` does
not carry the flags their task JVMs fail on the first reflective access into
`java.base`, for example:
```
Error: java.lang.reflect.InaccessibleObjectException: Unable to make field
private volatile java.lang.String java.net.URI.string accessible:
module java.base does not "opens java.net" to unnamed module
at org.apache.hadoop.hive.common.StringInternUtils.<clinit>
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init
```
`MRCompactor` is still the default compactor for full-CRUD tables
(`hive.compactor.crud.query.based=false`), so this affects regular ACID
maintenance, not only the MR engine.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- New unit test `TestJavaVersionUtils#testAddOpensFlagsAppendedToMRJobOpts`
(existing `-Xmx` value preserved, all three keys end with the flag set).
- The qtest MiniMR drivers run tasks in the local job runner, where
`*.java.opts` is ignored, so the container command line cannot be exercised in
CI. An equivalent patch on a 3.1.x-based build was verified on a 3-node YARN
cluster (Hadoop 3.4.3, JDK 17 on submitter and containers, no cluster-side
`--add-opens`): before the patch `ALTER TABLE ... CONCATENATE` failed as above;
after it the AM and task `launch_container.sh` of the merge job, the RCFile
column-truncate job and a metastore-launched major compaction all carry the
flags and the jobs succeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
