Copilot commented on code in PR #8531: URL: https://github.com/apache/ozone/pull/8531#discussion_r2116575409
########## hadoop-hdds/docs/content/integration/DistCp.md: ########## @@ -87,3 +87,72 @@ By specifying the appropriate checksum configuration or skipping the validation, When data resides in an HDFS encryption zone or Ozone encrypted buckets, the file checksum will not match. This is because the underlying block data differs due to the use of a new EDEK (Encryption Data Encryption Key) at the destination. In such cases, specify the `-skipcrccheck` parameter to avoid job failures. For more information about using Hadoop DistCp, consult the [DistCp Guide](https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html). + +## Troubleshooting Common Issues + +### Delegation Token Issues +If a DistCp command fails and the error output contains "OzoneToken", indicating an issue with retrieving a delegation token from the destination (or source) Ozone cluster, ensure that Ozone's security is explicitly enabled in the client's Hadoop configuration. + +Add the following property to `core-site.xml` or `ozone-site.xml` on the node where you run the DistCp command: + + <property> Review Comment: Consider using fenced code blocks (e.g., triple backticks with 'xml') for XML configuration snippets to improve readability and consistency in the documentation. ########## hadoop-hdds/docs/content/integration/DistCp.md: ########## @@ -87,3 +87,72 @@ By specifying the appropriate checksum configuration or skipping the validation, When data resides in an HDFS encryption zone or Ozone encrypted buckets, the file checksum will not match. This is because the underlying block data differs due to the use of a new EDEK (Encryption Data Encryption Key) at the destination. In such cases, specify the `-skipcrccheck` parameter to avoid job failures. For more information about using Hadoop DistCp, consult the [DistCp Guide](https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html). + +## Troubleshooting Common Issues + +### Delegation Token Issues +If a DistCp command fails and the error output contains "OzoneToken", indicating an issue with retrieving a delegation token from the destination (or source) Ozone cluster, ensure that Ozone's security is explicitly enabled in the client's Hadoop configuration. + +Add the following property to `core-site.xml` or `ozone-site.xml` on the node where you run the DistCp command: + + <property> + <name>ozone.security.enabled</name> + <value>true</value> + </property> + +This helps the client correctly engage in secure communication protocols with Ozone. + +### Cross-Realm Kerberos (Ozone 1.x) + +**Affected Versions:** Ozone 1.x + +When issuing DistCp commands (or other HDFS-compatible commands like `hdfs dfs -ls`) against an Ozone cluster in a different Kerberos realm than the client or source/destination cluster, Ozone 1.x versions may produce an error similar to: + + 24/02/07 18:47:36 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: java.io.IOException: DestHost:destPort ccycloud-1.weichiu-dst.root.comops.site:9862, LocalHost:localPort ccycloud-1.weichiu-src.local/10.140.99.144:0. Failed on local exception: java.io.IOException: Couldn't set up IO streams: java.lang.IllegalArgumentException: Server has invalid Kerberos principal: om/ccycloud-1.weichiu-dst.local@DST.LOCAL, expecting: OM/ccycloud-1.weichiu-dst.local@REALM, while invoking $Proxy10.submitRequest over nodeId=om26,nodeAddress=ccycloud-1.weichiu-dst.local:9862 after 3 failover attempts. Trying to failover immediately. + + +**Cause:** +This typically occurs because the Ozone Manager's Kerberos principal (`ozone.om.kerberos.principal`) is not defined or interpreted in a way that accommodates cross-realm interactions. The client expects a principal from its own realm or a specifically trusted one, and a mismatch occurs. This issue is addressed in Ozone 2.0. + +**Workaround (for Ozone 1.x):** +To resolve this in Ozone 1.x, add the following property to the `ozone-site.xml` on the client machine running DistCp (and potentially on the Ozone Managers if they also need to inter-communicate across realms): + + <property> + <name>ozone.om.kerberos.principal.pattern</name> + <value>*</value> + </property> + +This configuration relaxes the principal matching, allowing the client to accept Ozone Manager principals from different realms. + +**Fix:** +This bug is tracked by [HDDS-10328](https://issues.apache.org/jira/browse/HDDS-10328) and is fixed in Ozone 2.0 and later versions. Upgrading to Ozone 2.0+ is the recommended long-term solution. + +### Token Renewal Failures in Bidirectional Cross-Realm Trust Environments + +In environments with bidirectional cross-realm Kerberos trust, DistCp jobs (which run as MapReduce jobs) may fail during execution due to errors renewing delegation tokens. An example error is: + + 24/02/08 00:35:00 ERROR tools.DistCp: Exception encountered + java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1707350431298_0001 to YARN: Failed to renew token: Kind: HDFS_DELEGATION_TOKEN, Service: 10.140.99.144:8020, Ident: (token for systest: HDFS_DELEGATION_TOKEN owner=syst...@src.comops.site, renewer=yarn, realUser=, issueDate=1707352474394, maxDate=1707957274394, sequenceNumber=44, masterKeyId=14) + +This can happen when the MapReduce job attempts to renew a delegation token for a remote HDFS or Ozone filesystem, and the renewal fails due to cross-realm complexities. The `Service` field in the error (e.g., `10.140.99.144:8020`) usually indicates the filesystem whose token renewal failed. + +**Solution:** +You can prevent the DistCp MapReduce job from attempting to renew delegation tokens for specific HDFS-compatible filesystems by using the `-Dmapreduce.job.hdfs-servers.token-renewal.exclude` parameter. The value should be the authority (hostname or service ID) of the cluster whose tokens are causing renewal issues. + +**Parameter:** + + -Dmapreduce.job.hdfs-servers.token-renewal.exclude=<authority_of_filesystem_to_exclude> + +For an HDFS cluster, `<authority_of_filesystem_to_exclude>` would be its NameNode address (e.g., `namenode.example.com:8020` or just `namenode.example.com` if the port is standard). For an Ozone cluster, it would be its service ID (e.g., `ozoneprod`). + +**Example:** +If you are running the DistCp command on a YARN cluster associated with the *destination* Ozone cluster (`ofs://ozone1707264383/...`) and copying data *from* a source HDFS cluster (`hdfs://ccycloud-1.weichiu-src.root.comops.site:8020/...`), and the token renewal for the source HDFS cluster is failing: + + hadoop distcp \ Review Comment: Ensure that the example command is formatted in a fenced code block with shell syntax to enhance clarity for users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org For additional commands, e-mail: issues-h...@ozone.apache.org
