zheju_he created SPARK-44845:
--------------------------------
Summary: spark job copies jars repeatedly if fs.defaultFS and
application jar are same url
Key: SPARK-44845
URL: https://issues.apache.org/jira/browse/SPARK-44845
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 3.4.1
Reporter: zheju_he
In the org.apache.spark.deploy.yarn.Client#compareUri method,
hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different
file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is
that in the last pr, different URIs of user information are also regarded as
different file systems. Uri.getauthority is used to determine the user
information, but authority contains the host so the URI above must be different
from authority. To determine whether the user authentication information is
different, you only need to determine URI.getUserInfo.
the last pr and issue link:
https://issues.apache.org/jira/browse/SPARK-22587
https://github.com/apache/spark/pull/19885
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
