[
https://issues.apache.org/jira/browse/SPARK-44845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756652#comment-17756652
]
Snoot.io commented on SPARK-44845:
----------------------------------
User 'zekai-li' has created a pull request for this issue:
https://github.com/apache/spark/pull/42529
> spark job copies jars repeatedly if fs.defaultFS and application jar are same
> url
> ---------------------------------------------------------------------------------
>
> Key: SPARK-44845
> URL: https://issues.apache.org/jira/browse/SPARK-44845
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 3.4.1
> Reporter: zheju_he
> Priority: Major
>
> In the org.apache.spark.deploy.yarn.Client#compareUri method,
> hdfs://hadoop81:8020 and hdfs://192.168.0.81:8020 are regarded as different
> file systems (hadoop81 corresponds to 192.168.0.81). The specific reason is
> that in the last pr, different URIs of user information are also regarded as
> different file systems. Uri.getauthority is used to determine the user
> information, but authority contains the host so the URI above must be
> different from authority. To determine whether the user authentication
> information is different, you only need to determine URI.getUserInfo.
>
> the last pr and issue link:
> https://issues.apache.org/jira/browse/SPARK-22587
> https://github.com/apache/spark/pull/19885
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
