[
https://issues.apache.org/jira/browse/HADOOP-17628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388082#comment-17388082
]
Steve Loughran commented on HADOOP-17628:
-----------------------------------------
working on this as its taking so long the ROI will be immediate for us
developers
* s3a to to -direct always
* width and depth of dir tree massively reduced
* removed the ContractSecureDistCp because new abfs stores are secure only;
superfluous
Side issue, distcp with iterator is tangibly slower on abfs rather than distcp.
That's why it was always the testDistCpWithIterator test which times out.
This also implies that other code which uses listIterator &c is
underperformant. As it is adopted in more places (spark, iceberg), needs
attention.
> Distcp contract test is really slow with ABFS and S3A; timing out
> -----------------------------------------------------------------
>
> Key: HADOOP-17628
> URL: https://issues.apache.org/jira/browse/HADOOP-17628
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, fs/s3, test, tools/distcp
> Affects Versions: 3.4.0
> Reporter: Bilahari T H
> Priority: Minor
>
> The test case testDistCpWithIterator in AbstractContractDistCpTest is
> consistently timing out.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
