[
https://issues.apache.org/jira/browse/HUDI-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Guo updated HUDI-3028:
----------------------------
Description:
This is the script for Spark download in Azure CI:
{code:java}
echo 'Downloading spark-$(SPARK_VERSION)-bin-hadoop$(HADOOP_VERSION)'
wget
https://archive.apache.org/dist/spark/spark-$(SPARK_VERSION)/spark-$(SPARK_VERSION)-bin-hadoop$(HADOOP_VERSION).tgz
-O $(Pipeline.Workspace)/spark-$(SPARK_VERSION).tgz {code}
{code:java}
224650K .......... .......... .......... .......... ........ 100% 462K=27m17s
2021-12-15 17:40:59 (137 KB/s) - '/home/vsts/work/1/spark-2.4.4.tgz' saved
[230091034/230091034]{code}
It took more than 20 minutes to download Spark in one instance. We may find a
different source without much rate limiting.
> Spark binary download sometimes takes a long time in Azure CI IT tests
> ----------------------------------------------------------------------
>
> Key: HUDI-3028
> URL: https://issues.apache.org/jira/browse/HUDI-3028
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo
> Priority: Major
>
> This is the script for Spark download in Azure CI:
> {code:java}
> echo 'Downloading spark-$(SPARK_VERSION)-bin-hadoop$(HADOOP_VERSION)'
> wget
> https://archive.apache.org/dist/spark/spark-$(SPARK_VERSION)/spark-$(SPARK_VERSION)-bin-hadoop$(HADOOP_VERSION).tgz
> -O $(Pipeline.Workspace)/spark-$(SPARK_VERSION).tgz {code}
>
> {code:java}
> 224650K .......... .......... .......... .......... ........ 100%
> 462K=27m17s
> 2021-12-15 17:40:59 (137 KB/s) - '/home/vsts/work/1/spark-2.4.4.tgz' saved
> [230091034/230091034]{code}
> It took more than 20 minutes to download Spark in one instance. We may find
> a different source without much rate limiting.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
