shane knapp created SPARK-3745:
----------------------------------
Summary: curl on maven search repo apache rat url returns search
status, not jar file
Key: SPARK-3745
URL: https://issues.apache.org/jira/browse/SPARK-3745
Project: Spark
Issue Type: Bug
Components: Build
Environment: centos 6.5
Reporter: shane knapp
in spark/dev/check-license, there are four attempts to download the apache rat
jar from maven:
URL1="http://search.maven.org/remotecontent?filepath=org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar";
URL2="http://repo1.maven.org/maven2/org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar";
*snip*
if hash curl 2>/dev/null; then
(curl --progress-bar ${URL1} > "$JAR_DL" || curl --progress-bar ${URL2} >
"$JAR_DL") && mv "$JAR_DL" "$JAR"
elif hash wget 2>/dev/null; then
(wget --progress=bar ${URL1} -O "$JAR_DL" || wget --progress=bar ${URL2}
-O "$JAR_DL") && mv "$JAR_DL" "$JAR"
the first attempt is on the search repo via curl, which returns a "YEP! WE
FOUND IT!" html blob:
[root@test01 sknapp]# curl --progress-bar
http://search.maven.org/remotecontent?filepath=org/apache/rat/apache-rat/0.10/apache-rat-0.10.jar
> wtf.part
######################################################################## 100.0%
[root@test01 sknapp]# cat wtf.part
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx/0.8.55</center>
</body>
</html>
this is failing to DL for EVERY time the test is run. i've run curl on the 2nd
url, which points at the repo itself and it successfully downloads. wget does
the correct thing for both URLs.
there is also no error checking on the downloaded file, short of file existence.
potential fixes, in no particular order:
1) run unzip -tq ${$JAR}, check for 0 exist status to ensure it's a compressed
archive
2) run wget before curl
3) only run curl on the 2nd URL (pointing directly to the repo)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
