Any advise on this issue? I'm having the same problem. On Fri, Oct 9, 2015 at 4:13 AM, David M <[email protected]> wrote:
> Hi everyone. > > I have Mesos cluster (0.24.1) for running Spark (1.5.2) that runs great. > > I have a requirement to move my Mesos cluster nodes behind a Squid http > proxy. > All cluster nodes previously had direct outbound Internet access so > accessing SPARK_EXECUTOR_URI from a public source was not a problem. > > System-wide I have http_proxy and https_proxy environment variables set. > Command line tools like curl and wget operate just fine against Internet > resources. > After configuring Maven's proxy settings the Mesos build completed > successfully. > > I copied my /etc/hosts file to HDFS and attempted the WordCount example > from: > http://documentation.altiscale.com/spark-shell-examples-1-1 > > It failed with this in the executors stderr file: > > I1008 15:39:48.417644 21698 logging.cpp:172] INFO level logging started! > I1008 15:39:48.417819 21698 fetcher.cpp:414] Fetcher Info: > {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/20151007-154648-2701359370-5050-25191-S3\/spark","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/ > d3kbcqa49mib13.cloudfront.net > \/spark-1.5.1-bin-hadoop2.6.tgz"}}],"sandbox_directory":"\/var\/run\/mesos\/slaves\/20151007-154648-2701359370-5050-25191-S3\/frameworks\/20151008-123957-2701359370-5050-6382-0001\/executors\/20151007-154648-2701359370-5050-25191-S3\/runs\/507827fb-cfb0-4a1d-977d-9b9afb972c29","user":"spark"} > I1008 15:39:48.418918 21698 fetcher.cpp:369] Fetching URI ' > http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz' > I1008 15:39:48.418936 21698 fetcher.cpp:243] Fetching directly into the > sandbox directory > I1008 15:39:48.418949 21698 fetcher.cpp:180] Fetching URI ' > http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz' > I1008 15:39:48.418958 21698 fetcher.cpp:127] Downloading resource from ' > http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz' to > '/var/run/mesos/slaves/20151007-154648-2701359370-5050-25191-S3/frameworks/20151008-123957-2701359370-5050-6382-0001/executors/20151007-154648-2701359370-5050-25191-S3/runs/507827fb-cfb0-4a1d-977d-9b9afb972c29/spark-1.5.1-bin-hadoop2.6.tgz' > Failed to fetch ' > http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz': > Error downloading resource, received HTTP return code 400 > Failed to synchronize with slave (it's probably exited) > > Long troubleshooting story short it appears that libcurl isn't finding out > about my proxy. > > In ./3rdparty/libprocess/3rdparty/stout/include/stout/posix/net.hpp > > I added > > curl_easy_setopt(curl, CURLOPT_VERBOSE, true); > curl_easy_setopt(curl, CURLOPT_PROXY, "<my squid server hostname here>"); > curl_easy_setopt(curl, CURLOPT_PROXYPORT, <my squid server port here>); > > before > > CURLcode curlErrorCode = curl_easy_perform(curl); > > I then recompiled Mesos and the WordCount example is successful. > > What is the correct way to set proxy so that libcurl will make use of it? > > Thank you. > David > > > > -- *Intel SSG/STO/BDT* 880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai, China +862161166500
