Hi everyone. I have Mesos cluster (0.24.1) for running Spark (1.5.2) that runs great.
I have a requirement to move my Mesos cluster nodes behind a Squid http proxy. All cluster nodes previously had direct outbound Internet access so accessing SPARK_EXECUTOR_URI from a public source was not a problem. System-wide I have http_proxy and https_proxy environment variables set. Command line tools like curl and wget operate just fine against Internet resources. After configuring Maven's proxy settings the Mesos build completed successfully. I copied my /etc/hosts file to HDFS and attempted the WordCount example from: http://documentation.altiscale.com/spark-shell-examples-1-1 It failed with this in the executors stderr file: I1008 15:39:48.417644 21698 logging.cpp:172] INFO level logging started! I1008 15:39:48.417819 21698 fetcher.cpp:414] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/20151007-154648-2701359370-5050-25191-S3\/spark","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/ d3kbcqa49mib13.cloudfront.net \/spark-1.5.1-bin-hadoop2.6.tgz"}}],"sandbox_directory":"\/var\/run\/mesos\/slaves\/20151007-154648-2701359370-5050-25191-S3\/frameworks\/20151008-123957-2701359370-5050-6382-0001\/executors\/20151007-154648-2701359370-5050-25191-S3\/runs\/507827fb-cfb0-4a1d-977d-9b9afb972c29","user":"spark"} I1008 15:39:48.418918 21698 fetcher.cpp:369] Fetching URI ' http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz' I1008 15:39:48.418936 21698 fetcher.cpp:243] Fetching directly into the sandbox directory I1008 15:39:48.418949 21698 fetcher.cpp:180] Fetching URI ' http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz' I1008 15:39:48.418958 21698 fetcher.cpp:127] Downloading resource from ' http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz' to '/var/run/mesos/slaves/20151007-154648-2701359370-5050-25191-S3/frameworks/20151008-123957-2701359370-5050-6382-0001/executors/20151007-154648-2701359370-5050-25191-S3/runs/507827fb-cfb0-4a1d-977d-9b9afb972c29/spark-1.5.1-bin-hadoop2.6.tgz' Failed to fetch ' http://d3kbcqa49mib13.cloudfront.net/spark-1.5.1-bin-hadoop2.6.tgz': Error downloading resource, received HTTP return code 400 Failed to synchronize with slave (it's probably exited) Long troubleshooting story short it appears that libcurl isn't finding out about my proxy. In ./3rdparty/libprocess/3rdparty/stout/include/stout/posix/net.hpp I added curl_easy_setopt(curl, CURLOPT_VERBOSE, true); curl_easy_setopt(curl, CURLOPT_PROXY, "<my squid server hostname here>"); curl_easy_setopt(curl, CURLOPT_PROXYPORT, <my squid server port here>); before CURLcode curlErrorCode = curl_easy_perform(curl); I then recompiled Mesos and the WordCount example is successful. What is the correct way to set proxy so that libcurl will make use of it? Thank you. David
