Hi,
I've made a simple script in scala that after doing a spark sql query it
sends the result to AWS's cloudwatch.
I've tested both parts individually (the spark sql one and the
cloudwatch one) and they worked fine. The trouble comes when I execute
the script through spark-submit that gives me the following exception:
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.http.params.HttpConnectionParams.setSoKeepalive(Lorg/apache/http/params/HttpParams;Z)V
at
com.amazonaws.http.HttpClientFactory.createHttpClient(HttpClientFactory.java:95)
at
com.amazonaws.http.AmazonHttpClient.<init>(AmazonHttpClient.java:193)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:120)
at
com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:104)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:171)
at
com.amazonaws.services.cloudwatch.AmazonCloudWatchClient.<init>(AmazonCloudWatchClient.java:152)
at
com.scmspain.synapse.SynapseMonitor$.sendMetrics(SynapseMonitor.scala:50)
at
com.scmspain.synapse.SynapseMonitor$.main(SynapseMonitor.scala:45)
at
com.scmspain.synapse.SynapseMonitor.main(SynapseMonitor.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at
org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
at
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
my class is com.scmspain.synapse.SynapseMonitor
I've build my script with sbt assembly, having the following
dependencies:
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.1.0" % "provided",
"org.apache.spark" %% "spark-sql" % "1.1.0" % "provided",
"com.amazonaws" % "aws-java-sdk-cloudwatch" % "1.9.10"
)
I've unzipped the generated jar assembly and searched for the
HttpConnectionParams.class and I've found it out under
org/apache/http/params and having the following signature for
setSoKeepalive:
public static void setSoKeepalive(HttpParams params, boolean
enableKeepalive)
At this point I'm stuck and didn't know where to keep looking... some
help would be greatly appreciated :)
Thank you very much!
--
Albert Manyà
alber...@eml.cc
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
