Actually we managed to resolve the problem so just in case someone else 
stumbles upon this thread I will describe how we resolved it.

In our case we were depending on a jar in our Spark project (Google Adwords 
ads-lib) which in turn depended on guava-20.0 and 
commons-configuration-1.10.
When we launched our Spark job we were getting the "NoSuchMethodError" 
reported above for MapConfiguration in commons-configuration.
We tried to package the commons-configuration lib in our fat jar but this 
did not work.
After some investigation we found that Spark includes a bunch of libs in 
its build in addition to the provided Spark libs.
It contained older versions of both commons-configuration and guava, of 
which the older version of commons-configuration did not contain the 
function which was being used by ads-lib (actually I think it was a 
constructor with a different signature).

This was in my local Spark installation on my dev machine...
ls -alrh /usr/local/Cellar/apache-
spark/2.2.0/libexec/jars | grep -E 'commons-configuration|\sguava'
-rw-r--r--    1 pmcgeever  admin   2.1M Jul  1 00:09 guava-14.0.1.jar
-rw-r--r--    1 pmcgeever  admin   292K Jul  1 00:09 
commons-configuration-1.6.jar

So we found that no matter what we did this older version of 
commons-configuration provided by Spark was being used.
After we resolved this problem we had the same issue with a method not 
being found in Guava because the older version provided by Spark was being 
used.

The fix...

We provide the jars for commons-configuration-1.10 and guava-20.0 to 
spark-submit and instruct it to load them instead of the versions it has in 
its build.

e.g.

spark-submit --deploy-mode cluster \
  --jars ./lib/guava-20.0.jar,./lib/commons-configuration-1.10.jar \
  --conf 
"spark.driver.extraClassPath=guava-20.0.jar:commons-configuration-1.10.jar" \
  --conf 
"spark.executor.extraClassPath=guava-20.0.jar:commons-configuration-1.10.jar" \
./some_spark_application.jar


I hope this helps someone.


On Friday, 28 July 2017 04:06:34 UTC+1, Peter Oliquino wrote:
>
> Hi Patrick,
>
> This appears to be a very old thread. This said, and also for better 
> tracking, could you create a new one? We'll continue our discussion on the 
> issue you are encountering from there.
>
> Best regards,
> Peter
> AdWords API Team
>

-- 
-- 
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwords-api@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to adwords-api+unsubscr...@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/adwords-api/a4a34ad7-ba64-4e2f-a439-d0a8b410ef84%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to