Ottomata has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/369997 )
Change subject: Use newer guava jar in druid hdfs cdh storage extension
......................................................................
Use newer guava jar in druid hdfs cdh storage extension
The guava jar that cdh uses caused an error for larger jobs in Druid 0.9.2
hadoop indexing tasks:
2017-08-03T18:01:09,897 ERROR [main] org.apache.hadoop.mapred.YarnChild - Error
running child : java.lang.NoSuchMethodError:
com.google.common.io.Files.asByteSink(Ljava/io/File;[Lcom/google/common/io/FileWriteMode;)Lcom/google/common/io/ByteSink;
at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:801)
at io.druid.segment.IndexMerger.merge(IndexMerger.java:438)
at io.druid.segment.IndexMerger.persist(IndexMerger.java:186)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.persist(IndexGeneratorJob.java:510)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:688)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:478)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:421)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Bug: T170590
Change-Id: I2e7dcff24324e87dd1251a2131dea8c9e78ec1dc
---
M modules/druid/files/druid-hdfs-storage-cdh-link.sh
1 file changed, 4 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/97/369997/1
diff --git a/modules/druid/files/druid-hdfs-storage-cdh-link.sh
b/modules/druid/files/druid-hdfs-storage-cdh-link.sh
index f0e66f5..5d8c8bf 100755
--- a/modules/druid/files/druid-hdfs-storage-cdh-link.sh
+++ b/modules/druid/files/druid-hdfs-storage-cdh-link.sh
@@ -20,6 +20,10 @@
# We need to link to the druid-hdfs-storage jar from the $source directory.
if [ "${base_jar}" == 'druid-hdfs-storage.jar' ]; then
ln -sfv $storage_jar ${dest}/${base_jar}
+ # Guava is a special case, we need to keep the newer guava jar that comes
with
+ # Druid's hdfs extension, rather than using the older one from Cloudera.
+ elif [ "${base_jar}" == 'guava.jar' ]; then
+ ln -sfv $storage_jar ${dest}/${base_jar}
# Otherwise assume this jar is provided by cdh in /var/lib/hadoop/client.
# Symlink it into our new extension from here.
else
--
To view, visit https://gerrit.wikimedia.org/r/369997
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2e7dcff24324e87dd1251a2131dea8c9e78ec1dc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata <ao...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
