[MediaWiki-commits] [Gerrit] operations/puppet[production]: Use newer guava jar in druid hdfs cdh storage extension
Ottomata has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/369997 )
Change subject: Use newer guava jar in druid hdfs cdh storage extension
..
Use newer guava jar in druid hdfs cdh storage extension
The guava jar that cdh uses caused an error for larger jobs in Druid 0.9.2
hadoop indexing tasks:
2017-08-03T18:01:09,897 ERROR [main] org.apache.hadoop.mapred.YarnChild - Error
running child : java.lang.NoSuchMethodError:
com.google.common.io.Files.asByteSink(Ljava/io/File;[Lcom/google/common/io/FileWriteMode;)Lcom/google/common/io/ByteSink;
at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:801)
at io.druid.segment.IndexMerger.merge(IndexMerger.java:438)
at io.druid.segment.IndexMerger.persist(IndexMerger.java:186)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.persist(IndexGeneratorJob.java:510)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:688)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:478)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:421)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Bug: T170590
Change-Id: I2e7dcff24324e87dd1251a2131dea8c9e78ec1dc
---
M modules/druid/files/druid-hdfs-storage-cdh-link.sh
1 file changed, 4 insertions(+), 0 deletions(-)
Approvals:
Ottomata: Verified; Looks good to me, approved
diff --git a/modules/druid/files/druid-hdfs-storage-cdh-link.sh
b/modules/druid/files/druid-hdfs-storage-cdh-link.sh
index f0e66f5..5d8c8bf 100755
--- a/modules/druid/files/druid-hdfs-storage-cdh-link.sh
+++ b/modules/druid/files/druid-hdfs-storage-cdh-link.sh
@@ -20,6 +20,10 @@
# We need to link to the druid-hdfs-storage jar from the $source directory.
if [ "${base_jar}" == 'druid-hdfs-storage.jar' ]; then
ln -sfv $storage_jar ${dest}/${base_jar}
+# Guava is a special case, we need to keep the newer guava jar that comes
with
+# Druid's hdfs extension, rather than using the older one from Cloudera.
+elif [ "${base_jar}" == 'guava.jar' ]; then
+ln -sfv $storage_jar ${dest}/${base_jar}
# Otherwise assume this jar is provided by cdh in /var/lib/hadoop/client.
# Symlink it into our new extension from here.
else
--
To view, visit https://gerrit.wikimedia.org/r/369997
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I2e7dcff24324e87dd1251a2131dea8c9e78ec1dc
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Use newer guava jar in druid hdfs cdh storage extension
Ottomata has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/369997 )
Change subject: Use newer guava jar in druid hdfs cdh storage extension
..
Use newer guava jar in druid hdfs cdh storage extension
The guava jar that cdh uses caused an error for larger jobs in Druid 0.9.2
hadoop indexing tasks:
2017-08-03T18:01:09,897 ERROR [main] org.apache.hadoop.mapred.YarnChild - Error
running child : java.lang.NoSuchMethodError:
com.google.common.io.Files.asByteSink(Ljava/io/File;[Lcom/google/common/io/FileWriteMode;)Lcom/google/common/io/ByteSink;
at io.druid.segment.IndexMerger.makeIndexFiles(IndexMerger.java:801)
at io.druid.segment.IndexMerger.merge(IndexMerger.java:438)
at io.druid.segment.IndexMerger.persist(IndexMerger.java:186)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.persist(IndexGeneratorJob.java:510)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:688)
at
io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:478)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:421)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Bug: T170590
Change-Id: I2e7dcff24324e87dd1251a2131dea8c9e78ec1dc
---
M modules/druid/files/druid-hdfs-storage-cdh-link.sh
1 file changed, 4 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/97/369997/1
diff --git a/modules/druid/files/druid-hdfs-storage-cdh-link.sh
b/modules/druid/files/druid-hdfs-storage-cdh-link.sh
index f0e66f5..5d8c8bf 100755
--- a/modules/druid/files/druid-hdfs-storage-cdh-link.sh
+++ b/modules/druid/files/druid-hdfs-storage-cdh-link.sh
@@ -20,6 +20,10 @@
# We need to link to the druid-hdfs-storage jar from the $source directory.
if [ "${base_jar}" == 'druid-hdfs-storage.jar' ]; then
ln -sfv $storage_jar ${dest}/${base_jar}
+# Guava is a special case, we need to keep the newer guava jar that comes
with
+# Druid's hdfs extension, rather than using the older one from Cloudera.
+elif [ "${base_jar}" == 'guava.jar' ]; then
+ln -sfv $storage_jar ${dest}/${base_jar}
# Otherwise assume this jar is provided by cdh in /var/lib/hadoop/client.
# Symlink it into our new extension from here.
else
--
To view, visit https://gerrit.wikimedia.org/r/369997
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2e7dcff24324e87dd1251a2131dea8c9e78ec1dc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
