[
https://issues.apache.org/jira/browse/BIGTOP-952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jay vyas updated BIGTOP-952:
----------------------------
Attachment: BIGTOP-952.patch
hey mister [~cos] ! Finally ! Heres an tested and functional patch for
BIGTOP-1200 based provisioning.
Shall we also create a JIRA to deprecate init-hdfs.sh in the interim before we
polish of BIGTOP-1235?
I've tested it and it works for my VMs:
{noformat}
bash-4.1$ groovy -classpath
/usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar:/root/.m2/repository/org/apache/bigtop/itest/itest-common/0.8.0-SNAPSHOT/itest-common-0.8.0-SNAPSHOT.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/etc/hadoop/conf/:/usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar
/vagrant/provision.groovy /vagrant/init-hcfs.json
Mar 13, 2014 4:01:31 AM org.apache.commons.logging.Log$info
callalpha.jar:/root/.m2/repository/org/apache/bigtop/itest/itest-common/0.8.0-SNAPSHOT/itest-common-0.8.0-SNAPSHOT.jar:/usr/lib/hadINFO:
Provisioning file system for file system from Configuration:
hdfs://vagrant.bigtop1:17020ib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/haSLF4J:
Failed to load class
"org.slf4j.impl.StaticLoggerBinder".r/lib/hadoop-hdfs/hadoop-hdfs.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.2/repository/org/apache/bigtop/itest/itest-common/0.8.0-SNAPSHOT/itest-common-0.8.0-SNAPSHOT.jar:/usr/lib/hadMar
13, 2014 4:01:31 AM org.apache.hadoop.util.NativeCodeLoader
<clinit>-2.0.6-alpha.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/haWARNING:
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicableoop/lib/protobuf-java-2.4.0a.jar
/vagrant/pop/hadoop-common-2.0.6-alpha.jar:/usr/Mar 13, 2014 4:01:32 AM
org.apache.commons.logging.Log$info
call/commons-lang-2.5.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.INFO:
PROVISIONING WITH FILE SYSTEM : class
org.apache.hadoop.hdfs.DistributedFileSystem
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /tmp null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /tmp
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /var/log yarn mapred
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /tmp/hadoop-yarn mapred mapred
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /var/log/hadoop-yarn/apps yarn mapred
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /hbase hbase hbase
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /solr solr solr
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /benchmarks null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /benchmarks
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user HCFS_SUPER_USER null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/history mapred mapred
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/jenkins jenkins null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/hive null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/hive
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/root root null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/hue hue null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/sqoop sqoop null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie oozie null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/hive null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/hive
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/mapreduce null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/mapreduce
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/mapreduce-streaming null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/mapreduce-streaming
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/distcp null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/distcp
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/pig null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/pig
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/hive null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/hive
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/mapreduce-streaming null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/mapreduce-streaming
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/distcp null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/distcp
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/pig null null
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$warn$0 call
WARNING: No owner specified for /user/oozie/share/lib/pig
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: current user: tom
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: current user: alice
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: current user: bigtop
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: Now copying Jars into the DFS for oozie
Mar 13, 2014 4:01:32 AM org.apache.commons.logging.Log$info call
INFO: This might take a few seconds...
Mar 13, 2014 4:01:35 AM org.apache.commons.logging.Log$info call
INFO: Total jars copied into the DFS : 79
{noformat}
> init-hdfs.sh is dog slow. Let's replace it with a single VM call and better
> layout management
> ---------------------------------------------------------------------------------------------
>
> Key: BIGTOP-952
> URL: https://issues.apache.org/jira/browse/BIGTOP-952
> Project: Bigtop
> Issue Type: Improvement
> Components: Deployment
> Affects Versions: 0.5.0
> Reporter: Konstantin Boudnik
> Assignee: jay vyas
> Priority: Blocker
> Fix For: 0.8.0
>
> Attachments: BIGTOP-952-tested-refined.patch,
> BIGTOP-952-tested.patch, BIGTOP-952.patch, provision2.groovy, untar.groovy
>
>
> As has been proposed in [this
> patch|https://issues.apache.org/jira/secure/attachment/12575644/untarHdfs.groovy]
> by [~rvs] there's a very efficient way of creating layout in HDFS using a
> tarfile and Groovy script with direct call into DFS APIs.
> Let's making it happen.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
