[
https://issues.apache.org/jira/browse/BIGTOP-952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13929961#comment-13929961
]
jay vyas commented on BIGTOP-952:
---------------------------------
Okay! Heres a preliminary working copy (with a few System.outs floating around).
https://gist.github.com/jayunit100/9479790
The idea here is to just copy the files directly, without using globbing. I
tried the FSShell calls and they seemed to fail. And also it seemed like bad
practice to "recode" FSShell in a different context, with all the reflection
and stuff.
Any other preliminary thoughts?
After this ill do some cleanup and then have a patch you can apply directly !
It is working to provision on my machines:
{noformat}
bash-4.1$ groovy -classpath
/usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar:/root/.m2/repository/org/apache/bigtop/itest/itest-common/0.8.0-SNAPSHOT/itest-common-0.8.0-SNAPSHOT.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/etc/hadoop/conf/:/usr/lib/hadoop/hadoop-common-2.0.6-alpha.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/lib/slf4j-api-1.6.1.jar:/usr/lib/hadoop-hdfs/hadoop-hdfs.jar:/usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar
provision.groovy /vagrant/init-hcfs.json
Mar 11, 2014 5:09:29 AM org.apache.commons.logging.Log$info call
INFO: Provisioning file system for file system from Configuration:
hdfs://vagrant.bigtop1:17020
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Mar 11, 2014 5:09:30 AM org.apache.hadoop.util.NativeCodeLoader <clinit>
WARNING: Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO:
PROVISIONING WITH FILE SYSTEM : class
org.apache.hadoop.hdfs.DistributedFileSystem
here [/tmp, 1777, null, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /tmp 1777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /tmp null null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/var/log, 1775, yarn, mapred]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /var/log 1775
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /var/log yarn mapred
here [/tmp/hadoop-yarn, 777, mapred, mapred]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /tmp/hadoop-yarn 777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /tmp/hadoop-yarn mapred mapred
here [/var/log/hadoop-yarn/apps, 1777, yarn, mapred]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /var/log/hadoop-yarn/apps 1777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /var/log/hadoop-yarn/apps yarn mapred
here [/hbase, null, hbase, hbase]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /hbase null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /hbase hbase hbase
here [/solr, null, solr, solr]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /solr null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /solr solr solr
here [/benchmarks, 777, null, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /benchmarks 777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /benchmarks null null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user, 755, HCFS_SUPER_USER, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user 755
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user HCFS_SUPER_USER null
here [/user/history, 755, mapred, mapred]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/history 755
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/history mapred mapred
here [/user/jenkins, 777, jenkins, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/jenkins 777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/jenkins jenkins null
here [/user/hive, 777, null, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/hive 777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/hive null null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/root, 777, root, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/root 777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/root root null
here [/user/hue, 777, hue, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/hue 777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/hue hue null
here [/user/sqoop, 777, sqoop, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/sqoop 777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/sqoop sqoop null
here [/user/oozie, 777, oozie]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie 777
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie oozie null
here [/user/oozie/share, null, null, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share null null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib, null, null, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib null null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/hive, null, null, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/hive null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/hive null null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/mapreduce, null, null, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/mapreduce null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/mapreduce null null
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/mapreduce-streaming, null, null, null]
Mar 11, 2014 5:09:30 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/mapreduce-streaming null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/mapreduce-streaming null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/distcp, null, null, null]
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/distcp null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/distcp null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/pig, null, null, null]
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/pig null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/pig null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share, null, null, null]
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib, null, null, null]
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/hive, null, null, null]
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/hive null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/hive null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/mapreduce-streaming, null, null, null]
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/mapreduce-streaming null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/mapreduce-streaming null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/distcp, null, null, null]
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/distcp null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/distcp null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
here [/user/oozie/share/lib/pig, null, null, null]
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: mkdirs /user/oozie/share/lib/pig null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Owner /user/oozie/share/lib/pig null null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Skipping ... user null
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: current user: tom
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: current user: alice
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: current user: bigtop
Mar 11, 2014 5:09:31 AM org.apache.commons.logging.Log$info call
INFO: Now running some basic shell commands for setting up oozie shared
libraries.
copying /usr/lib/hive/lib/hive-beeline.jar
copying /usr/lib/hive/lib/jackson-jaxrs-1.8.8.jar
copying /usr/lib/hive/lib/hive-hwi-0.11.0.jar
copying /usr/lib/hive/lib/hive-jdbc-0.11.0.jar
copying /usr/lib/hive/lib/antlr-runtime-3.4.jar
copying /usr/lib/hive/lib/ST4-4.0.4.jar
copying /usr/lib/hive/lib/jline-0.9.94.jar
copying /usr/lib/hive/lib/jetty-util-6.1.26.jar
copying /usr/lib/hive/lib/hive-common.jar
copying /usr/lib/hive/lib/hive-cli.jar
copying /usr/lib/hive/lib/hive-metastore.jar
copying /usr/lib/hive/lib/hive-shims-0.11.0.jar
copying /usr/lib/hive/lib/JavaEWAH-0.3.2.jar
copying /usr/lib/hive/lib/hive-shims.jar
copying /usr/lib/hive/lib/snappy-0.2.jar
copying /usr/lib/hive/lib/jackson-mapper-asl-1.8.8.jar
copying /usr/lib/hive/lib/hive-hwi.jar
copying /usr/lib/hive/lib/commons-dbcp-1.4.jar
copying /usr/lib/hive/lib/servlet-api-2.5-20081211.jar
copying /usr/lib/hive/lib/commons-configuration-1.6.jar
copying /usr/lib/hive/lib/jackson-core-asl-1.8.8.jar
copying /usr/lib/hive/lib/avro-1.7.1.jar
copying /usr/lib/hive/lib/derby-10.4.2.0.jar
copying /usr/lib/hive/lib/tempus-fugit-1.1.jar
copying /usr/lib/hive/lib/avro-mapred-1.7.1.jar
copying /usr/lib/hive/lib/hive-contrib.jar
copying /usr/lib/hive/lib/datanucleus-enhancer-2.0.3.jar
copying /usr/lib/hive/lib/datanucleus-rdbms-2.0.3.jar
copying /usr/lib/hive/lib/hive-metastore-0.11.0.jar
copying /usr/lib/hive/lib/log4j-1.2.16.jar
copying /usr/lib/hive/lib/hive-service-0.11.0.jar
copying /usr/lib/hive/lib/commons-lang-2.4.jar
copying /usr/lib/hive/lib/commons-io-2.4.jar
copying /usr/lib/hive/lib/hive-serde-0.11.0.jar
copying /usr/lib/hive/lib/guava-11.0.2.jar
copying /usr/lib/hive/lib/hive-common-0.11.0.jar
copying /usr/lib/hive/lib/slf4j-api-1.6.1.jar
copying /usr/lib/hive/lib/commons-pool-1.5.4.jar
copying /usr/lib/hive/lib/hive-hbase-handler-0.11.0.jar
copying /usr/lib/hive/lib/hive-contrib-0.11.0.jar
copying /usr/lib/hive/lib/commons-logging-1.0.4.jar
copying /usr/lib/hive/lib/json-20090211.jar
copying /usr/lib/hive/lib/zookeeper.jar
copying /usr/lib/hive/lib/xz-1.0.jar
copying /usr/lib/hive/lib/commons-collections-3.2.1.jar
copying /usr/lib/hive/lib/hive-hbase-handler.jar
copying /usr/lib/hive/lib/libthrift-0.9.0.jar
copying /usr/lib/hive/lib/hive-cli-0.11.0.jar
copying /usr/lib/hive/lib/commons-logging-api-1.0.4.jar
copying /usr/lib/hive/lib/javolution-5.5.1.jar
copying /usr/lib/hive/lib/jackson-xc-1.8.8.jar
copying /usr/lib/hive/lib/protobuf-java-2.4.1.jar
copying /usr/lib/hive/lib/jdo2-api-2.3-ec.jar
copying /usr/lib/hive/lib/maven-ant-tasks-2.1.3.jar
copying /usr/lib/hive/lib/datanucleus-connectionpool-2.0.3.jar
copying /usr/lib/hive/lib/jetty-6.1.26.jar
copying /usr/lib/hive/lib/hive-beeline-0.11.0.jar
copying /usr/lib/hive/lib/commons-compress-1.4.1.jar
copying /usr/lib/hive/lib/metrics-core-2.1.2.jar
copying /usr/lib/hive/lib/hive-service.jar
copying /usr/lib/hive/lib/commons-codec-1.4.jar
copying /usr/lib/hive/lib/commons-cli-1.2.jar
copying /usr/lib/hive/lib/hive-exec.jar
copying /usr/lib/hive/lib/libfb303-0.9.0.jar
copying /usr/lib/hive/lib/datanucleus-core-2.0.3.jar
copying /usr/lib/hive/lib/hive-jdbc.jar
copying /usr/lib/hive/lib/hive-serde.jar
copying /usr/lib/hive/lib/hive-exec-0.11.0.jar
copying /usr/lib/hadoop-mapreduce/hadoop-streaming.jar
copying /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.6-alpha.jar
copying /usr/lib/hadoop-mapreduce/hadoop-distcp-2.0.6-alpha.jar
copying /usr/lib/hadoop-mapreduce/hadoop-distcp.jar
copying /usr/lib/pig/lib/jython-standalone-2.5.3.jar
copying /usr/lib/pig/pig-0.11.1-smoketests.jar
copying /usr/lib/pig/piggybank.jar
copying /usr/lib/pig/pig-0.11.1.jar
copying /usr/lib/pig/pig-0.11.1-withouthadoop.jar
copying /usr/lib/pig/pig.jar
copying /usr/lib/pig/lib/jython-standalone-2.5.3.jar
{noformat}
> init-hdfs.sh is dog slow. Let's replace it with a single VM call and better
> layout management
> ---------------------------------------------------------------------------------------------
>
> Key: BIGTOP-952
> URL: https://issues.apache.org/jira/browse/BIGTOP-952
> Project: Bigtop
> Issue Type: Improvement
> Components: Deployment
> Affects Versions: 0.5.0
> Reporter: Konstantin Boudnik
> Assignee: jay vyas
> Priority: Blocker
> Fix For: 0.8.0
>
> Attachments: BIGTOP-952-tested-refined.patch,
> BIGTOP-952-tested.patch, provision2.groovy, untar.groovy
>
>
> As has been proposed in [this
> patch|https://issues.apache.org/jira/secure/attachment/12575644/untarHdfs.groovy]
> by [~rvs] there's a very efficient way of creating layout in HDFS using a
> tarfile and Groovy script with direct call into DFS APIs.
> Let's making it happen.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
