[ https://issues.apache.org/jira/browse/HBASE-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572296#comment-15572296 ]
stack commented on HBASE-7912: ------------------------------ My CLASSPATH is fine enough to do cluster start/stop, loadings, shell, MR bulk jobs. It is pasted below. Tell me what is missing. I can add my config if that'd help. How did you test against a cluster? What was your setup like? {code} stack@ve0524:~$ ./hbase/bin/hbase --config ~/conf_hbase classpath /home/stack/conf_hbase:/home/stack/bin/jdk//lib/tools.jar:/home/stack/hbase/bin/..:/home/stack/hbase/bin/../lib/activation-1.1.jar:/home/stack/hbase/bin/../lib/antisamy-1.4.3.jar:/home/stack/hbase/bin/../lib/aopalliance-1.0.jar:/home/stack/hbase/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/stack/hbase/bin/../lib/api-asn1-api-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/api-util-1.0.0-M20.jar:/home/stack/hbase/bin/../lib/asm-3.1.jar:/home/stack/hbase/bin/../lib/avro-1.7.4.jar:/home/stack/hbase/bin/../lib/batik-css-1.7.jar:/home/stack/hbase/bin/../lib/batik-ext-1.7.jar:/home/stack/hbase/bin/../lib/batik-util-1.7.jar:/home/stack/hbase/bin/../lib/bsh-core-2.0b4.jar:/home/stack/hbase/bin/../lib/commons-beanutils-1.7.0.jar:/home/stack/hbase/bin/../lib/commons-beanutils-core-1.7.0.jar:/home/stack/hbase/bin/../lib/commons-cli-1.2.jar:/home/stack/hbase/bin/../lib/commons-codec-1.9.jar:/home/stack/hbase/bin/../lib/commons-collections-3.2.2.jar:/home/stack/hbase/bin/../lib/commons-compress-1.4.1.jar:/home/stack/hbase/bin/../lib/commons-configuration-1.6.jar:/home/stack/hbase/bin/../lib/commons-daemon-1.0.13.jar:/home/stack/hbase/bin/../lib/commons-digester-1.8.jar:/home/stack/hbase/bin/../lib/commons-el-1.0.jar:/home/stack/hbase/bin/../lib/commons-fileupload-1.2.jar:/home/stack/hbase/bin/../lib/commons-httpclient-3.1.jar:/home/stack/hbase/bin/../lib/commons-io-2.4.jar:/home/stack/hbase/bin/../lib/commons-lang-2.6.jar:/home/stack/hbase/bin/../lib/commons-logging-1.2.jar:/home/stack/hbase/bin/../lib/commons-math-2.2.jar:/home/stack/hbase/bin/../lib/commons-math3-3.1.1.jar:/home/stack/hbase/bin/../lib/commons-net-3.1.jar:/home/stack/hbase/bin/../lib/curator-client-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-framework-2.7.1.jar:/home/stack/hbase/bin/../lib/curator-recipes-2.7.1.jar:/home/stack/hbase/bin/../lib/disruptor-3.3.0.jar:/home/stack/hbase/bin/../lib/esapi-2.1.0.jar:/home/stack/hbase/bin/../lib/findbugs-annotations-1.3.9-1.jar:/home/stack/hbase/bin/../lib/gson-2.2.4.jar:/home/stack/hbase/bin/../lib/guava-12.0.1.jar:/home/stack/hbase/bin/../lib/guice-3.0.jar:/home/stack/hbase/bin/../lib/guice-servlet-3.0.jar:/home/stack/hbase/bin/../lib/hadoop-annotations-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-auth-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-distcp-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-hdfs-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-app-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-core-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-jobclient-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-mapreduce-client-shuffle-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-api-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-client-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hadoop-yarn-server-common-2.7.1.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-annotations-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-client-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-common-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-examples-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-external-blockcache-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop2-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-hadoop-compat-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-it-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-prefix-tree-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-procedure-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-protocol-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-resource-bundle-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-rest-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-server-2.0.0-SNAPSHOT-tests.jar:/home/stack/hbase/bin/../lib/hbase-shell-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-spark-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/hbase-thrift-2.0.0-SNAPSHOT.jar:/home/stack/hbase/bin/../lib/htrace-core-3.1.0-incubating.jar:/home/stack/hbase/bin/../lib/httpclient-4.2.5.jar:/home/stack/hbase/bin/../lib/httpcore-4.4.1.jar:/home/stack/hbase/bin/../lib/jackson-core-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-jaxrs-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-mapper-asl-1.9.13.jar:/home/stack/hbase/bin/../lib/jackson-xc-1.9.13.jar:/home/stack/hbase/bin/../lib/jamon-runtime-2.4.1.jar:/home/stack/hbase/bin/../lib/jasper-compiler-5.5.23.jar:/home/stack/hbase/bin/../lib/jasper-runtime-5.5.23.jar:/home/stack/hbase/bin/../lib/javax.inject-1.jar:/home/stack/hbase/bin/../lib/java-xmlbuilder-0.4.jar:/home/stack/hbase/bin/../lib/jaxb-api-2.2.2.jar:/home/stack/hbase/bin/../lib/jaxb-impl-2.2.3-1.jar:/home/stack/hbase/bin/../lib/jcodings-1.0.8.jar:/home/stack/hbase/bin/../lib/jersey-client-1.9.jar:/home/stack/hbase/bin/../lib/jersey-core-1.9.jar:/home/stack/hbase/bin/../lib/jersey-guice-1.9.jar:/home/stack/hbase/bin/../lib/jersey-json-1.9.jar:/home/stack/hbase/bin/../lib/jersey-server-1.9.jar:/home/stack/hbase/bin/../lib/jets3t-0.9.0.jar:/home/stack/hbase/bin/../lib/jettison-1.3.3.jar:/home/stack/hbase/bin/../lib/jetty-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-sslengine-6.1.26.jar:/home/stack/hbase/bin/../lib/jetty-util-6.1.26.jar:/home/stack/hbase/bin/../lib/joni-2.1.2.jar:/home/stack/hbase/bin/../lib/jruby-complete-1.6.8.jar:/home/stack/hbase/bin/../lib/jsch-0.1.42.jar:/home/stack/hbase/bin/../lib/jsp-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/stack/hbase/bin/../lib/jsr305-1.3.9.jar:/home/stack/hbase/bin/../lib/junit-4.12.jar:/home/stack/hbase/bin/../lib/leveldbjni-all-1.8.jar:/home/stack/hbase/bin/../lib/libthrift-0.9.3.jar:/home/stack/hbase/bin/../lib/log4j-1.2.17.jar:/home/stack/hbase/bin/../lib/metrics-core-3.1.2.jar:/home/stack/hbase/bin/../lib/nekohtml-1.9.12.jar:/home/stack/hbase/bin/../lib/netty-all-4.0.30.Final.jar:/home/stack/hbase/bin/../lib/paranamer-2.3.jar:/home/stack/hbase/bin/../lib/protobuf-java-2.5.0.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5-6.1.14.jar:/home/stack/hbase/bin/../lib/servlet-api-2.5.jar:/home/stack/hbase/bin/../lib/slf4j-api-1.7.7.jar:/home/stack/hbase/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/stack/hbase/bin/../lib/snappy-java-1.1.2.jar:/home/stack/hbase/bin/../lib/spymemcached-2.11.6.jar:/home/stack/hbase/bin/../lib/xalan-2.7.0.jar:/home/stack/hbase/bin/../lib/xercesImpl-2.9.1.jar:/home/stack/hbase/bin/../lib/xml-apis-1.3.03.jar:/home/stack/hbase/bin/../lib/xml-apis-ext-1.3.04.jar:/home/stack/hbase/bin/../lib/xmlenc-0.52.jar:/home/stack/hbase/bin/../lib/xom-1.2.5.jar:/home/stack/hbase/bin/../lib/xz-1.0.jar:/home/stack/hbase/bin/../lib/zookeeper-3.4.6.jar:/home/stack/hadoop-2.7.3-SNAPSHOT/etc/hadoop:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/common/lib/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/common/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/hdfs:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/hdfs/lib/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/hdfs/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/yarn/lib/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/yarn/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/mapreduce/lib/*:/home/stack/hadoop-2.7.3-SNAPSHOT/share/hadoop/mapreduce/*:/home/stack/hadoop/contrib/capacity-scheduler/*.jar {code} > HBase Backup/Restore Based on HBase Snapshot > -------------------------------------------- > > Key: HBASE-7912 > URL: https://issues.apache.org/jira/browse/HBASE-7912 > Project: HBase > Issue Type: Sub-task > Reporter: Richard Ding > Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: Backup-and-Restore-Apache_19Sep2016.pdf, > Backup-and-Restore-Apache_9Sep2016.pdf, HBaseBackupAndRestore - v0.8.pdf, > HBaseBackupAndRestore -0.91.pdf, HBaseBackupAndRestore-v0.9.pdf, > HBaseBackupAndRestore.pdf, HBaseBackupRestore-Jira-7912-DesignDoc-v1.pdf, > HBaseBackupRestore-Jira-7912-DesignDoc-v2.pdf, > HBaseBackupRestore-Jira-7912-v4.pdf, HBaseBackupRestore-Jira-7912-v5 .pdf, > HBaseBackupRestore-Jira-7912-v6.pdf, HBase_BackupRestore-Jira-7912-CLI-v1.pdf > > > Finally, we completed the implementation of our backup/restore solution, and > would like to share with community through this jira. > We are leveraging existing hbase snapshot feature, and provide a general > solution to common users. Our full backup is using snapshot to capture > metadata locally and using exportsnapshot to move data to another cluster; > the incremental backup is using offline-WALplayer to backup HLogs; we also > leverage global distribution rolllog and flush to improve performance; other > added-on values such as convert, merge, progress report, and CLI commands. So > that a common user can backup hbase data without in-depth knowledge of hbase. > Our solution also contains some usability features for enterprise users. > The detail design document and CLI command will be attached in this jira. We > plan to use 10~12 subtasks to share each of the following features, and > document the detail implement in the subtasks: > * *Full Backup* : provide local and remote back/restore for a list of tables > * *offline-WALPlayer* to convert HLog to HFiles offline (for incremental > backup) > * *distributed* Logroll and distributed flush > * Backup *Manifest* and history > * *Incremental* backup: to build on top of full backup as daily/weekly backup > * *Convert* incremental backup WAL files into hfiles > * *Merge* several backup images into one(like merge weekly into monthly) > * *add and remove* table to and from Backup image > * *Cancel* a backup process > * backup progress *status* > * full backup based on *existing snapshot* > *-------------------------------------------------------------------------------------------------------------* > *Below is the original description, to keep here as the history for the > design and discussion back in 2013* > There have been attempts in the past to come up with a viable HBase > backup/restore solution (e.g., HBASE-4618). Recently, there are many > advancements and new features in HBase, for example, FileLink, Snapshot, and > Distributed Barrier Procedure. This is a proposal for a backup/restore > solution that utilizes these new features to achieve better performance and > consistency. > > A common practice of backup and restore in database is to first take full > baseline backup, and then periodically take incremental backup that capture > the changes since the full baseline backup. HBase cluster can store massive > amount data. Combination of full backups with incremental backups has > tremendous benefit for HBase as well. The following is a typical scenario > for full and incremental backup. > # The user takes a full backup of a table or a set of tables in HBase. > # The user schedules periodical incremental backups to capture the changes > from the full backup, or from last incremental backup. > # The user needs to restore table data to a past point of time. > # The full backup is restored to the table(s) or to different table name(s). > Then the incremental backups that are up to the desired point in time are > applied on top of the full backup. > We would support the following key features and capabilities. > * Full backup uses HBase snapshot to capture HFiles. > * Use HBase WALs to capture incremental changes, but we use bulk load of > HFiles for fast incremental restore. > * Support single table or a set of tables, and column family level backup and > restore. > * Restore to different table names. > * Support adding additional tables or CF to backup set without interruption > of incremental backup schedule. > * Support rollup/combining of incremental backups into longer period and > bigger incremental backups. > * Unified command line interface for all the above. > The solution will support HBase backup to FileSystem, either on the same > cluster or across clusters. It has the flexibility to support backup to > other devices and servers in the future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)