git commit: ACCUMULO-1946 Note in the README about dfs.datanode.synconclose, with a slight amendment to hadoop version application.
Updated Branches: refs/heads/1.4.5-SNAPSHOT 1b3d071a8 - 47403ba4f ACCUMULO-1946 Note in the README about dfs.datanode.synconclose, with a slight amendment to hadoop version application. Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/47403ba4 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/47403ba4 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/47403ba4 Branch: refs/heads/1.4.5-SNAPSHOT Commit: 47403ba4f5d552cabb94bc710e050dc114bcc922 Parents: 1b3d071 Author: Josh Elser els...@apache.org Authored: Mon Dec 2 12:43:58 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:41:36 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/47403ba4/README -- diff --git a/README b/README index 878bbe3..bc469fc 100644 --- a/README +++ b/README @@ -73,6 +73,11 @@ You will need to have hadoop installed and configured on your system. Accumulo 1.4.5-SNAPSHOT has been tested with hadoop version 0.20.2 and 0.20.203.0. +Additionally, it is strongly recommended that you enable 'dfs.datanode.synconclose' +(only available in Apache Hadoop =1.1.1 or =0.23) in your hdfs-site.xml configuration +file to ensure that, in the face of unexpected power loss to a datanode, files are +wholly synced to disk. + The example accumulo configuration files are placed in directories based on the memory footprint for the accumulo processes. If you are using native libraries for you tablet server in-memory map, then you can use the files in native-standalone.
[1/2] git commit: ACCUMULO-1946 Note in the README about dfs.datanode.synconclose, with a slight amendment to hadoop version application.
Updated Branches: refs/heads/1.5.1-SNAPSHOT cd96f85e0 - 747c7cdbd ACCUMULO-1946 Note in the README about dfs.datanode.synconclose, with a slight amendment to hadoop version application. Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/47403ba4 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/47403ba4 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/47403ba4 Branch: refs/heads/1.5.1-SNAPSHOT Commit: 47403ba4f5d552cabb94bc710e050dc114bcc922 Parents: 1b3d071 Author: Josh Elser els...@apache.org Authored: Mon Dec 2 12:43:58 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:41:36 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/47403ba4/README -- diff --git a/README b/README index 878bbe3..bc469fc 100644 --- a/README +++ b/README @@ -73,6 +73,11 @@ You will need to have hadoop installed and configured on your system. Accumulo 1.4.5-SNAPSHOT has been tested with hadoop version 0.20.2 and 0.20.203.0. +Additionally, it is strongly recommended that you enable 'dfs.datanode.synconclose' +(only available in Apache Hadoop =1.1.1 or =0.23) in your hdfs-site.xml configuration +file to ensure that, in the face of unexpected power loss to a datanode, files are +wholly synced to disk. + The example accumulo configuration files are placed in directories based on the memory footprint for the accumulo processes. If you are using native libraries for you tablet server in-memory map, then you can use the files in native-standalone.
[2/2] git commit: Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/747c7cdb Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/747c7cdb Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/747c7cdb Branch: refs/heads/1.5.1-SNAPSHOT Commit: 747c7cdbd38f33d5c611dded1d04b3adbe848cd8 Parents: cd96f85 47403ba Author: Josh Elser els...@apache.org Authored: Tue Dec 3 11:46:31 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:46:31 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/747c7cdb/README -- diff --cc README index ab6f3fc,bc469fc..a100077 --- a/README +++ b/README @@@ -76,28 -69,15 +76,33 @@@ the machines in the cluster and that ha found in the same location on every machine in the cluster. You will need to have password-less ssh set up as described in the hadoop documentation. -You will need to have hadoop installed and configured on your system. -Accumulo 1.4.5-SNAPSHOT has been tested with hadoop version 0.20.2 and -0.20.203.0. +You will need to have hadoop installed and configured on your system. Accumulo +1.5.0 has been tested with hadoop version 1.0.4. To avoid data loss, +you must enable HDFS durable sync. How you enable this depends on your version +of Hadoop. Please consult the table below for information regarding your version. +If you need to set the coniguration, please be sure to restart HDFS. See +ACCUMULO-623 for more information. + +HADOOP RELEASE VERSION SYNC NAME DEFAULT +Apache Hadoop 0.20.205 dfs.support.appendfalse +Apache Hadoop0.23.x dfs.support.appendtrue +Apache Hadoop 1.0.x dfs.support.appendfalse +Apache Hadoop 1.1.x dfs.durable.sync true +Apache Hadoop 2.0.0-2.0.4dfs.support.appendtrue +Cloudera CDH 3u0-3u3 true +Cloudera CDH 3u4dfs.support.appendtrue +Hortonworks HDP `1.0dfs.support.appendfalse +Hortonworks HDP `1.1dfs.support.appendfalse + +Additionally, it is strongly recommended that you enable 'dfs.datanode.synconclose' +in your hdfs-site.xml configuration file to ensure that, in the face of unexpected +power loss to a datanode, files are wholly synced to disk. + Additionally, it is strongly recommended that you enable 'dfs.datanode.synconclose' + (only available in Apache Hadoop =1.1.1 or =0.23) in your hdfs-site.xml configuration + file to ensure that, in the face of unexpected power loss to a datanode, files are + wholly synced to disk. + The example accumulo configuration files are placed in directories based on the memory footprint for the accumulo processes. If you are using native libraries for you tablet server in-memory map, then you can use the files in native-standalone.
[1/3] git commit: ACCUMULO-1946 Note in the README about dfs.datanode.synconclose, with a slight amendment to hadoop version application.
Updated Branches: refs/heads/1.6.0-SNAPSHOT 890ee25cf - b5227cf31 ACCUMULO-1946 Note in the README about dfs.datanode.synconclose, with a slight amendment to hadoop version application. Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/47403ba4 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/47403ba4 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/47403ba4 Branch: refs/heads/1.6.0-SNAPSHOT Commit: 47403ba4f5d552cabb94bc710e050dc114bcc922 Parents: 1b3d071 Author: Josh Elser els...@apache.org Authored: Mon Dec 2 12:43:58 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:41:36 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/47403ba4/README -- diff --git a/README b/README index 878bbe3..bc469fc 100644 --- a/README +++ b/README @@ -73,6 +73,11 @@ You will need to have hadoop installed and configured on your system. Accumulo 1.4.5-SNAPSHOT has been tested with hadoop version 0.20.2 and 0.20.203.0. +Additionally, it is strongly recommended that you enable 'dfs.datanode.synconclose' +(only available in Apache Hadoop =1.1.1 or =0.23) in your hdfs-site.xml configuration +file to ensure that, in the face of unexpected power loss to a datanode, files are +wholly synced to disk. + The example accumulo configuration files are placed in directories based on the memory footprint for the accumulo processes. If you are using native libraries for you tablet server in-memory map, then you can use the files in native-standalone.
[3/3] git commit: Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT
Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/b5227cf3 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/b5227cf3 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/b5227cf3 Branch: refs/heads/1.6.0-SNAPSHOT Commit: b5227cf31aeae46a8b4e92eeaf256fd4218cb1ea Parents: 890ee25 747c7cd Author: Josh Elser els...@apache.org Authored: Tue Dec 3 11:54:51 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:54:51 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/b5227cf3/README --
[2/4] git commit: Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/747c7cdb Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/747c7cdb Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/747c7cdb Branch: refs/heads/master Commit: 747c7cdbd38f33d5c611dded1d04b3adbe848cd8 Parents: cd96f85 47403ba Author: Josh Elser els...@apache.org Authored: Tue Dec 3 11:46:31 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:46:31 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/747c7cdb/README -- diff --cc README index ab6f3fc,bc469fc..a100077 --- a/README +++ b/README @@@ -76,28 -69,15 +76,33 @@@ the machines in the cluster and that ha found in the same location on every machine in the cluster. You will need to have password-less ssh set up as described in the hadoop documentation. -You will need to have hadoop installed and configured on your system. -Accumulo 1.4.5-SNAPSHOT has been tested with hadoop version 0.20.2 and -0.20.203.0. +You will need to have hadoop installed and configured on your system. Accumulo +1.5.0 has been tested with hadoop version 1.0.4. To avoid data loss, +you must enable HDFS durable sync. How you enable this depends on your version +of Hadoop. Please consult the table below for information regarding your version. +If you need to set the coniguration, please be sure to restart HDFS. See +ACCUMULO-623 for more information. + +HADOOP RELEASE VERSION SYNC NAME DEFAULT +Apache Hadoop 0.20.205 dfs.support.appendfalse +Apache Hadoop0.23.x dfs.support.appendtrue +Apache Hadoop 1.0.x dfs.support.appendfalse +Apache Hadoop 1.1.x dfs.durable.sync true +Apache Hadoop 2.0.0-2.0.4dfs.support.appendtrue +Cloudera CDH 3u0-3u3 true +Cloudera CDH 3u4dfs.support.appendtrue +Hortonworks HDP `1.0dfs.support.appendfalse +Hortonworks HDP `1.1dfs.support.appendfalse + +Additionally, it is strongly recommended that you enable 'dfs.datanode.synconclose' +in your hdfs-site.xml configuration file to ensure that, in the face of unexpected +power loss to a datanode, files are wholly synced to disk. + Additionally, it is strongly recommended that you enable 'dfs.datanode.synconclose' + (only available in Apache Hadoop =1.1.1 or =0.23) in your hdfs-site.xml configuration + file to ensure that, in the face of unexpected power loss to a datanode, files are + wholly synced to disk. + The example accumulo configuration files are placed in directories based on the memory footprint for the accumulo processes. If you are using native libraries for you tablet server in-memory map, then you can use the files in native-standalone.
[1/4] git commit: ACCUMULO-1946 Note in the README about dfs.datanode.synconclose, with a slight amendment to hadoop version application.
Updated Branches: refs/heads/master ccec95b69 - cf309e544 ACCUMULO-1946 Note in the README about dfs.datanode.synconclose, with a slight amendment to hadoop version application. Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/47403ba4 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/47403ba4 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/47403ba4 Branch: refs/heads/master Commit: 47403ba4f5d552cabb94bc710e050dc114bcc922 Parents: 1b3d071 Author: Josh Elser els...@apache.org Authored: Mon Dec 2 12:43:58 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:41:36 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/47403ba4/README -- diff --git a/README b/README index 878bbe3..bc469fc 100644 --- a/README +++ b/README @@ -73,6 +73,11 @@ You will need to have hadoop installed and configured on your system. Accumulo 1.4.5-SNAPSHOT has been tested with hadoop version 0.20.2 and 0.20.203.0. +Additionally, it is strongly recommended that you enable 'dfs.datanode.synconclose' +(only available in Apache Hadoop =1.1.1 or =0.23) in your hdfs-site.xml configuration +file to ensure that, in the face of unexpected power loss to a datanode, files are +wholly synced to disk. + The example accumulo configuration files are placed in directories based on the memory footprint for the accumulo processes. If you are using native libraries for you tablet server in-memory map, then you can use the files in native-standalone.
[4/4] git commit: Merge branch '1.6.0-SNAPSHOT'
Merge branch '1.6.0-SNAPSHOT' Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/cf309e54 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/cf309e54 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/cf309e54 Branch: refs/heads/master Commit: cf309e544be6a8e412441b9616f08a85640e3ba8 Parents: ccec95b b5227cf Author: Josh Elser els...@apache.org Authored: Tue Dec 3 11:56:38 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:56:38 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) --
[3/4] git commit: Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT
Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/b5227cf3 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/b5227cf3 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/b5227cf3 Branch: refs/heads/master Commit: b5227cf31aeae46a8b4e92eeaf256fd4218cb1ea Parents: 890ee25 747c7cd Author: Josh Elser els...@apache.org Authored: Tue Dec 3 11:54:51 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 11:54:51 2013 -0500 -- README | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/b5227cf3/README --
git commit: ACCUMULO-1905 added documentation about hsync to tserver.mutation.queue.max
Updated Branches:
refs/heads/1.5.1-SNAPSHOT 747c7cdbd - 39260cb90
ACCUMULO-1905 added documentation about hsync to tserver.mutation.queue.max
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/39260cb9
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/39260cb9
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/39260cb9
Branch: refs/heads/1.5.1-SNAPSHOT
Commit: 39260cb90be7129d91c21a3f85f625f8578ae554
Parents: 747c7cd
Author: Keith Turner ktur...@apache.org
Authored: Tue Dec 3 12:18:59 2013 -0500
Committer: Keith Turner ktur...@apache.org
Committed: Tue Dec 3 12:18:59 2013 -0500
--
core/src/main/java/org/apache/accumulo/core/conf/Property.java | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/39260cb9/core/src/main/java/org/apache/accumulo/core/conf/Property.java
--
diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index f4c1778..424b737 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -122,7 +122,10 @@ public enum Property {
TSERV_PORTSEARCH(tserver.port.search, false, PropertyType.BOOLEAN, if
the ports above are in use, search higher ports until one is available),
TSERV_CLIENTPORT(tserver.port.client, 9997, PropertyType.PORT, The port
used for handling client connections on the tablet servers),
TSERV_MUTATION_QUEUE_MAX(tserver.mutation.queue.max, 256K,
PropertyType.MEMORY,
- The amount of memory to use to store write-ahead-log
mutations-per-session before flushing them.),
+ The amount of memory to use to store write-ahead-log
mutations-per-session before flushing them. Since the buffer is per write
session, consider the
+ + max number of concurrent writer when configuring. When using
Hadoop 2, Accumulo will call hsync() on the WAL . For a small number of
+ + concurrent writers, increasing this buffer size decreases the
frequncy of hsync calls. For a large number of concurrent writers a small
buffers
+ + size is ok because of group commit.),
TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN(tserver.tablet.split.midpoint.files.max,
30, PropertyType.COUNT,
To find a tablets split points, all index files are opened. This
setting determines how many index
+ files can be opened at once. When there are more index files than
this setting multiple passes
[1/2] git commit: ACCUMULO-1905 added documentation about hsync to tserver.mutation.queue.max
Updated Branches:
refs/heads/1.6.0-SNAPSHOT b5227cf31 - eab163dcf
ACCUMULO-1905 added documentation about hsync to tserver.mutation.queue.max
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/39260cb9
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/39260cb9
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/39260cb9
Branch: refs/heads/1.6.0-SNAPSHOT
Commit: 39260cb90be7129d91c21a3f85f625f8578ae554
Parents: 747c7cd
Author: Keith Turner ktur...@apache.org
Authored: Tue Dec 3 12:18:59 2013 -0500
Committer: Keith Turner ktur...@apache.org
Committed: Tue Dec 3 12:18:59 2013 -0500
--
core/src/main/java/org/apache/accumulo/core/conf/Property.java | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/39260cb9/core/src/main/java/org/apache/accumulo/core/conf/Property.java
--
diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index f4c1778..424b737 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -122,7 +122,10 @@ public enum Property {
TSERV_PORTSEARCH(tserver.port.search, false, PropertyType.BOOLEAN, if
the ports above are in use, search higher ports until one is available),
TSERV_CLIENTPORT(tserver.port.client, 9997, PropertyType.PORT, The port
used for handling client connections on the tablet servers),
TSERV_MUTATION_QUEUE_MAX(tserver.mutation.queue.max, 256K,
PropertyType.MEMORY,
- The amount of memory to use to store write-ahead-log
mutations-per-session before flushing them.),
+ The amount of memory to use to store write-ahead-log
mutations-per-session before flushing them. Since the buffer is per write
session, consider the
+ + max number of concurrent writer when configuring. When using
Hadoop 2, Accumulo will call hsync() on the WAL . For a small number of
+ + concurrent writers, increasing this buffer size decreases the
frequncy of hsync calls. For a large number of concurrent writers a small
buffers
+ + size is ok because of group commit.),
TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN(tserver.tablet.split.midpoint.files.max,
30, PropertyType.COUNT,
To find a tablets split points, all index files are opened. This
setting determines how many index
+ files can be opened at once. When there are more index files than
this setting multiple passes
[2/2] git commit: Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT
Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/eab163dc Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/eab163dc Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/eab163dc Branch: refs/heads/1.6.0-SNAPSHOT Commit: eab163dcfd9c8eda422d8d0da6b757efba90abdf Parents: b5227cf 39260cb Author: Keith Turner ktur...@apache.org Authored: Tue Dec 3 12:20:30 2013 -0500 Committer: Keith Turner ktur...@apache.org Committed: Tue Dec 3 12:20:30 2013 -0500 -- core/src/main/java/org/apache/accumulo/core/conf/Property.java | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/eab163dc/core/src/main/java/org/apache/accumulo/core/conf/Property.java --
[1/8] ACCUMULO-1961 Fix warnings
Updated Branches:
refs/heads/1.5.1-SNAPSHOT 39260cb90 - cccdb8cb4
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
index 2472107..6f63147 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.io.Text;
/**
* @deprecated since 1.4 (will become CreateRandomRFile)
*/
+@Deprecated
public class CreateRandomMapFile {
private static int num;
private static String file;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
index f29ca76..91439dd 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
@@ -29,6 +29,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class DumpMapFile {
private static final Logger log = Logger.getLogger(DumpMapFile.class);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
b/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
index 14fbe26..4bc94e3 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
@@ -38,6 +38,7 @@ import org.apache.hadoop.io.Text;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class MMFITest {
private static MyMapFile.Reader[] mapfiles = null;
private static Text startRow;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
b/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
index 52094d0..2029b82 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
@@ -39,6 +39,7 @@ import org.apache.hadoop.io.WritableComparable;
/**
* @deprecated since 1.4
*/
+@Deprecated
class MultipleIndexIterator2 {
private MySequenceFile.Reader[] readers;
@@ -125,6 +126,7 @@ class MultipleIndexIterator2 {
/**
* @deprecated since 1.4
*/
+@Deprecated
public class MidPointPerfTest2 {
private static String dir;
private static int numFiles;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
b/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
index 3b7ea1d..f837721 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
@@ -33,6 +33,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class CountDiskRows {
private static final Logger log = Logger.getLogger(CountDiskRows.class);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
index 36c5b7e..dfcc6da 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
@@ -29,6 +29,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated
*/
+@Deprecated
public class DumpMapFile {
private static final Logger log = Logger.getLogger(DumpMapFile.class);
[06/10] Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
http://git-wip-us.apache.org/repos/asf/accumulo/blob/cccdb8cb/server/src/main/java/org/apache/accumulo/server/util/Initialize.java
--
diff --cc server/src/main/java/org/apache/accumulo/server/util/Initialize.java
index 6b8c1d9,000..d196e35
mode 100644,00..100644
--- a/server/src/main/java/org/apache/accumulo/server/util/Initialize.java
+++ b/server/src/main/java/org/apache/accumulo/server/util/Initialize.java
@@@ -1,533 -1,0 +1,535 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.server.util;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map.Entry;
+import java.util.UUID;
+
+import jline.ConsoleReader;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.cli.Help;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.impl.thrift.ThriftSecurityException;
+import org.apache.accumulo.core.conf.AccumuloConfiguration;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.conf.SiteConfiguration;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.KeyExtent;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.file.FileOperations;
+import org.apache.accumulo.core.file.FileSKVWriter;
+import org.apache.accumulo.core.file.FileUtil;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.master.state.tables.TableState;
+import org.apache.accumulo.core.master.thrift.MasterGoalState;
+import org.apache.accumulo.core.security.SecurityUtil;
+import org.apache.accumulo.core.util.CachedConfiguration;
+import org.apache.accumulo.core.zookeeper.ZooUtil;
+import org.apache.accumulo.fate.zookeeper.IZooReaderWriter;
+import org.apache.accumulo.fate.zookeeper.ZooUtil.NodeExistsPolicy;
+import org.apache.accumulo.fate.zookeeper.ZooUtil.NodeMissingPolicy;
+import org.apache.accumulo.server.ServerConstants;
+import org.apache.accumulo.server.client.HdfsZooInstance;
+import org.apache.accumulo.server.conf.ServerConfiguration;
+import org.apache.accumulo.server.constraints.MetadataConstraints;
+import org.apache.accumulo.server.iterators.MetadataBulkLoadFilter;
+import org.apache.accumulo.server.master.state.tables.TableManager;
+import org.apache.accumulo.server.security.AuditedSecurityOperation;
+import org.apache.accumulo.server.security.SecurityConstants;
+import org.apache.accumulo.server.tabletserver.TabletTime;
+import org.apache.accumulo.server.zookeeper.ZooReaderWriter;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.log4j.Logger;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.ZooDefs.Ids;
+
+import com.beust.jcommander.Parameter;
+
+/**
+ * This class is used to setup the directory structure and the root tablet to
get an instance started
+ *
+ */
+public class Initialize {
+ private static final Logger log = Logger.getLogger(Initialize.class);
+ private static final String DEFAULT_ROOT_USER = root;
-
++
+ private static ConsoleReader reader = null;
+ private static IZooReaderWriter zoo = ZooReaderWriter.getInstance();
-
++
+ private static ConsoleReader getConsoleReader() throws IOException {
+if (reader == null)
+ reader = new ConsoleReader();
+return reader;
+ }
++
+ /**
+ * Sets this class's ZooKeeper reader/writer.
-*
-* @param izoo reader/writer
++ *
++ * @param izoo
++ * reader/writer
+ */
+ static void setZooReaderWriter(IZooReaderWriter izoo) {
+zoo = izoo;
+ }
++
+ /**
+ * Gets this class's ZooKeeper reader/writer.
-*
++ *
+ * @return reader/writer
+ */
+ static IZooReaderWriter getZooReaderWriter() {
+return zoo;
+ }
-
++
+ private static HashMapString,String initialMetadataConf = new
[01/10] ACCUMULO-1961 Fix warnings
Updated Branches:
refs/heads/1.6.0-SNAPSHOT f1f28be82 - 8a34937aa
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
index 2472107..6f63147 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.io.Text;
/**
* @deprecated since 1.4 (will become CreateRandomRFile)
*/
+@Deprecated
public class CreateRandomMapFile {
private static int num;
private static String file;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
index f29ca76..91439dd 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
@@ -29,6 +29,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class DumpMapFile {
private static final Logger log = Logger.getLogger(DumpMapFile.class);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
b/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
index 14fbe26..4bc94e3 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
@@ -38,6 +38,7 @@ import org.apache.hadoop.io.Text;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class MMFITest {
private static MyMapFile.Reader[] mapfiles = null;
private static Text startRow;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
b/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
index 52094d0..2029b82 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
@@ -39,6 +39,7 @@ import org.apache.hadoop.io.WritableComparable;
/**
* @deprecated since 1.4
*/
+@Deprecated
class MultipleIndexIterator2 {
private MySequenceFile.Reader[] readers;
@@ -125,6 +126,7 @@ class MultipleIndexIterator2 {
/**
* @deprecated since 1.4
*/
+@Deprecated
public class MidPointPerfTest2 {
private static String dir;
private static int numFiles;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
b/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
index 3b7ea1d..f837721 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
@@ -33,6 +33,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class CountDiskRows {
private static final Logger log = Logger.getLogger(CountDiskRows.class);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
index 36c5b7e..dfcc6da 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
@@ -29,6 +29,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated
*/
+@Deprecated
public class DumpMapFile {
private static final Logger log = Logger.getLogger(DumpMapFile.class);
[09/10] Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT
http://git-wip-us.apache.org/repos/asf/accumulo/blob/8a34937a/server/tserver/src/test/java/org/apache/accumulo/tserver/log/MultiReaderTest.java
--
diff --cc
server/tserver/src/test/java/org/apache/accumulo/tserver/log/MultiReaderTest.java
index 168842e,000..53fb27c
mode 100644,00..100644
---
a/server/tserver/src/test/java/org/apache/accumulo/tserver/log/MultiReaderTest.java
+++
b/server/tserver/src/test/java/org/apache/accumulo/tserver/log/MultiReaderTest.java
@@@ -1,149 -1,0 +1,147 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.tserver.log;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+
+import org.apache.accumulo.server.fs.VolumeManager;
+import org.apache.accumulo.server.fs.VolumeManagerImpl;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.MapFile.Writer;
- import org.apache.log4j.Level;
- import org.apache.log4j.Logger;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class MultiReaderTest {
+
+ VolumeManager fs;
+ TemporaryFolder root = new TemporaryFolder(new
File(System.getProperty(user.dir) + /target));
+
+ @Before
+ public void setUp() throws Exception {
+fs = VolumeManagerImpl.getLocal();
+root.create();
+String path = root.getRoot().getAbsolutePath();
+Path root = new Path(file:// + path + /manyMaps);
+fs.mkdirs(root);
+fs.create(new Path(root, finished)).close();
+FileSystem ns = fs.getDefaultVolume();
-
++
+@SuppressWarnings(deprecation)
+Writer oddWriter = new Writer(ns.getConf(), ns, new Path(root,
odd).toString(), IntWritable.class, BytesWritable.class);
+BytesWritable value = new BytesWritable(someValue.getBytes());
+for (int i = 1; i 1000; i += 2) {
+ oddWriter.append(new IntWritable(i), value);
+}
+oddWriter.close();
-
++
+@SuppressWarnings(deprecation)
+Writer evenWriter = new Writer(ns.getConf(), ns, new Path(root,
even).toString(), IntWritable.class, BytesWritable.class);
+for (int i = 0; i 1000; i += 2) {
+ if (i == 10)
+continue;
+ evenWriter.append(new IntWritable(i), value);
+}
+evenWriter.close();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+root.create();
+ }
+
+ private void scan(MultiReader reader, int start) throws IOException {
+IntWritable key = new IntWritable();
+BytesWritable value = new BytesWritable();
+
+for (int i = start + 1; i 1000; i++) {
+ if (i == 10)
+continue;
+ assertTrue(reader.next(key, value));
+ assertEquals(i, key.get());
+}
+ }
+
+ private void scanOdd(MultiReader reader, int start) throws IOException {
+IntWritable key = new IntWritable();
+BytesWritable value = new BytesWritable();
+
+for (int i = start + 2; i 1000; i += 2) {
+ assertTrue(reader.next(key, value));
+ assertEquals(i, key.get());
+}
+ }
+
+ @Test
+ public void testMultiReader() throws IOException {
+Path manyMaps = new Path(file:// + root.getRoot().getAbsolutePath() +
/manyMaps);
+MultiReader reader = new MultiReader(fs, manyMaps);
+IntWritable key = new IntWritable();
+BytesWritable value = new BytesWritable();
+
+for (int i = 0; i 1000; i++) {
+ if (i == 10)
+continue;
+ assertTrue(reader.next(key, value));
+ assertEquals(i, key.get());
+}
+assertEquals(value.compareTo(new BytesWritable(someValue.getBytes())),
0);
+assertFalse(reader.next(key, value));
+
+key.set(500);
+assertTrue(reader.seek(key));
+scan(reader, 500);
+key.set(10);
+assertFalse(reader.seek(key));
+scan(reader, 10);
+key.set(1000);
+assertFalse(reader.seek(key));
+
[1/5] ACCUMULO-1961 Fix warnings
Updated Branches:
refs/heads/1.4.5-SNAPSHOT 47403ba4f - a3a889554
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
index 2472107..6f63147 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.io.Text;
/**
* @deprecated since 1.4 (will become CreateRandomRFile)
*/
+@Deprecated
public class CreateRandomMapFile {
private static int num;
private static String file;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
index f29ca76..91439dd 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
@@ -29,6 +29,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class DumpMapFile {
private static final Logger log = Logger.getLogger(DumpMapFile.class);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
b/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
index 14fbe26..4bc94e3 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
@@ -38,6 +38,7 @@ import org.apache.hadoop.io.Text;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class MMFITest {
private static MyMapFile.Reader[] mapfiles = null;
private static Text startRow;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
b/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
index 52094d0..2029b82 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
@@ -39,6 +39,7 @@ import org.apache.hadoop.io.WritableComparable;
/**
* @deprecated since 1.4
*/
+@Deprecated
class MultipleIndexIterator2 {
private MySequenceFile.Reader[] readers;
@@ -125,6 +126,7 @@ class MultipleIndexIterator2 {
/**
* @deprecated since 1.4
*/
+@Deprecated
public class MidPointPerfTest2 {
private static String dir;
private static int numFiles;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
b/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
index 3b7ea1d..f837721 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
@@ -33,6 +33,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class CountDiskRows {
private static final Logger log = Logger.getLogger(CountDiskRows.class);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
index 36c5b7e..dfcc6da 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
@@ -29,6 +29,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated
*/
+@Deprecated
public class DumpMapFile {
private static final Logger log = Logger.getLogger(DumpMapFile.class);
[03/14] git commit: ACCUMULO-1905 increased tserver.mutation.queue.max default to 1M
ACCUMULO-1905 increased tserver.mutation.queue.max default to 1M
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/f1f28be8
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/f1f28be8
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/f1f28be8
Branch: refs/heads/master
Commit: f1f28be82a2c27ad905d3bde0669b10ab93d4cf5
Parents: eab163d
Author: Keith Turner ktur...@apache.org
Authored: Tue Dec 3 12:26:54 2013 -0500
Committer: Keith Turner ktur...@apache.org
Committed: Tue Dec 3 12:26:54 2013 -0500
--
core/src/main/java/org/apache/accumulo/core/conf/Property.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/f1f28be8/core/src/main/java/org/apache/accumulo/core/conf/Property.java
--
diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index 4fd2718..47f9c37 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -172,7 +172,7 @@ public enum Property {
TSERV_INDEXCACHE_SIZE(tserver.cache.index.size, 512M,
PropertyType.MEMORY, Specifies the size of the cache for file indices.),
TSERV_PORTSEARCH(tserver.port.search, false, PropertyType.BOOLEAN, if
the ports above are in use, search higher ports until one is available),
TSERV_CLIENTPORT(tserver.port.client, 9997, PropertyType.PORT, The port
used for handling client connections on the tablet servers),
- TSERV_MUTATION_QUEUE_MAX(tserver.mutation.queue.max, 256K,
PropertyType.MEMORY,
+ TSERV_MUTATION_QUEUE_MAX(tserver.mutation.queue.max, 1M,
PropertyType.MEMORY,
The amount of memory to use to store write-ahead-log
mutations-per-session before flushing them. Since the buffer is per write
session, consider the
+ max number of concurrent writer when configuring. When using
Hadoop 2, Accumulo will call hsync() on the WAL . For a small number of
+ concurrent writers, increasing this buffer size decreases the
frequncy of hsync calls. For a large number of concurrent writers a small
buffers
[02/14] git commit: Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT
Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/eab163dc Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/eab163dc Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/eab163dc Branch: refs/heads/master Commit: eab163dcfd9c8eda422d8d0da6b757efba90abdf Parents: b5227cf 39260cb Author: Keith Turner ktur...@apache.org Authored: Tue Dec 3 12:20:30 2013 -0500 Committer: Keith Turner ktur...@apache.org Committed: Tue Dec 3 12:20:30 2013 -0500 -- core/src/main/java/org/apache/accumulo/core/conf/Property.java | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/eab163dc/core/src/main/java/org/apache/accumulo/core/conf/Property.java --
[01/14] git commit: ACCUMULO-1905 added documentation about hsync to tserver.mutation.queue.max
Updated Branches:
refs/heads/master cf309e544 - 5a8a28af9
ACCUMULO-1905 added documentation about hsync to tserver.mutation.queue.max
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/39260cb9
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/39260cb9
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/39260cb9
Branch: refs/heads/master
Commit: 39260cb90be7129d91c21a3f85f625f8578ae554
Parents: 747c7cd
Author: Keith Turner ktur...@apache.org
Authored: Tue Dec 3 12:18:59 2013 -0500
Committer: Keith Turner ktur...@apache.org
Committed: Tue Dec 3 12:18:59 2013 -0500
--
core/src/main/java/org/apache/accumulo/core/conf/Property.java | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/39260cb9/core/src/main/java/org/apache/accumulo/core/conf/Property.java
--
diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index f4c1778..424b737 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -122,7 +122,10 @@ public enum Property {
TSERV_PORTSEARCH(tserver.port.search, false, PropertyType.BOOLEAN, if
the ports above are in use, search higher ports until one is available),
TSERV_CLIENTPORT(tserver.port.client, 9997, PropertyType.PORT, The port
used for handling client connections on the tablet servers),
TSERV_MUTATION_QUEUE_MAX(tserver.mutation.queue.max, 256K,
PropertyType.MEMORY,
- The amount of memory to use to store write-ahead-log
mutations-per-session before flushing them.),
+ The amount of memory to use to store write-ahead-log
mutations-per-session before flushing them. Since the buffer is per write
session, consider the
+ + max number of concurrent writer when configuring. When using
Hadoop 2, Accumulo will call hsync() on the WAL . For a small number of
+ + concurrent writers, increasing this buffer size decreases the
frequncy of hsync calls. For a large number of concurrent writers a small
buffers
+ + size is ok because of group commit.),
TSERV_TABLET_SPLIT_FINDMIDPOINT_MAXOPEN(tserver.tablet.split.midpoint.files.max,
30, PropertyType.COUNT,
To find a tablets split points, all index files are opened. This
setting determines how many index
+ files can be opened at once. When there are more index files than
this setting multiple passes
[14/14] git commit: Merge branch '1.6.0-SNAPSHOT'
Merge branch '1.6.0-SNAPSHOT' Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/5a8a28af Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/5a8a28af Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/5a8a28af Branch: refs/heads/master Commit: 5a8a28af9e8084cd7d553d182d04bbd32059b080 Parents: cf309e5 8a34937 Author: Christopher Tubbs ctubb...@apache.org Authored: Tue Dec 3 16:15:27 2013 -0500 Committer: Christopher Tubbs ctubb...@apache.org Committed: Tue Dec 3 16:15:27 2013 -0500 -- .../core/client/mapred/InputFormatBase.java | 5 +- .../core/client/mapreduce/InputFormatBase.java | 4 +- .../org/apache/accumulo/core/conf/Property.java | 7 +- .../mapreduce/AccumuloInputFormatTest.java | 4 +- .../apache/accumulo/server/init/Initialize.java | 97 ++-- .../accumulo/server/init/InitializeTest.java| 17 ++-- .../accumulo/tserver/log/MultiReaderTest.java | 6 +- 7 files changed, 74 insertions(+), 66 deletions(-) --
[12/14] Merge branch '1.5.1-SNAPSHOT' into 1.6.0-SNAPSHOT
http://git-wip-us.apache.org/repos/asf/accumulo/blob/8a34937a/server/tserver/src/test/java/org/apache/accumulo/tserver/log/MultiReaderTest.java
--
diff --cc
server/tserver/src/test/java/org/apache/accumulo/tserver/log/MultiReaderTest.java
index 168842e,000..53fb27c
mode 100644,00..100644
---
a/server/tserver/src/test/java/org/apache/accumulo/tserver/log/MultiReaderTest.java
+++
b/server/tserver/src/test/java/org/apache/accumulo/tserver/log/MultiReaderTest.java
@@@ -1,149 -1,0 +1,147 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.tserver.log;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+import java.io.IOException;
+
+import org.apache.accumulo.server.fs.VolumeManager;
+import org.apache.accumulo.server.fs.VolumeManagerImpl;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.BytesWritable;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.MapFile.Writer;
- import org.apache.log4j.Level;
- import org.apache.log4j.Logger;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+public class MultiReaderTest {
+
+ VolumeManager fs;
+ TemporaryFolder root = new TemporaryFolder(new
File(System.getProperty(user.dir) + /target));
+
+ @Before
+ public void setUp() throws Exception {
+fs = VolumeManagerImpl.getLocal();
+root.create();
+String path = root.getRoot().getAbsolutePath();
+Path root = new Path(file:// + path + /manyMaps);
+fs.mkdirs(root);
+fs.create(new Path(root, finished)).close();
+FileSystem ns = fs.getDefaultVolume();
-
++
+@SuppressWarnings(deprecation)
+Writer oddWriter = new Writer(ns.getConf(), ns, new Path(root,
odd).toString(), IntWritable.class, BytesWritable.class);
+BytesWritable value = new BytesWritable(someValue.getBytes());
+for (int i = 1; i 1000; i += 2) {
+ oddWriter.append(new IntWritable(i), value);
+}
+oddWriter.close();
-
++
+@SuppressWarnings(deprecation)
+Writer evenWriter = new Writer(ns.getConf(), ns, new Path(root,
even).toString(), IntWritable.class, BytesWritable.class);
+for (int i = 0; i 1000; i += 2) {
+ if (i == 10)
+continue;
+ evenWriter.append(new IntWritable(i), value);
+}
+evenWriter.close();
+ }
+
+ @After
+ public void tearDown() throws Exception {
+root.create();
+ }
+
+ private void scan(MultiReader reader, int start) throws IOException {
+IntWritable key = new IntWritable();
+BytesWritable value = new BytesWritable();
+
+for (int i = start + 1; i 1000; i++) {
+ if (i == 10)
+continue;
+ assertTrue(reader.next(key, value));
+ assertEquals(i, key.get());
+}
+ }
+
+ private void scanOdd(MultiReader reader, int start) throws IOException {
+IntWritable key = new IntWritable();
+BytesWritable value = new BytesWritable();
+
+for (int i = start + 2; i 1000; i += 2) {
+ assertTrue(reader.next(key, value));
+ assertEquals(i, key.get());
+}
+ }
+
+ @Test
+ public void testMultiReader() throws IOException {
+Path manyMaps = new Path(file:// + root.getRoot().getAbsolutePath() +
/manyMaps);
+MultiReader reader = new MultiReader(fs, manyMaps);
+IntWritable key = new IntWritable();
+BytesWritable value = new BytesWritable();
+
+for (int i = 0; i 1000; i++) {
+ if (i == 10)
+continue;
+ assertTrue(reader.next(key, value));
+ assertEquals(i, key.get());
+}
+assertEquals(value.compareTo(new BytesWritable(someValue.getBytes())),
0);
+assertFalse(reader.next(key, value));
+
+key.set(500);
+assertTrue(reader.seek(key));
+scan(reader, 500);
+key.set(10);
+assertFalse(reader.seek(key));
+scan(reader, 10);
+key.set(1000);
+assertFalse(reader.seek(key));
+
[11/14] git commit: Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/cccdb8cb Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/cccdb8cb Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/cccdb8cb Branch: refs/heads/master Commit: cccdb8cb4d4645df25e07e96d7ecac333b316348 Parents: 39260cb a3a8895 Author: Christopher Tubbs ctubb...@apache.org Authored: Tue Dec 3 15:30:01 2013 -0500 Committer: Christopher Tubbs ctubb...@apache.org Committed: Tue Dec 3 15:30:01 2013 -0500 -- .../core/client/mapred/InputFormatBase.java | 185 ++-- .../core/client/mapreduce/InputFormatBase.java | 282 +-- .../core/client/mapreduce/RangeInputSplit.java | 72 ++--- .../mapreduce/AccumuloInputFormatTest.java | 22 +- .../apache/accumulo/server/util/Initialize.java | 116 .../tabletserver/log/MultiReaderTest.java | 33 ++- .../accumulo/server/util/InitializeTest.java| 11 +- 7 files changed, 363 insertions(+), 358 deletions(-) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/cccdb8cb/core/src/main/java/org/apache/accumulo/core/client/mapred/InputFormatBase.java -- diff --cc core/src/main/java/org/apache/accumulo/core/client/mapred/InputFormatBase.java index 0be4706,000..fe75453 mode 100644,00..100644 --- a/core/src/main/java/org/apache/accumulo/core/client/mapred/InputFormatBase.java +++ b/core/src/main/java/org/apache/accumulo/core/client/mapred/InputFormatBase.java @@@ -1,913 -1,0 +1,910 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the License); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.accumulo.core.client.mapred; + +import java.io.IOException; +import java.net.InetAddress; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; + +import org.apache.accumulo.core.Constants; +import org.apache.accumulo.core.client.AccumuloException; +import org.apache.accumulo.core.client.AccumuloSecurityException; +import org.apache.accumulo.core.client.ClientSideIteratorScanner; +import org.apache.accumulo.core.client.Connector; +import org.apache.accumulo.core.client.Instance; +import org.apache.accumulo.core.client.IsolatedScanner; +import org.apache.accumulo.core.client.IteratorSetting; +import org.apache.accumulo.core.client.RowIterator; +import org.apache.accumulo.core.client.Scanner; +import org.apache.accumulo.core.client.TableDeletedException; +import org.apache.accumulo.core.client.TableNotFoundException; +import org.apache.accumulo.core.client.TableOfflineException; +import org.apache.accumulo.core.client.ZooKeeperInstance; +import org.apache.accumulo.core.client.impl.OfflineScanner; +import org.apache.accumulo.core.client.impl.Tables; +import org.apache.accumulo.core.client.impl.TabletLocator; - import org.apache.accumulo.core.client.mapred.RangeInputSplit; +import org.apache.accumulo.core.client.mapreduce.lib.util.InputConfigurator; +import org.apache.accumulo.core.client.mock.MockInstance; +import org.apache.accumulo.core.client.security.tokens.AuthenticationToken; +import org.apache.accumulo.core.data.Key; +import org.apache.accumulo.core.data.KeyExtent; +import org.apache.accumulo.core.data.PartialKey; +import org.apache.accumulo.core.data.Range; +import org.apache.accumulo.core.data.Value; +import org.apache.accumulo.core.master.state.tables.TableState; +import org.apache.accumulo.core.security.Authorizations; +import org.apache.accumulo.core.security.CredentialHelper; +import org.apache.accumulo.core.security.thrift.TCredentials; +import org.apache.accumulo.core.util.Pair; +import org.apache.accumulo.core.util.UtilWaitThread; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapred.InputFormat; +import org.apache.hadoop.mapred.InputSplit; +import
[04/14] ACCUMULO-1961 Fix warnings
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
index 2472107..6f63147 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/test/CreateRandomMapFile.java
@@ -33,6 +33,7 @@ import org.apache.hadoop.io.Text;
/**
* @deprecated since 1.4 (will become CreateRandomRFile)
*/
+@Deprecated
public class CreateRandomMapFile {
private static int num;
private static String file;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
index f29ca76..91439dd 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/DumpMapFile.java
@@ -29,6 +29,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class DumpMapFile {
private static final Logger log = Logger.getLogger(DumpMapFile.class);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
b/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
index 14fbe26..4bc94e3 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/test/MMFITest.java
@@ -38,6 +38,7 @@ import org.apache.hadoop.io.Text;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class MMFITest {
private static MyMapFile.Reader[] mapfiles = null;
private static Text startRow;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
b/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
index 52094d0..2029b82 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/test/MidPointPerfTest2.java
@@ -39,6 +39,7 @@ import org.apache.hadoop.io.WritableComparable;
/**
* @deprecated since 1.4
*/
+@Deprecated
class MultipleIndexIterator2 {
private MySequenceFile.Reader[] readers;
@@ -125,6 +126,7 @@ class MultipleIndexIterator2 {
/**
* @deprecated since 1.4
*/
+@Deprecated
public class MidPointPerfTest2 {
private static String dir;
private static int numFiles;
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
b/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
index 3b7ea1d..f837721 100644
---
a/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
+++
b/src/server/src/main/java/org/apache/accumulo/server/util/CountDiskRows.java
@@ -33,6 +33,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated since 1.4
*/
+@Deprecated
public class CountDiskRows {
private static final Logger log = Logger.getLogger(CountDiskRows.class);
http://git-wip-us.apache.org/repos/asf/accumulo/blob/a3a88955/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
--
diff --git
a/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
b/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
index 36c5b7e..dfcc6da 100644
--- a/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
+++ b/src/server/src/main/java/org/apache/accumulo/server/util/DumpMapFile.java
@@ -29,6 +29,7 @@ import org.apache.log4j.Logger;
/**
* @deprecated
*/
+@Deprecated
public class DumpMapFile {
private static final Logger log = Logger.getLogger(DumpMapFile.class);
[09/14] Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
http://git-wip-us.apache.org/repos/asf/accumulo/blob/cccdb8cb/server/src/main/java/org/apache/accumulo/server/util/Initialize.java
--
diff --cc server/src/main/java/org/apache/accumulo/server/util/Initialize.java
index 6b8c1d9,000..d196e35
mode 100644,00..100644
--- a/server/src/main/java/org/apache/accumulo/server/util/Initialize.java
+++ b/server/src/main/java/org/apache/accumulo/server/util/Initialize.java
@@@ -1,533 -1,0 +1,535 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.server.util;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Locale;
+import java.util.Map.Entry;
+import java.util.UUID;
+
+import jline.ConsoleReader;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.cli.Help;
+import org.apache.accumulo.core.client.AccumuloSecurityException;
+import org.apache.accumulo.core.client.impl.thrift.ThriftSecurityException;
+import org.apache.accumulo.core.conf.AccumuloConfiguration;
+import org.apache.accumulo.core.conf.Property;
+import org.apache.accumulo.core.conf.SiteConfiguration;
+import org.apache.accumulo.core.data.Key;
+import org.apache.accumulo.core.data.KeyExtent;
+import org.apache.accumulo.core.data.Value;
+import org.apache.accumulo.core.file.FileOperations;
+import org.apache.accumulo.core.file.FileSKVWriter;
+import org.apache.accumulo.core.file.FileUtil;
+import org.apache.accumulo.core.iterators.user.VersioningIterator;
+import org.apache.accumulo.core.master.state.tables.TableState;
+import org.apache.accumulo.core.master.thrift.MasterGoalState;
+import org.apache.accumulo.core.security.SecurityUtil;
+import org.apache.accumulo.core.util.CachedConfiguration;
+import org.apache.accumulo.core.zookeeper.ZooUtil;
+import org.apache.accumulo.fate.zookeeper.IZooReaderWriter;
+import org.apache.accumulo.fate.zookeeper.ZooUtil.NodeExistsPolicy;
+import org.apache.accumulo.fate.zookeeper.ZooUtil.NodeMissingPolicy;
+import org.apache.accumulo.server.ServerConstants;
+import org.apache.accumulo.server.client.HdfsZooInstance;
+import org.apache.accumulo.server.conf.ServerConfiguration;
+import org.apache.accumulo.server.constraints.MetadataConstraints;
+import org.apache.accumulo.server.iterators.MetadataBulkLoadFilter;
+import org.apache.accumulo.server.master.state.tables.TableManager;
+import org.apache.accumulo.server.security.AuditedSecurityOperation;
+import org.apache.accumulo.server.security.SecurityConstants;
+import org.apache.accumulo.server.tabletserver.TabletTime;
+import org.apache.accumulo.server.zookeeper.ZooReaderWriter;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.io.Text;
+import org.apache.log4j.Logger;
+import org.apache.zookeeper.KeeperException;
+import org.apache.zookeeper.ZooDefs.Ids;
+
+import com.beust.jcommander.Parameter;
+
+/**
+ * This class is used to setup the directory structure and the root tablet to
get an instance started
+ *
+ */
+public class Initialize {
+ private static final Logger log = Logger.getLogger(Initialize.class);
+ private static final String DEFAULT_ROOT_USER = root;
-
++
+ private static ConsoleReader reader = null;
+ private static IZooReaderWriter zoo = ZooReaderWriter.getInstance();
-
++
+ private static ConsoleReader getConsoleReader() throws IOException {
+if (reader == null)
+ reader = new ConsoleReader();
+return reader;
+ }
++
+ /**
+ * Sets this class's ZooKeeper reader/writer.
-*
-* @param izoo reader/writer
++ *
++ * @param izoo
++ * reader/writer
+ */
+ static void setZooReaderWriter(IZooReaderWriter izoo) {
+zoo = izoo;
+ }
++
+ /**
+ * Gets this class's ZooKeeper reader/writer.
-*
++ *
+ * @return reader/writer
+ */
+ static IZooReaderWriter getZooReaderWriter() {
+return zoo;
+ }
-
++
+ private static HashMapString,String initialMetadataConf = new
[08/10] git commit: Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT
Merge branch '1.4.5-SNAPSHOT' into 1.5.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/cccdb8cb Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/cccdb8cb Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/cccdb8cb Branch: refs/heads/1.6.0-SNAPSHOT Commit: cccdb8cb4d4645df25e07e96d7ecac333b316348 Parents: 39260cb a3a8895 Author: Christopher Tubbs ctubb...@apache.org Authored: Tue Dec 3 15:30:01 2013 -0500 Committer: Christopher Tubbs ctubb...@apache.org Committed: Tue Dec 3 15:30:01 2013 -0500 -- .../core/client/mapred/InputFormatBase.java | 185 ++-- .../core/client/mapreduce/InputFormatBase.java | 282 +-- .../core/client/mapreduce/RangeInputSplit.java | 72 ++--- .../mapreduce/AccumuloInputFormatTest.java | 22 +- .../apache/accumulo/server/util/Initialize.java | 116 .../tabletserver/log/MultiReaderTest.java | 33 ++- .../accumulo/server/util/InitializeTest.java| 11 +- 7 files changed, 363 insertions(+), 358 deletions(-) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/cccdb8cb/core/src/main/java/org/apache/accumulo/core/client/mapred/InputFormatBase.java -- diff --cc core/src/main/java/org/apache/accumulo/core/client/mapred/InputFormatBase.java index 0be4706,000..fe75453 mode 100644,00..100644 --- a/core/src/main/java/org/apache/accumulo/core/client/mapred/InputFormatBase.java +++ b/core/src/main/java/org/apache/accumulo/core/client/mapred/InputFormatBase.java @@@ -1,913 -1,0 +1,910 @@@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the License); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an AS IS BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.accumulo.core.client.mapred; + +import java.io.IOException; +import java.net.InetAddress; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.Set; + +import org.apache.accumulo.core.Constants; +import org.apache.accumulo.core.client.AccumuloException; +import org.apache.accumulo.core.client.AccumuloSecurityException; +import org.apache.accumulo.core.client.ClientSideIteratorScanner; +import org.apache.accumulo.core.client.Connector; +import org.apache.accumulo.core.client.Instance; +import org.apache.accumulo.core.client.IsolatedScanner; +import org.apache.accumulo.core.client.IteratorSetting; +import org.apache.accumulo.core.client.RowIterator; +import org.apache.accumulo.core.client.Scanner; +import org.apache.accumulo.core.client.TableDeletedException; +import org.apache.accumulo.core.client.TableNotFoundException; +import org.apache.accumulo.core.client.TableOfflineException; +import org.apache.accumulo.core.client.ZooKeeperInstance; +import org.apache.accumulo.core.client.impl.OfflineScanner; +import org.apache.accumulo.core.client.impl.Tables; +import org.apache.accumulo.core.client.impl.TabletLocator; - import org.apache.accumulo.core.client.mapred.RangeInputSplit; +import org.apache.accumulo.core.client.mapreduce.lib.util.InputConfigurator; +import org.apache.accumulo.core.client.mock.MockInstance; +import org.apache.accumulo.core.client.security.tokens.AuthenticationToken; +import org.apache.accumulo.core.data.Key; +import org.apache.accumulo.core.data.KeyExtent; +import org.apache.accumulo.core.data.PartialKey; +import org.apache.accumulo.core.data.Range; +import org.apache.accumulo.core.data.Value; +import org.apache.accumulo.core.master.state.tables.TableState; +import org.apache.accumulo.core.security.Authorizations; +import org.apache.accumulo.core.security.CredentialHelper; +import org.apache.accumulo.core.security.thrift.TCredentials; +import org.apache.accumulo.core.util.Pair; +import org.apache.accumulo.core.util.UtilWaitThread; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapred.InputFormat; +import org.apache.hadoop.mapred.InputSplit;
[1/2] git commit: ACCUMULO-1960 Fall back onto sudo if the user doesn't run it as root and isn't running processes as their user for agitation.
Updated Branches:
refs/heads/1.5.1-SNAPSHOT cccdb8cb4 - 57af8138a
ACCUMULO-1960 Fall back onto sudo if the user doesn't run it as root and isn't
running processes as their user for agitation.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/35ff0e50
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/35ff0e50
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/35ff0e50
Branch: refs/heads/1.5.1-SNAPSHOT
Commit: 35ff0e50ad7d8678e66d193cc53be40f6a72956f
Parents: cccdb8c
Author: Josh Elser els...@apache.org
Authored: Tue Dec 3 21:17:03 2013 -0500
Committer: Josh Elser els...@apache.org
Committed: Tue Dec 3 21:17:03 2013 -0500
--
test/system/continuous/agitator.pl | 71
.../system/continuous/continuous-env.sh.example | 3 +-
test/system/continuous/start-agitator.sh| 14 +++-
test/system/continuous/stop-agitator.sh | 8 ++-
4 files changed, 77 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/35ff0e50/test/system/continuous/agitator.pl
--
diff --git a/test/system/continuous/agitator.pl
b/test/system/continuous/agitator.pl
index f1abcd5..faeb231 100755
--- a/test/system/continuous/agitator.pl
+++ b/test/system/continuous/agitator.pl
@@ -19,21 +19,22 @@
use POSIX qw(strftime);
use Cwd qw();
-if(scalar(@ARGV) != 4 scalar(@ARGV) != 2){
- print Usage : agitator.pl min sleep before kill in minutes[:max sleep
before kill in minutes] min sleep before tup in minutes[:max sleep before tup
in minutes] [min kill max kill]\n;
+if(scalar(@ARGV) != 6 scalar(@ARGV) != 4){
+ print Usage : agitator.pl min sleep before kill in minutes[:max sleep
before kill in minutes] min sleep before tup in minutes[:max sleep before tup
in minutes] hdfs_user accumulo_user [min kill max kill]\n;
exit(1);
}
-$ACCUMULO_USER='accumulo';
-$HDFS_USER='hdfs';
+$myself=`whoami`;
+chomp($myself);
+$am_root=($myself eq 'root');
$cwd=Cwd::cwd();
$ACCUMULO_HOME=$cwd . '/../../..';
$HADOOP_PREFIX=$ENV{HADOOP_PREFIX};
-print STDERR Current directory: $cwd\n;
-print STDERR ACCUMULO_HOME=$ACCUMULO_HOME\n;
-print STDERR HADOOP_PREFIX=$HADOOP_PREFIX\n;
+print Current directory: $cwd\n;
+print ACCUMULO_HOME=$ACCUMULO_HOME\n;
+print HADOOP_PREFIX=$HADOOP_PREFIX\n;
@sleeprange1 = split(/:/, $ARGV[0]);
$sleep1 = $sleeprange1[0];
@@ -67,9 +68,15 @@ if(defined $ENV{'ACCUMULO_CONF_DIR'}){
$ACCUMULO_CONF_DIR = $ACCUMULO_HOME . '/conf';
}
-if(scalar(@ARGV) == 4){
- $minKill = $ARGV[2];
- $maxKill = $ARGV[3];
+$HDFS_USER=$ARGV[2];
+$ACCUMULO_USER=$ARGV[3];
+
+$am_hdfs_user=($HDFS_USER eq $myself);
+$am_accumulo_user=($ACCUMULO_USER eq $myself);
+
+if(scalar(@ARGV) == 6){
+ $minKill = $ARGV[4];
+ $maxKill = $ARGV[5];
}else{
$minKill = 1;
$maxKill = 1;
@@ -133,11 +140,29 @@ while(1){
print STDERR $t Killing $server $kill_tserver $kill_datanode\n;
if ($kill_tserver) {
- system(su -c '$ACCUMULO_HOME/bin/stop-server.sh $server
\accumulo-start.jar\ tserver KILL' - $ACCUMULO_USER);
+ if ($am_root) {
+# We're root, switch to the Accumulo user and try to stop gracefully
+system(su -c '$ACCUMULO_HOME/bin/stop-server.sh $server
\accumulo-start.jar\ tserver KILL' - $ACCUMULO_USER);
+ } elsif ($am_accumulo_user) {
+# We're the accumulo user, just run the commandj
+system($ACCUMULO_HOME/bin/stop-server.sh $server 'accumulo-start.jar'
tserver KILL);
+ } else {
+# We're not the accumulo user, try to use sudo
+system(sudo -u $ACCUMULO_USER $ACCUMULO_HOME/bin/stop-server.sh
$server accumulo-start.jar tserver KILL);
+ }
}
if ($kill_datanode) {
- system(ssh $server pkill -9 -f [p]roc_datanode);
+ if ($am_root) {
+# We're root, switch to HDFS to ssh and kill the process
+system(su -c 'ssh $server pkill -9 -f [p]roc_datanode' - $HDFS_USER);
+ } elsif ($am_hdfs_user) {
+# We're the HDFS user, just kill the process
+system(ssh $server \pkill -9 -f '[p]roc_datanode'\);
+ } else {
+# We're not the hdfs user, try to use sudo
+system(ssh $server 'sudo -u $HDFS_USER pkill -9 -f
\'[p]roc_datanode\'');
+ }
}
}
@@ -145,11 +170,29 @@ while(1){
sleep($nextsleep2 * 60);
$t = strftime %Y%m%d %H:%M:%S, localtime;
print STDERR $t Running tup\n;
- system(su -c $ACCUMULO_HOME/bin/tup.sh - $ACCUMULO_USER);
+ if ($am_root) {
+# Running as root, su to the accumulo user
+system(su -c $ACCUMULO_HOME/bin/tup.sh - $ACCUMULO_USER);
+ } elsif ($am_accumulo_user) {
+# restart the as them as the accumulo user
+system($ACCUMULO_HOME/bin/tup.sh);
+ } else {
+# Not the accumulo
[2/2] git commit: ACCUMULO-1960 Use $ACCUMULO_CONF_DIR/gc to kill the gc, not $ACCUMULO_CONF_DIR/masters
ACCUMULO-1960 Use $ACCUMULO_CONF_DIR/gc to kill the gc, not
$ACCUMULO_CONF_DIR/masters
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/57af8138
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/57af8138
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/57af8138
Branch: refs/heads/1.5.1-SNAPSHOT
Commit: 57af8138a88c8753693de3a9ce9ba24dbf6a6be6
Parents: 35ff0e5
Author: Josh Elser els...@apache.org
Authored: Tue Dec 3 21:18:29 2013 -0500
Committer: Josh Elser els...@apache.org
Committed: Tue Dec 3 21:18:29 2013 -0500
--
test/system/continuous/magitator.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/57af8138/test/system/continuous/magitator.pl
--
diff --git a/test/system/continuous/magitator.pl
b/test/system/continuous/magitator.pl
index 921c4d1..e26d143 100755
--- a/test/system/continuous/magitator.pl
+++ b/test/system/continuous/magitator.pl
@@ -60,7 +60,7 @@ while(1){
$cmd = pssh -h $ACCUMULO_CONF_DIR/masters \pkill -f '[
]org.apache.accumulo.start.*master'\ /dev/null;
print $t $cmd\n;
system($cmd);
- $cmd = pssh -h $ACCUMULO_CONF_DIR/masters \pkill -f '[
]org.apache.accumulo.start.*gc'\ /dev/null;
+ $cmd = pssh -h $ACCUMULO_CONF_DIR/gc \pkill -f '[
]org.apache.accumulo.start.*gc'\ /dev/null;
print $t $cmd\n;
system($cmd);
}
git commit: ACCUMULO-1960 Update README for continuous ingest for agitator improvements.
Updated Branches: refs/heads/1.5.1-SNAPSHOT 57af8138a - 0817cdc73 ACCUMULO-1960 Update README for continuous ingest for agitator improvements. Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/0817cdc7 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/0817cdc7 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/0817cdc7 Branch: refs/heads/1.5.1-SNAPSHOT Commit: 0817cdc73d3dcfd5cfef53b57476a89f42f25490 Parents: 57af813 Author: Josh Elser els...@apache.org Authored: Tue Dec 3 21:53:32 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 21:53:32 2013 -0500 -- test/system/continuous/README | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/0817cdc7/test/system/continuous/README -- diff --git a/test/system/continuous/README b/test/system/continuous/README index 62af7e3..713c18f 100644 --- a/test/system/continuous/README +++ b/test/system/continuous/README @@ -35,7 +35,13 @@ service that continually collect statistics about accumulo and HDFS. stop-stats.sh Optionally, start the agitator to periodically kill the tabletserver and/or datanode -process(es) on random nodes. This must be run as root. +process(es) on random nodes. You can run this script as root and it will properly start +processes as the user you configured in continuous-env.sh (HDFS_USER for the Datanode and +ACCUMULO_USER for Accumulo processes). If you run it as yourself and the HDFS_USER and +ACCUMULO_USER values are the same as your user, the agitator will not change users. In +the case where you run the agitator as a non-privileged user which isn't the same as HDFS_USER +nor ACCUMULO_USER, the agitator will attempt to `sudo` to these users which relies on correct +configuration of sudo. start-agitator.sh stop-agitator.sh
[5/5] git commit: Merge branch '1.6.0-SNAPSHOT'
Merge branch '1.6.0-SNAPSHOT' Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/5b84dc2a Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/5b84dc2a Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/5b84dc2a Branch: refs/heads/master Commit: 5b84dc2a1d67299f3f97236e7476d404c6c4e988 Parents: 5a8a28a 5312aea Author: Josh Elser els...@apache.org Authored: Tue Dec 3 22:01:55 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 22:01:55 2013 -0500 -- test/system/continuous/README | 8 ++- test/system/continuous/agitator.pl | 71 .../system/continuous/continuous-env.sh.example | 3 +- test/system/continuous/magitator.pl | 2 +- test/system/continuous/start-agitator.sh| 14 +++- test/system/continuous/stop-agitator.sh | 8 ++- 6 files changed, 85 insertions(+), 21 deletions(-) --
[2/5] git commit: ACCUMULO-1960 Use $ACCUMULO_CONF_DIR/gc to kill the gc, not $ACCUMULO_CONF_DIR/masters
ACCUMULO-1960 Use $ACCUMULO_CONF_DIR/gc to kill the gc, not
$ACCUMULO_CONF_DIR/masters
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/57af8138
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/57af8138
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/57af8138
Branch: refs/heads/master
Commit: 57af8138a88c8753693de3a9ce9ba24dbf6a6be6
Parents: 35ff0e5
Author: Josh Elser els...@apache.org
Authored: Tue Dec 3 21:18:29 2013 -0500
Committer: Josh Elser els...@apache.org
Committed: Tue Dec 3 21:18:29 2013 -0500
--
test/system/continuous/magitator.pl | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/57af8138/test/system/continuous/magitator.pl
--
diff --git a/test/system/continuous/magitator.pl
b/test/system/continuous/magitator.pl
index 921c4d1..e26d143 100755
--- a/test/system/continuous/magitator.pl
+++ b/test/system/continuous/magitator.pl
@@ -60,7 +60,7 @@ while(1){
$cmd = pssh -h $ACCUMULO_CONF_DIR/masters \pkill -f '[
]org.apache.accumulo.start.*master'\ /dev/null;
print $t $cmd\n;
system($cmd);
- $cmd = pssh -h $ACCUMULO_CONF_DIR/masters \pkill -f '[
]org.apache.accumulo.start.*gc'\ /dev/null;
+ $cmd = pssh -h $ACCUMULO_CONF_DIR/gc \pkill -f '[
]org.apache.accumulo.start.*gc'\ /dev/null;
print $t $cmd\n;
system($cmd);
}
[1/4] git commit: ACCUMULO-1960 Fall back onto sudo if the user doesn't run it as root and isn't running processes as their user for agitation.
Updated Branches:
refs/heads/1.6.0-SNAPSHOT 8a34937aa - 5312aea50
ACCUMULO-1960 Fall back onto sudo if the user doesn't run it as root and isn't
running processes as their user for agitation.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/35ff0e50
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/35ff0e50
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/35ff0e50
Branch: refs/heads/1.6.0-SNAPSHOT
Commit: 35ff0e50ad7d8678e66d193cc53be40f6a72956f
Parents: cccdb8c
Author: Josh Elser els...@apache.org
Authored: Tue Dec 3 21:17:03 2013 -0500
Committer: Josh Elser els...@apache.org
Committed: Tue Dec 3 21:17:03 2013 -0500
--
test/system/continuous/agitator.pl | 71
.../system/continuous/continuous-env.sh.example | 3 +-
test/system/continuous/start-agitator.sh| 14 +++-
test/system/continuous/stop-agitator.sh | 8 ++-
4 files changed, 77 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/35ff0e50/test/system/continuous/agitator.pl
--
diff --git a/test/system/continuous/agitator.pl
b/test/system/continuous/agitator.pl
index f1abcd5..faeb231 100755
--- a/test/system/continuous/agitator.pl
+++ b/test/system/continuous/agitator.pl
@@ -19,21 +19,22 @@
use POSIX qw(strftime);
use Cwd qw();
-if(scalar(@ARGV) != 4 scalar(@ARGV) != 2){
- print Usage : agitator.pl min sleep before kill in minutes[:max sleep
before kill in minutes] min sleep before tup in minutes[:max sleep before tup
in minutes] [min kill max kill]\n;
+if(scalar(@ARGV) != 6 scalar(@ARGV) != 4){
+ print Usage : agitator.pl min sleep before kill in minutes[:max sleep
before kill in minutes] min sleep before tup in minutes[:max sleep before tup
in minutes] hdfs_user accumulo_user [min kill max kill]\n;
exit(1);
}
-$ACCUMULO_USER='accumulo';
-$HDFS_USER='hdfs';
+$myself=`whoami`;
+chomp($myself);
+$am_root=($myself eq 'root');
$cwd=Cwd::cwd();
$ACCUMULO_HOME=$cwd . '/../../..';
$HADOOP_PREFIX=$ENV{HADOOP_PREFIX};
-print STDERR Current directory: $cwd\n;
-print STDERR ACCUMULO_HOME=$ACCUMULO_HOME\n;
-print STDERR HADOOP_PREFIX=$HADOOP_PREFIX\n;
+print Current directory: $cwd\n;
+print ACCUMULO_HOME=$ACCUMULO_HOME\n;
+print HADOOP_PREFIX=$HADOOP_PREFIX\n;
@sleeprange1 = split(/:/, $ARGV[0]);
$sleep1 = $sleeprange1[0];
@@ -67,9 +68,15 @@ if(defined $ENV{'ACCUMULO_CONF_DIR'}){
$ACCUMULO_CONF_DIR = $ACCUMULO_HOME . '/conf';
}
-if(scalar(@ARGV) == 4){
- $minKill = $ARGV[2];
- $maxKill = $ARGV[3];
+$HDFS_USER=$ARGV[2];
+$ACCUMULO_USER=$ARGV[3];
+
+$am_hdfs_user=($HDFS_USER eq $myself);
+$am_accumulo_user=($ACCUMULO_USER eq $myself);
+
+if(scalar(@ARGV) == 6){
+ $minKill = $ARGV[4];
+ $maxKill = $ARGV[5];
}else{
$minKill = 1;
$maxKill = 1;
@@ -133,11 +140,29 @@ while(1){
print STDERR $t Killing $server $kill_tserver $kill_datanode\n;
if ($kill_tserver) {
- system(su -c '$ACCUMULO_HOME/bin/stop-server.sh $server
\accumulo-start.jar\ tserver KILL' - $ACCUMULO_USER);
+ if ($am_root) {
+# We're root, switch to the Accumulo user and try to stop gracefully
+system(su -c '$ACCUMULO_HOME/bin/stop-server.sh $server
\accumulo-start.jar\ tserver KILL' - $ACCUMULO_USER);
+ } elsif ($am_accumulo_user) {
+# We're the accumulo user, just run the commandj
+system($ACCUMULO_HOME/bin/stop-server.sh $server 'accumulo-start.jar'
tserver KILL);
+ } else {
+# We're not the accumulo user, try to use sudo
+system(sudo -u $ACCUMULO_USER $ACCUMULO_HOME/bin/stop-server.sh
$server accumulo-start.jar tserver KILL);
+ }
}
if ($kill_datanode) {
- system(ssh $server pkill -9 -f [p]roc_datanode);
+ if ($am_root) {
+# We're root, switch to HDFS to ssh and kill the process
+system(su -c 'ssh $server pkill -9 -f [p]roc_datanode' - $HDFS_USER);
+ } elsif ($am_hdfs_user) {
+# We're the HDFS user, just kill the process
+system(ssh $server \pkill -9 -f '[p]roc_datanode'\);
+ } else {
+# We're not the hdfs user, try to use sudo
+system(ssh $server 'sudo -u $HDFS_USER pkill -9 -f
\'[p]roc_datanode\'');
+ }
}
}
@@ -145,11 +170,29 @@ while(1){
sleep($nextsleep2 * 60);
$t = strftime %Y%m%d %H:%M:%S, localtime;
print STDERR $t Running tup\n;
- system(su -c $ACCUMULO_HOME/bin/tup.sh - $ACCUMULO_USER);
+ if ($am_root) {
+# Running as root, su to the accumulo user
+system(su -c $ACCUMULO_HOME/bin/tup.sh - $ACCUMULO_USER);
+ } elsif ($am_accumulo_user) {
+# restart the as them as the accumulo user
+system($ACCUMULO_HOME/bin/tup.sh);
+ } else {
+# Not the accumulo
[3/4] git commit: ACCUMULO-1960 Update README for continuous ingest for agitator improvements.
ACCUMULO-1960 Update README for continuous ingest for agitator improvements. Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/0817cdc7 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/0817cdc7 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/0817cdc7 Branch: refs/heads/1.6.0-SNAPSHOT Commit: 0817cdc73d3dcfd5cfef53b57476a89f42f25490 Parents: 57af813 Author: Josh Elser els...@apache.org Authored: Tue Dec 3 21:53:32 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 21:53:32 2013 -0500 -- test/system/continuous/README | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/0817cdc7/test/system/continuous/README -- diff --git a/test/system/continuous/README b/test/system/continuous/README index 62af7e3..713c18f 100644 --- a/test/system/continuous/README +++ b/test/system/continuous/README @@ -35,7 +35,13 @@ service that continually collect statistics about accumulo and HDFS. stop-stats.sh Optionally, start the agitator to periodically kill the tabletserver and/or datanode -process(es) on random nodes. This must be run as root. +process(es) on random nodes. You can run this script as root and it will properly start +processes as the user you configured in continuous-env.sh (HDFS_USER for the Datanode and +ACCUMULO_USER for Accumulo processes). If you run it as yourself and the HDFS_USER and +ACCUMULO_USER values are the same as your user, the agitator will not change users. In +the case where you run the agitator as a non-privileged user which isn't the same as HDFS_USER +nor ACCUMULO_USER, the agitator will attempt to `sudo` to these users which relies on correct +configuration of sudo. start-agitator.sh stop-agitator.sh
[1/5] git commit: ACCUMULO-1960 Fall back onto sudo if the user doesn't run it as root and isn't running processes as their user for agitation.
Updated Branches:
refs/heads/master 5a8a28af9 - 5b84dc2a1
ACCUMULO-1960 Fall back onto sudo if the user doesn't run it as root and isn't
running processes as their user for agitation.
Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo
Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/35ff0e50
Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/35ff0e50
Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/35ff0e50
Branch: refs/heads/master
Commit: 35ff0e50ad7d8678e66d193cc53be40f6a72956f
Parents: cccdb8c
Author: Josh Elser els...@apache.org
Authored: Tue Dec 3 21:17:03 2013 -0500
Committer: Josh Elser els...@apache.org
Committed: Tue Dec 3 21:17:03 2013 -0500
--
test/system/continuous/agitator.pl | 71
.../system/continuous/continuous-env.sh.example | 3 +-
test/system/continuous/start-agitator.sh| 14 +++-
test/system/continuous/stop-agitator.sh | 8 ++-
4 files changed, 77 insertions(+), 19 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/accumulo/blob/35ff0e50/test/system/continuous/agitator.pl
--
diff --git a/test/system/continuous/agitator.pl
b/test/system/continuous/agitator.pl
index f1abcd5..faeb231 100755
--- a/test/system/continuous/agitator.pl
+++ b/test/system/continuous/agitator.pl
@@ -19,21 +19,22 @@
use POSIX qw(strftime);
use Cwd qw();
-if(scalar(@ARGV) != 4 scalar(@ARGV) != 2){
- print Usage : agitator.pl min sleep before kill in minutes[:max sleep
before kill in minutes] min sleep before tup in minutes[:max sleep before tup
in minutes] [min kill max kill]\n;
+if(scalar(@ARGV) != 6 scalar(@ARGV) != 4){
+ print Usage : agitator.pl min sleep before kill in minutes[:max sleep
before kill in minutes] min sleep before tup in minutes[:max sleep before tup
in minutes] hdfs_user accumulo_user [min kill max kill]\n;
exit(1);
}
-$ACCUMULO_USER='accumulo';
-$HDFS_USER='hdfs';
+$myself=`whoami`;
+chomp($myself);
+$am_root=($myself eq 'root');
$cwd=Cwd::cwd();
$ACCUMULO_HOME=$cwd . '/../../..';
$HADOOP_PREFIX=$ENV{HADOOP_PREFIX};
-print STDERR Current directory: $cwd\n;
-print STDERR ACCUMULO_HOME=$ACCUMULO_HOME\n;
-print STDERR HADOOP_PREFIX=$HADOOP_PREFIX\n;
+print Current directory: $cwd\n;
+print ACCUMULO_HOME=$ACCUMULO_HOME\n;
+print HADOOP_PREFIX=$HADOOP_PREFIX\n;
@sleeprange1 = split(/:/, $ARGV[0]);
$sleep1 = $sleeprange1[0];
@@ -67,9 +68,15 @@ if(defined $ENV{'ACCUMULO_CONF_DIR'}){
$ACCUMULO_CONF_DIR = $ACCUMULO_HOME . '/conf';
}
-if(scalar(@ARGV) == 4){
- $minKill = $ARGV[2];
- $maxKill = $ARGV[3];
+$HDFS_USER=$ARGV[2];
+$ACCUMULO_USER=$ARGV[3];
+
+$am_hdfs_user=($HDFS_USER eq $myself);
+$am_accumulo_user=($ACCUMULO_USER eq $myself);
+
+if(scalar(@ARGV) == 6){
+ $minKill = $ARGV[4];
+ $maxKill = $ARGV[5];
}else{
$minKill = 1;
$maxKill = 1;
@@ -133,11 +140,29 @@ while(1){
print STDERR $t Killing $server $kill_tserver $kill_datanode\n;
if ($kill_tserver) {
- system(su -c '$ACCUMULO_HOME/bin/stop-server.sh $server
\accumulo-start.jar\ tserver KILL' - $ACCUMULO_USER);
+ if ($am_root) {
+# We're root, switch to the Accumulo user and try to stop gracefully
+system(su -c '$ACCUMULO_HOME/bin/stop-server.sh $server
\accumulo-start.jar\ tserver KILL' - $ACCUMULO_USER);
+ } elsif ($am_accumulo_user) {
+# We're the accumulo user, just run the commandj
+system($ACCUMULO_HOME/bin/stop-server.sh $server 'accumulo-start.jar'
tserver KILL);
+ } else {
+# We're not the accumulo user, try to use sudo
+system(sudo -u $ACCUMULO_USER $ACCUMULO_HOME/bin/stop-server.sh
$server accumulo-start.jar tserver KILL);
+ }
}
if ($kill_datanode) {
- system(ssh $server pkill -9 -f [p]roc_datanode);
+ if ($am_root) {
+# We're root, switch to HDFS to ssh and kill the process
+system(su -c 'ssh $server pkill -9 -f [p]roc_datanode' - $HDFS_USER);
+ } elsif ($am_hdfs_user) {
+# We're the HDFS user, just kill the process
+system(ssh $server \pkill -9 -f '[p]roc_datanode'\);
+ } else {
+# We're not the hdfs user, try to use sudo
+system(ssh $server 'sudo -u $HDFS_USER pkill -9 -f
\'[p]roc_datanode\'');
+ }
}
}
@@ -145,11 +170,29 @@ while(1){
sleep($nextsleep2 * 60);
$t = strftime %Y%m%d %H:%M:%S, localtime;
print STDERR $t Running tup\n;
- system(su -c $ACCUMULO_HOME/bin/tup.sh - $ACCUMULO_USER);
+ if ($am_root) {
+# Running as root, su to the accumulo user
+system(su -c $ACCUMULO_HOME/bin/tup.sh - $ACCUMULO_USER);
+ } elsif ($am_accumulo_user) {
+# restart the as them as the accumulo user
+system($ACCUMULO_HOME/bin/tup.sh);
+ } else {
+# Not the accumulo user, try to
[3/5] git commit: ACCUMULO-1960 Update README for continuous ingest for agitator improvements.
ACCUMULO-1960 Update README for continuous ingest for agitator improvements. Project: http://git-wip-us.apache.org/repos/asf/accumulo/repo Commit: http://git-wip-us.apache.org/repos/asf/accumulo/commit/0817cdc7 Tree: http://git-wip-us.apache.org/repos/asf/accumulo/tree/0817cdc7 Diff: http://git-wip-us.apache.org/repos/asf/accumulo/diff/0817cdc7 Branch: refs/heads/master Commit: 0817cdc73d3dcfd5cfef53b57476a89f42f25490 Parents: 57af813 Author: Josh Elser els...@apache.org Authored: Tue Dec 3 21:53:32 2013 -0500 Committer: Josh Elser els...@apache.org Committed: Tue Dec 3 21:53:32 2013 -0500 -- test/system/continuous/README | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/accumulo/blob/0817cdc7/test/system/continuous/README -- diff --git a/test/system/continuous/README b/test/system/continuous/README index 62af7e3..713c18f 100644 --- a/test/system/continuous/README +++ b/test/system/continuous/README @@ -35,7 +35,13 @@ service that continually collect statistics about accumulo and HDFS. stop-stats.sh Optionally, start the agitator to periodically kill the tabletserver and/or datanode -process(es) on random nodes. This must be run as root. +process(es) on random nodes. You can run this script as root and it will properly start +processes as the user you configured in continuous-env.sh (HDFS_USER for the Datanode and +ACCUMULO_USER for Accumulo processes). If you run it as yourself and the HDFS_USER and +ACCUMULO_USER values are the same as your user, the agitator will not change users. In +the case where you run the agitator as a non-privileged user which isn't the same as HDFS_USER +nor ACCUMULO_USER, the agitator will attempt to `sudo` to these users which relies on correct +configuration of sudo. start-agitator.sh stop-agitator.sh
