Author: kturner
Date: Mon Jun 18 18:29:00 2012
New Revision: 1351452
URL: http://svn.apache.org/viewvc?rev=1351452&view=rev
Log:
ACCUMULO-637 ACCUMULO-614 merge changes from 1.4 branch
Modified:
accumulo/branches/1.4.1/ (props changed)
accumulo/branches/1.4.1/CHANGES
accumulo/branches/1.4.1/src/ (props changed)
accumulo/branches/1.4.1/src/core/ (props changed)
accumulo/branches/1.4.1/src/server/ (props changed)
accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java
accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example
accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh
accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh
accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh
accumulo/branches/1.4.1/test/system/continuous/start-stats.sh
accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh
Propchange: accumulo/branches/1.4.1/
------------------------------------------------------------------------------
Merged /accumulo/trunk:r1350779
Merged
/accumulo/branches/1.4:r1346082-1346088,1346090-1346833,1346835-1349971,1349973-1351449
Modified: accumulo/branches/1.4.1/CHANGES
URL:
http://svn.apache.org/viewvc/accumulo/branches/1.4.1/CHANGES?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/CHANGES (original)
+++ accumulo/branches/1.4.1/CHANGES Mon Jun 18 18:29:00 2012
@@ -1,3 +1,62 @@
+Release Notes - Apache Accumulo - Version 1.4.1
+
+** Bug
+ * [ACCUMULO-355] - functional tests should run under a non-hadoop user
+ * [ACCUMULO-398] - Table tablets not evenly spread.
+ * [ACCUMULO-428] - loggers are not closed when high hold times are detected
+ * [ACCUMULO-449] - Failed log copy is not restarted
+ * [ACCUMULO-468] - low-memory warning message is incorrect
+ * [ACCUMULO-487] - Batch Scanner can get stuck when external thread closes
scanner
+ * [ACCUMULO-488] - InputFormats' RecordReaders should call Context.progress
+ * [ACCUMULO-496] - Cloudtrace.thrift needs to have it's namespace updated
to the new package
+ * [ACCUMULO-499] - Garbage collector should call Tracer.offNoFlush instead
of Tracer.off
+ * [ACCUMULO-505] - "% of Used DFS" is overflowing on a large cluster
+ * [ACCUMULO-509] - default walog copy/sort uses replication of 1
+ * [ACCUMULO-512] - DEFAULT_MAX_LATENCY in AccumuloOutputFormat wrong units
+ * [ACCUMULO-516] - Column family search with sparse files is painfully long
+ * [ACCUMULO-520] - assigned tablets are not considered offline
+ * [ACCUMULO-521] - examples functional test is using the wrong class name
+ * [ACCUMULO-533] - make system iterators thread-safer
+ * [ACCUMULO-542] - with a large root tablet, tablet servers will get stuck
loading other metadata tablets
+ * [ACCUMULO-561] - Monitor scan mb/s does not show batch scan activity
+ * [ACCUMULO-562] - Multi term grep in shell fails
+ * [ACCUMULO-566] - monitor should display zero tablet servers as red, even
if there's only one tablet server configured
+ * [ACCUMULO-570] - dirlist example README is a little inconsistent
+ * [ACCUMULO-585] - Complain loudly when env file is missing
+ * [ACCUMULO-591] - Need to clear tablet location cache before computing
input splits in input format
+ * [ACCUMULO-592] - We lost the security policy example
+ * [ACCUMULO-601] - client fails to resolve master hostname when not using
fully qualified domain name
+ * [ACCUMULO-615] - accumulo runs fine on Hadoop 1.0
+ * [ACCUMULO-633] - FirstEntryInRowIterator is broken and has no test
+
+** Improvement
+ * [ACCUMULO-337] - Make logging an example file
+ * [ACCUMULO-364] - Security Policy file should be an example
+ * [ACCUMULO-399] - need a randomwalk node that checks tablet balance
+ * [ACCUMULO-459] - Accumulo init should fail fast
+ * [ACCUMULO-471] - allow wikisearch ingest to run on uncompressed input
+ * [ACCUMULO-480] - config.html should be more clear with regards to "needs
restart"
+ * [ACCUMULO-489] - Input Format puts Base64 encoded passwords in
Configuration, which is world readable
+ * [ACCUMULO-494] - Document relationship between max walogs and max
tserver memory
+ * [ACCUMULO-502] - Document what replaces the deprecated
PerColumnIteratorConfig class.
+ * [ACCUMULO-503] - start-here does unecessary SSHs
+ * [ACCUMULO-510] - Rename scanner threads to identify table being scanned
+ * [ACCUMULO-550] - Colocate rfile index entries within file
+ * [ACCUMULO-554] - monitor should display red when time-since last report
is higher than a few minutes
+ * [ACCUMULO-573] - Fix grammar in shell commands
+ * [ACCUMULO-574] - Document reseek of iterators
+ * [ACCUMULO-637] - Make entries written configurable for continuous ingest
+
+** New Feature
+ * [ACCUMULO-49] - optionally monitor swappiness on every server
+ * [ACCUMULO-404] - Support running on-top of Kerberos-enabled HDFS
+ * [ACCUMULO-492] - Provide method for gathering system stats to API
+
+** Task
+ * [ACCUMULO-491] - Manual has UTF8 characters
+ * [ACCUMULO-546] - User manual inconsistencies
+
+
Release notes for Apache Accumulo 1.4.0
Major features added prior to Incubation
Propchange: accumulo/branches/1.4.1/src/
------------------------------------------------------------------------------
Merged /accumulo/trunk/src:r1350779
Merged
/accumulo/branches/1.4/src:r1346082-1346088,1346090-1346833,1346835-1349971,1349973-1351449
Merged /accumulo/trunk:r1350779
Propchange: accumulo/branches/1.4.1/src/core/
------------------------------------------------------------------------------
Merged /accumulo/trunk/src/core:r1350779
Merged /accumulo/trunk/core:r1350779
Merged
/accumulo/branches/1.4/src/core:r1346082-1346088,1346090-1346833,1346835-1349971,1349973-1351449
Propchange: accumulo/branches/1.4.1/src/server/
------------------------------------------------------------------------------
Merged /accumulo/trunk/src/server:r1350779
Merged
/accumulo/branches/1.4/src/server:r1346082-1346088,1346090-1346833,1346835-1349971,1349973-1351449
Merged /accumulo/trunk/server:r1350779
Modified:
accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java
URL:
http://svn.apache.org/viewvc/accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
---
accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java
(original)
+++
accumulo/branches/1.4.1/src/server/src/main/java/org/apache/accumulo/server/test/continuous/ContinuousIngest.java
Mon Jun 18 18:29:00 2012
@@ -32,6 +32,7 @@ import org.apache.accumulo.core.client.B
import org.apache.accumulo.core.client.Connector;
import org.apache.accumulo.core.client.Instance;
import org.apache.accumulo.core.client.MutationsRejectedException;
+import org.apache.accumulo.core.client.TableExistsException;
import org.apache.accumulo.core.client.ZooKeeperInstance;
import org.apache.accumulo.core.data.Mutation;
import org.apache.accumulo.core.data.Value;
@@ -67,11 +68,11 @@ public class ContinuousIngest {
args = processOptions(args);
- if (args.length != 13) {
+ if (args.length != 14) {
throw new IllegalArgumentException(
"usage : "
+ ContinuousIngest.class.getName()
- + " [--debug <debug log>] <instance name> <zookeepers> <user>
<pass> <table> <min> <max> <max colf> <max colq> <max mem> <max latency> <max
threads> <enable checksum>");
+ + " [--debug <debug log>] <instance name> <zookeepers> <user>
<pass> <table> <num> <min> <max> <max colf> <max colq> <max mem> <max latency>
<max threads> <enable checksum>");
}
if (debugLog != null) {
@@ -89,16 +90,17 @@ public class ContinuousIngest {
String table = args[4];
- long min = Long.parseLong(args[5]);
- long max = Long.parseLong(args[6]);
- short maxColF = Short.parseShort(args[7]);
- short maxColQ = Short.parseShort(args[8]);
-
- long maxMemory = Long.parseLong(args[9]);
- long maxLatency = Integer.parseInt(args[10]);
- int maxWriteThreads = Integer.parseInt(args[11]);
+ long num = Long.parseLong(args[5]);
+ long min = Long.parseLong(args[6]);
+ long max = Long.parseLong(args[7]);
+ short maxColF = Short.parseShort(args[8]);
+ short maxColQ = Short.parseShort(args[9]);
+
+ long maxMemory = Long.parseLong(args[10]);
+ long maxLatency = Integer.parseInt(args[11]);
+ int maxWriteThreads = Integer.parseInt(args[12]);
- boolean checksum = Boolean.parseBoolean(args[12]);
+ boolean checksum = Boolean.parseBoolean(args[13]);
if (min < 0 || max < 0 || max <= min) {
throw new IllegalArgumentException("bad min and max");
@@ -109,6 +111,11 @@ public class ContinuousIngest {
String path = ZooUtil.getRoot(instance) + Constants.ZTRACERS;
Tracer.getInstance().addReceiver(new ZooSpanClient(zooKeepers, path,
localhost, "cingest", 1000));
+ if (!conn.tableOperations().exists(table))
+ try {
+ conn.tableOperations().create(table);
+ } catch (TableExistsException tee) {}
+
BatchWriter bw = conn.createBatchWriter(table, maxMemory, maxLatency,
maxWriteThreads);
bw = Trace.wrapAll(bw, new CountSampler(1024));
@@ -133,7 +140,7 @@ public class ContinuousIngest {
long lastFlushTime = System.currentTimeMillis();
- while (true) {
+ out: while (true) {
// generate first set of nodes
for (int index = 0; index < flushInterval; index++) {
long rowLong = genLong(min, max, r);
@@ -152,6 +159,8 @@ public class ContinuousIngest {
}
lastFlushTime = flush(bw, count, flushInterval, lastFlushTime);
+ if (count >= num)
+ break out;
// generate subsequent sets of nodes that link to previous set of nodes
for (int depth = 1; depth < maxDepth; depth++) {
@@ -165,6 +174,8 @@ public class ContinuousIngest {
}
lastFlushTime = flush(bw, count, flushInterval, lastFlushTime);
+ if (count >= num)
+ break out;
}
// create one big linked list, this makes all of the first inserts
@@ -175,7 +186,11 @@ public class ContinuousIngest {
bw.addMutation(m);
}
lastFlushTime = flush(bw, count, flushInterval, lastFlushTime);
+ if (count >= num)
+ break out;
}
+
+ bw.close();
}
private static long flush(BatchWriter bw, long count, final int
flushInterval, long lastFlushTime) throws MutationsRejectedException {
Modified:
accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example
URL:
http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example
(original)
+++ accumulo/branches/1.4.1/test/system/continuous/continuous-env.sh.example
Mon Jun 18 18:29:00 2012
@@ -32,6 +32,9 @@ DEBUG_WALKER=off
DEBUG_BATCH_WALKER=off
DEBUG_SCANNER=off
+#the number of entries each client should write
+NUM=9223372036854775807
+
#the minimum random row to generate
MIN=0
Modified: accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh
URL:
http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh
(original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-batchwalkers.sh Mon
Jun 18 18:29:00 2012
@@ -24,5 +24,5 @@ if [ "$DEBUG_BATCH_WALKER" = "on" ] ; th
DEBUG_OPT="--debug $CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.log";
fi
-pssh -h batch_walkers.txt "nohup $ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousBatchWalker $DEBUG_OPT
$INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $BATCH_WALKER_SLEEP
$BATCH_WALKER_BATCH_SIZE $BATCH_WALKER_THREADS >$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.err &" < /dev/null
+pssh -h batch_walkers.txt "mkdir -p $CONTINUOUS_LOG_DIR; nohup
$ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousBatchWalker $DEBUG_OPT
$INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $BATCH_WALKER_SLEEP
$BATCH_WALKER_BATCH_SIZE $BATCH_WALKER_THREADS >$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_batch_walk.err &" < /dev/null
Modified: accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh
URL:
http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-ingest.sh Mon Jun 18
18:29:00 2012
@@ -24,5 +24,10 @@ if [ "$DEBUG_INGEST" = "on" ] ; then
DEBUG_OPT="--debug $CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_ingest.log";
fi
-pssh -h ingesters.txt "nohup $ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousIngest $DEBUG_OPT
$INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $MAX_CF $MAX_CQ
$MAX_MEM $MAX_LATENCY $NUM_THREADS $CHECKSUM >$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_ingest.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_ingest.err &" < /dev/null
+#do this check so that script will work w/ 1.4.0 config file
+if [ -z $NUM ]; then
+ NUM=9223372036854775807
+fi
+
+pssh -h ingesters.txt "mkdir -p $CONTINUOUS_LOG_DIR; nohup
$ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousIngest $DEBUG_OPT
$INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $NUM $MIN $MAX $MAX_CF $MAX_CQ
$MAX_MEM $MAX_LATENCY $NUM_THREADS $CHECKSUM >$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_ingest.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_ingest.err &" < /dev/null
Modified: accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh
URL:
http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-scanners.sh Mon Jun 18
18:29:00 2012
@@ -24,5 +24,5 @@ if [ "$DEBUG_SCANNER" = "on" ] ; then
DEBUG_OPT="--debug $CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_scanner.log";
fi
-pssh -h scanners.txt "nohup $ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousScanner $DEBUG_SCANNERS
$INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $SCANNER_SLEEP_TIME
$SCANNER_ENTRIES >$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_scanner.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_scanner.err &" < /dev/null
+pssh -h scanners.txt "mkdir -p $CONTINUOUS_LOG_DIR; nohup
$ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousScanner $DEBUG_SCANNERS
$INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $SCANNER_SLEEP_TIME
$SCANNER_ENTRIES >$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_scanner.out 2>$CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_scanner.err &" < /dev/null
Modified: accumulo/branches/1.4.1/test/system/continuous/start-stats.sh
URL:
http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-stats.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-stats.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-stats.sh Mon Jun 18
18:29:00 2012
@@ -18,6 +18,8 @@
. continuous-env.sh
+mkdir -p $CONTINUOUS_LOG_DIR
+
CONFIG_OUT=$CONTINUOUS_LOG_DIR/`date +%Y%m%d%H%M%S`_`hostname`_config.out
cat $ACCUMULO_HOME/conf/accumulo-env.sh > $CONFIG_OUT
Modified: accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh
URL:
http://svn.apache.org/viewvc/accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh?rev=1351452&r1=1351451&r2=1351452&view=diff
==============================================================================
--- accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh (original)
+++ accumulo/branches/1.4.1/test/system/continuous/start-walkers.sh Mon Jun 18
18:29:00 2012
@@ -24,5 +24,5 @@ if [ "$DEBUG_WALKER" = "on" ] ; then
DEBUG_OPT="--debug $CONTINUOUS_LOG_DIR/\`date
+%Y%m%d%H%M%S\`_\`hostname\`_walk.log";
fi
-pssh -h walkers.txt "nohup $ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousWalk $DEBUG_OPT
$INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $SLEEP_TIME
>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_walk.out
2>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_walk.err &" <
/dev/null
+pssh -h walkers.txt "mkdir -p $CONTINUOUS_LOG_DIR; nohup
$ACCUMULO_HOME/bin/accumulo
org.apache.accumulo.server.test.continuous.ContinuousWalk $DEBUG_OPT
$INSTANCE_NAME $ZOO_KEEPERS $USER $PASS $TABLE $MIN $MAX $SLEEP_TIME
>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_walk.out
2>$CONTINUOUS_LOG_DIR/\`date +%Y%m%d%H%M%S\`_\`hostname\`_walk.err &" <
/dev/null
