[jira] [Assigned] (ACCUMULO-4569) Create separate mapreduce module

[Christopher Tubbs (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ACCUMULO-4569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christopher Tubbs reassigned ACCUMULO-4569:
---

Assignee: Christopher Tubbs

> Create separate mapreduce module
> 
>
> Key: ACCUMULO-4569
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4569
> Project: Accumulo
>  Issue Type: Task
>  Components: core, mapreduce
>Reporter: Christopher Tubbs
>Assignee: Christopher Tubbs
>Priority: Major
> Fix For: 2.0.0
>
>
> A separate MapReduce module was attempted in ACCUMULO-1880, but reverted in 
> ACCUMULO-3402 because of a concern about dependency changes across a minor 
> version number.
> For 2.0.0, this idea of creating a separate mapreduce module can be 
> revisited, as it will be a major version bump.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Accumulo-1.7 - Build # 404 - Fixed

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-1.7 (build #404)

Status: Fixed

Check console output at https://builds.apache.org/job/Accumulo-1.7/404/ to view 
the results.

[GitHub] keith-turner commented on a change in pull request #364: ACCUMULO-4778 Initial feedback for table name to id mapping cache

[GitBox]

keith-turner commented on a change in pull request #364: ACCUMULO-4778 Initial 
feedback for table name to id mapping cache
URL: https://github.com/apache/accumulo/pull/364#discussion_r164015119
 
 

 ##
 File path: 
core/src/main/java/org/apache/accumulo/core/client/impl/TableMap.java
 ##
 @@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.accumulo.core.client.impl;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.apache.accumulo.core.client.impl.Tables.qualified;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.accumulo.core.Constants;
+import org.apache.accumulo.core.client.Instance;
+import org.apache.accumulo.core.client.NamespaceNotFoundException;
+import org.apache.accumulo.core.zookeeper.ZooUtil;
+import org.apache.accumulo.fate.zookeeper.ZooCache;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.collect.ImmutableMap;
+
+/**
+ * Used for thread safe caching of immutable table ID maps. See ACCUMULO-4778.
+ */
+public class TableMap {
+  private static final Logger log = LoggerFactory.getLogger(TableMap.class);
+
+  private final Map tableNameToIdMap;
+  private final Map tableIdToNameMap;
+  private final Long creationTime;
 
 Review comment:
   this can be dropped now.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] keith-turner closed pull request #367: ACCUMULO-4779 made getting vfs config more efficient

[GitBox]

keith-turner closed pull request #367: ACCUMULO-4779 made getting vfs config 
more efficient
URL: https://github.com/apache/accumulo/pull/367
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java 
b/core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java
index 6828174b61..cb1ccb93f7 100644
--- a/core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java
+++ b/core/src/main/java/org/apache/accumulo/core/client/impl/ClientContext.java
@@ -176,6 +176,12 @@ public static AccumuloConfiguration 
convertClientConfig(final Configuration conf
 final AccumuloConfiguration defaults = DefaultConfiguration.getInstance();
 
 return new AccumuloConfiguration() {
+
+  @Override
+  protected String getArbitrarySystemPropertyImpl(String property) {
+return config.getString(property, null);
+  }
+
   @Override
   public String get(Property property) {
 final String key = property.getKey();
diff --git 
a/core/src/main/java/org/apache/accumulo/core/client/mock/MockConfiguration.java
 
b/core/src/main/java/org/apache/accumulo/core/client/mock/MockConfiguration.java
index 410105b5ac..f642770d2f 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/client/mock/MockConfiguration.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/client/mock/MockConfiguration.java
@@ -35,6 +35,11 @@ public void put(String k, String v) {
 map.put(k, v);
   }
 
+  @Override
+  protected String getArbitrarySystemPropertyImpl(String property) {
+return map.get(property);
+  }
+
   @Override
   public String get(Property property) {
 return map.get(property.getKey());
diff --git 
a/core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java 
b/core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java
index 25d4c60064..008418e185 100644
--- 
a/core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java
+++ 
b/core/src/main/java/org/apache/accumulo/core/conf/AccumuloConfiguration.java
@@ -33,6 +33,7 @@
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import com.google.common.base.Preconditions;
 import com.google.common.base.Predicate;
 import com.google.common.base.Predicates;
 
@@ -106,6 +107,32 @@ public boolean apply(String key) {
 
   private static final Logger log = 
LoggerFactory.getLogger(AccumuloConfiguration.class);
 
+  protected String getArbitrarySystemPropertyImpl(AccumuloConfiguration 
parent, String property) {
+return parent.getArbitrarySystemPropertyImpl(property);
+  }
+
+  /**
+   * This method is not called with sensitive or per table properties.
+   */
+  protected String getArbitrarySystemPropertyImpl(String property) {
+throw new UnsupportedOperationException();
+  }
+
+  /**
+   * This method was created because {@link #get(String)} is very slow. 
However this method does not properly handle everything that {@link 
#get(String)} does.
+   * For example it does not properly handle table or sensitive properties.
+   *
+   * 
+   * This method has a whitelist of prefixes it handles. To see the whitelist, 
check the implementation. When adding to the whitelist, ensure that all
+   * configurations can properly handle.
+   */
+  public String getArbitrarySystemProperty(Property prefix, String property) {
+Preconditions.checkArgument(prefix == 
Property.VFS_CONTEXT_CLASSPATH_PROPERTY);
+
+String key = prefix.getKey() + property;
+return getArbitrarySystemPropertyImpl(key);
+  }
+
   /**
* Gets a property value from this configuration.
*
diff --git 
a/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationCopy.java 
b/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationCopy.java
index 28b188fb1a..df4355760f 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationCopy.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/ConfigurationCopy.java
@@ -63,6 +63,11 @@ public String get(Property property) {
 return copy.get(property.getKey());
   }
 
+  @Override
+  protected String getArbitrarySystemPropertyImpl(String property) {
+return copy.get(property);
+  }
+
   @Override
   public void getProperties(Map props, Predicate 
filter) {
 for (Entry entry : copy.entrySet()) {
diff --git 
a/core/src/main/java/org/apache/accumulo/core/conf/DefaultConfiguration.java 
b/core/src/main/java/org/apache/accumulo/core/conf/DefaultConfiguration.java
index e1ff7e16bb..994d960e02 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/DefaultConfiguration.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/DefaultConfiguration.java
@@ -58,4 +58,9 @@ public void 

[GitHub] keith-turner commented on issue #367: ACCUMULO-4779 made getting vfs config more efficient

[GitBox]

keith-turner commented on issue #367: ACCUMULO-4779 made getting vfs config 
more efficient
URL: https://github.com/apache/accumulo/pull/367#issuecomment-360652824
 
 
   Merged 7bc630cdce4a0aa02be4873e2873ff1036a94e14


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] keith-turner commented on issue #365: ACCUMULO-4779 Avoid locks in ZooCache when data in cache

[GitBox]

keith-turner commented on issue #365: ACCUMULO-4779 Avoid locks in ZooCache 
when data in cache
URL: https://github.com/apache/accumulo/pull/365#issuecomment-360652676
 
 
   merged in cf9e754b045e3fac452df282bcf4ec97974038a0


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] keith-turner closed pull request #365: ACCUMULO-4779 Avoid locks in ZooCache when data in cache

[GitBox]

keith-turner closed pull request #365: ACCUMULO-4779 Avoid locks in ZooCache 
when data in cache
URL: https://github.com/apache/accumulo/pull/365
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooCache.java 
b/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooCache.java
index ba837f88c6..0c82d413f6 100644
--- a/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooCache.java
+++ b/fate/src/main/java/org/apache/accumulo/fate/zookeeper/ZooCache.java
@@ -28,6 +28,7 @@
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
+import java.util.Map;
 import java.util.concurrent.locks.Lock;
 import java.util.concurrent.locks.LockSupport;
 import java.util.concurrent.locks.ReadWriteLock;
@@ -43,6 +44,7 @@
 import org.slf4j.LoggerFactory;
 
 import com.google.common.annotations.VisibleForTesting;
+import com.google.common.collect.ImmutableList;
 
 /**
  * A cache for values stored in ZooKeeper. Values are kept up to date as they 
change.
@@ -63,6 +65,38 @@
 
   private final ZooReader zReader;
 
+  private static class ImmutableCacheCopies {
+final Map cache;
+final Map statCache;
+final Map childrenCache;
+
+ImmutableCacheCopies() {
+  cache = Collections.emptyMap();
+  statCache = Collections.emptyMap();
+  childrenCache = Collections.emptyMap();
+}
+
+ImmutableCacheCopies(Map cache, Map statCache, 
Map childrenCache) {
+  this.cache = Collections.unmodifiableMap(new HashMap<>(cache));
+  this.statCache = Collections.unmodifiableMap(new HashMap<>(statCache));
+  this.childrenCache = Collections.unmodifiableMap(new 
HashMap<>(childrenCache));
+}
+
+ImmutableCacheCopies(ImmutableCacheCopies prev, Map 
childrenCache) {
+  this.cache = prev.cache;
+  this.statCache = prev.statCache;
+  this.childrenCache = Collections.unmodifiableMap(new 
HashMap<>(childrenCache));
+}
+
+ImmutableCacheCopies(Map cache, Map statCache, 
ImmutableCacheCopies prev) {
+  this.cache = Collections.unmodifiableMap(new HashMap<>(cache));
+  this.statCache = Collections.unmodifiableMap(new HashMap<>(statCache));
+  this.childrenCache = prev.childrenCache;
+}
+  }
+
+  private volatile ImmutableCacheCopies immutableCache = new 
ImmutableCacheCopies();
+
   /**
* Returns a ZooKeeper session. Calls should be made within run of 
ZooRunnable after caches are checked. This will be performed at each retry of 
the run
* method. Calls to {@link #getZooKeeper()} should be made, ideally, after 
cache checks since other threads may have succeeded when updating the cache. 
Doing
@@ -223,19 +257,17 @@ public T retry() {
*  path of node
* @return children list, or null if node has no children or does not exist
*/
-  public synchronized List getChildren(final String zPath) {
+  public List getChildren(final String zPath) {
 
 ZooRunnable zr = new ZooRunnable() {
 
   @Override
   public List run() throws KeeperException, InterruptedException {
-try {
-  cacheReadLock.lock();
-  if (childrenCache.containsKey(zPath)) {
-return childrenCache.get(zPath);
-  }
-} finally {
-  cacheReadLock.unlock();
+
+// only read volatile once for consistency
+ImmutableCacheCopies lic = immutableCache;
+if (lic.childrenCache.containsKey(zPath)) {
+  return lic.childrenCache.get(zPath);
 }
 
 cacheWriteLock.lock();
@@ -247,7 +279,11 @@ public T retry() {
   final ZooKeeper zooKeeper = getZooKeeper();
 
   List children = zooKeeper.getChildren(zPath, watcher);
+  if (children != null) {
+children = ImmutableList.copyOf(children);
+  }
   childrenCache.put(zPath, children);
+  immutableCache = new ImmutableCacheCopies(immutableCache, 
childrenCache);
   return children;
 } catch (KeeperException ke) {
   if (ke.code() != Code.NONODE) {
@@ -261,12 +297,7 @@ public T retry() {
 
 };
 
-List children = zr.retry();
-
-if (children == null) {
-  return null;
-}
-return Collections.unmodifiableList(children);
+return zr.retry();
   }
 
   /**
@@ -295,15 +326,16 @@ public T retry() {
   @Override
   public byte[] run() throws KeeperException, InterruptedException {
 Stat stat = null;
-cacheReadLock.lock();
-try {
-  if (cache.containsKey(zPath)) {
-stat = statCache.get(zPath);
+
+// only read volatile once so following code 

Accumulo-Pull-Requests - Build # 992 - Fixed

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-Pull-Requests (build #992)

Status: Fixed

Check console output at 
https://builds.apache.org/job/Accumulo-Pull-Requests/992/ to view the results.

[GitHub] ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r164005780
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java
 ##
 @@ -629,6 +633,11 @@ public String getRawDefaultValue() {
* @return default value
*/
   public String getDefaultValue() {
+Preconditions.checkState(defaultValueComputed);
+return computedDefaultValue;
+  }
+
+  private void precomputeDefaultValue() {
 
 Review comment:
   Interesting that the static block runs after the constructors. I guess enum 
instances are like `private static final MyClass NAME = new MyClass()` in that 
way? But I thought even those ran after other static initializers. Maybe order 
matters. Best not to think about it. I think I'll leave Java class 
initialization above my head.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] keith-turner commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

keith-turner commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r164004925
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java
 ##
 @@ -629,6 +633,11 @@ public String getRawDefaultValue() {
* @return default value
*/
   public String getDefaultValue() {
+Preconditions.checkState(defaultValueComputed);
 
 Review comment:
   I think an IllegalStateException is ok.  I changed the code to provide an 
error message.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] keith-turner commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

keith-turner commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r164004925
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java
 ##
 @@ -629,6 +633,11 @@ public String getRawDefaultValue() {
* @return default value
*/
   public String getDefaultValue() {
+Preconditions.checkState(defaultValueComputed);
 
 Review comment:
   I think an IllegalStateExcetpion is ok.  I changed the code to provide an 
error message.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] keith-turner commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

keith-turner commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r164004677
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java
 ##
 @@ -629,6 +633,11 @@ public String getRawDefaultValue() {
* @return default value
*/
   public String getDefaultValue() {
+Preconditions.checkState(defaultValueComputed);
+return computedDefaultValue;
+  }
+
+  private void precomputeDefaultValue() {
 
 Review comment:
   Currently the code calls isInterpolated() which does not work until the 
static block runs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] keith-turner opened a new pull request #367: ACCUMULO-4779 made getting vfs config more efficient

[GitBox]

keith-turner opened a new pull request #367: ACCUMULO-4779 made getting vfs 
config more efficient
URL: https://github.com/apache/accumulo/pull/367
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Accumulo-1.7 - Build # 403 - Unstable

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-1.7 (build #403)

Status: Unstable

Check console output at https://builds.apache.org/job/Accumulo-1.7/403/ to view 
the results.

Accumulo-Pull-Requests - Build # 991 - Failure

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-Pull-Requests (build #991)

Status: Failure

Check console output at 
https://builds.apache.org/job/Accumulo-Pull-Requests/991/ to view the results.

[jira] [Commented] (ACCUMULO-4777) Root tablet got spammed with 1.8 million log entries

[Ivan Bella (JIRA)]

[ 
https://issues.apache.org/jira/browse/ACCUMULO-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340216#comment-16340216
 ] 

Ivan Bella commented on ACCUMULO-4777:
--

I applied the changes to 1.7, merged into 1.8 and subsequently into master.

> Root tablet got spammed with 1.8 million log entries
> 
>
> Key: ACCUMULO-4777
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4777
> Project: Accumulo
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Ivan Bella
>Assignee: Ivan Bella
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.7.4, 1.9.0, 2.0.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> We had a tserver that was handling accumulo.metadata tablets that somehow got 
> into a loop where it created over 22K empty wal logs.  There were around 70 
> metadata tablets and this resulted in around 1.8 million log entries in added 
> to the accumulo.root table.  The only reason it stopped creating wal logs is 
> because it ran out of open file handles.  This took us many hours and cups of 
> coffee to clean up.
> The log contained the following messages in a tight loop:
> log.TabletServerLogger INFO : Using next log hdfs://...
> tserver.TabletServfer INFO : Writing log marker for hdfs://...
> tserver.TabletServer INFO : Marking hdfs://... closed
> log.DfsLogger INFO : Slow sync cost ...
> ...
> Unfortunately we did not have DEBUG turned on so we have no debug messages.
> Tracking through the code there are three places where the 
> TabletServerLogger.close method is called:
> 1) via resetLoggers in the TabletServerLogger, but nothing calls this method 
> so this is ruled out
> 2) when the log gets too large or too old, but neither of those checks should 
> have been hitting here.
> 3) In a loop that is executed (while (!success)) in the 
> TabletServerLogger.write method.  In this case when we unsuccessfullty write 
> something to the wal, then that one is closed and a new one is created.  This 
> loop will go forever until we successfully write out the entry.  A 
> DfsLogger.LogClosedException seems the most logical reason.  This is most 
> likely because a ClosedChannelException was thrown from the DfsLogger.write 
> methods (around line 609 in DfsLogger).
> So the root cause was most likely hadoop related.  However in accumulo we 
> probably should not be doing a tight retry loop around a hadoop failure.  I 
> recommend at a minimum doing some sort of exponential back off and perhaps 
> setting a limit on the number of retries resulting in a critical tserver 
> failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (ACCUMULO-4777) Root tablet got spammed with 1.8 million log entries

[Ivan Bella (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ACCUMULO-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Bella resolved ACCUMULO-4777.
--
Resolution: Fixed

> Root tablet got spammed with 1.8 million log entries
> 
>
> Key: ACCUMULO-4777
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4777
> Project: Accumulo
>  Issue Type: Bug
>Affects Versions: 1.8.1
>Reporter: Ivan Bella
>Assignee: Ivan Bella
>Priority: Critical
>  Labels: pull-request-available
> Fix For: 1.7.4, 1.9.0, 2.0.0
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> We had a tserver that was handling accumulo.metadata tablets that somehow got 
> into a loop where it created over 22K empty wal logs.  There were around 70 
> metadata tablets and this resulted in around 1.8 million log entries in added 
> to the accumulo.root table.  The only reason it stopped creating wal logs is 
> because it ran out of open file handles.  This took us many hours and cups of 
> coffee to clean up.
> The log contained the following messages in a tight loop:
> log.TabletServerLogger INFO : Using next log hdfs://...
> tserver.TabletServfer INFO : Writing log marker for hdfs://...
> tserver.TabletServer INFO : Marking hdfs://... closed
> log.DfsLogger INFO : Slow sync cost ...
> ...
> Unfortunately we did not have DEBUG turned on so we have no debug messages.
> Tracking through the code there are three places where the 
> TabletServerLogger.close method is called:
> 1) via resetLoggers in the TabletServerLogger, but nothing calls this method 
> so this is ruled out
> 2) when the log gets too large or too old, but neither of those checks should 
> have been hitting here.
> 3) In a loop that is executed (while (!success)) in the 
> TabletServerLogger.write method.  In this case when we unsuccessfullty write 
> something to the wal, then that one is closed and a new one is created.  This 
> loop will go forever until we successfully write out the entry.  A 
> DfsLogger.LogClosedException seems the most logical reason.  This is most 
> likely because a ClosedChannelException was thrown from the DfsLogger.write 
> methods (around line 609 in DfsLogger).
> So the root cause was most likely hadoop related.  However in accumulo we 
> probably should not be doing a tight retry loop around a hadoop failure.  I 
> recommend at a minimum doing some sort of exponential back off and perhaps 
> setting a limit on the number of retries resulting in a critical tserver 
> failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ivakegg commented on issue #356: ACCUMULO-4777 Removed the unused sequence generator.

[GitBox]

ivakegg commented on issue #356: ACCUMULO-4777 Removed the unused sequence 
generator.
URL: https://github.com/apache/accumulo/pull/356#issuecomment-360623097
 
 
   I applied these changes to 1.7, and merged into 1.8 and subsequently into 
master


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ivakegg closed pull request #356: ACCUMULO-4777 Removed the unused sequence generator.

[GitBox]

ivakegg closed pull request #356: ACCUMULO-4777 Removed the unused sequence 
generator.
URL: https://github.com/apache/accumulo/pull/356
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/core/src/main/java/org/apache/accumulo/core/conf/Property.java 
b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
index 08232234db..26762fbd85 100644
--- a/core/src/main/java/org/apache/accumulo/core/conf/Property.java
+++ b/core/src/main/java/org/apache/accumulo/core/conf/Property.java
@@ -263,13 +263,13 @@
   "The maximum size for each write-ahead log. See comment for property 
tserver.memory.maps.max"),
   TSERV_WALOG_MAX_AGE("tserver.walog.max.age", "24h", 
PropertyType.TIMEDURATION, "The maximum age for each write-ahead log."),
   
TSERV_WALOG_TOLERATED_CREATION_FAILURES("tserver.walog.tolerated.creation.failures",
 "50", PropertyType.COUNT,
-  "The maximum number of failures tolerated when creating a new WAL file 
within the period specified by tserver.walog.failures.period."
-  + " Exceeding this number of failures in the period causes the 
TabletServer to exit."),
+  "The maximum number of failures tolerated when creating a new WAL file.  
Values < 0 will allow unlimited creation failures."
+  + " Exceeding this number of failures consecutively trying to create 
a new WAL causes the TabletServer to exit."),
   
TSERV_WALOG_TOLERATED_WAIT_INCREMENT("tserver.walog.tolerated.wait.increment", 
"1000ms", PropertyType.TIMEDURATION,
-  "The amount of time to wait between failures to create a WALog."),
+  "The amount of time to wait between failures to create or write a 
WALog."),
   // Never wait longer than 5 mins for a retry
   
TSERV_WALOG_TOLERATED_MAXIMUM_WAIT_DURATION("tserver.walog.maximum.wait.duration",
 "5m", PropertyType.TIMEDURATION,
-  "The maximum amount of time to wait after a failure to create a WAL 
file."),
+  "The maximum amount of time to wait after a failure to create or write a 
WAL file."),
   TSERV_MAJC_DELAY("tserver.compaction.major.delay", "30s", 
PropertyType.TIMEDURATION,
   "Time a tablet server will sleep between checking which tablets need 
compaction."),
   TSERV_MAJC_THREAD_MAXOPEN("tserver.compaction.major.thread.files.open.max", 
"10", PropertyType.COUNT,
diff --git a/fate/src/main/java/org/apache/accumulo/fate/zookeeper/Retry.java 
b/fate/src/main/java/org/apache/accumulo/fate/zookeeper/Retry.java
index e84b1af325..1f55d72fe6 100644
--- a/fate/src/main/java/org/apache/accumulo/fate/zookeeper/Retry.java
+++ b/fate/src/main/java/org/apache/accumulo/fate/zookeeper/Retry.java
@@ -25,12 +25,14 @@
 public class Retry {
   private static final Logger log = LoggerFactory.getLogger(Retry.class);
 
+  public static final long MAX_RETRY_DISABLED = -1;
+
   private long maxRetries, maxWait, waitIncrement;
   private long retriesDone, currentWait;
 
   /**
* @param maxRetries
-   *  Maximum times to retry
+   *  Maximum times to retry or MAX_RETRY_DISABLED if no maximum
* @param startWait
*  The amount of time (ms) to wait for the initial retry
* @param maxWait
@@ -46,6 +48,18 @@ public Retry(long maxRetries, long startWait, long 
waitIncrement, long maxWait)
 this.currentWait = startWait;
   }
 
+  /**
+   * @param startWait
+   *  The amount of time (ms) to wait for the initial retry
+   * @param maxWait
+   *  The maximum wait (ms)
+   * @param waitIncrement
+   *  The amount of time (ms) to increment next wait time by
+   */
+  public Retry(long startWait, long waitIncrement, long maxWait) {
+this(MAX_RETRY_DISABLED, startWait, waitIncrement, maxWait);
+  }
+
   // Visible for testing
   long getMaxRetries() {
 return maxRetries;
@@ -86,8 +100,12 @@ void setMaxWait(long maxWait) {
 this.maxWait = maxWait;
   }
 
+  public boolean isMaxRetryDisabled() {
+return maxRetries < 0;
+  }
+
   public boolean canRetry() {
-return retriesDone < maxRetries;
+return isMaxRetryDisabled() || (retriesDone < maxRetries);
   }
 
   public void useRetry() {
diff --git 
a/fate/src/main/java/org/apache/accumulo/fate/zookeeper/RetryFactory.java 
b/fate/src/main/java/org/apache/accumulo/fate/zookeeper/RetryFactory.java
index 63a1241700..aa6da20fa8 100644
--- a/fate/src/main/java/org/apache/accumulo/fate/zookeeper/RetryFactory.java
+++ b/fate/src/main/java/org/apache/accumulo/fate/zookeeper/RetryFactory.java
@@ -25,6 +25,18 @@
 
   private final long maxRetries, startWait, maxWait, waitIncrement;
 
+  /**
+   * Create a retry factor for retries with a limit
+   *
+   * @param maxRetries
+   *  The maximum number of retries
+   * @param startWait
+   *  The wait ms for the first retry
+   * @param waitIncrement
+  

Accumulo-Pull-Requests - Build # 990 - Fixed

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-Pull-Requests (build #990)

Status: Fixed

Check console output at 
https://builds.apache.org/job/Accumulo-Pull-Requests/990/ to view the results.

Accumulo-Pull-Requests - Build # 989 - Still unstable

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-Pull-Requests (build #989)

Status: Still unstable

Check console output at 
https://builds.apache.org/job/Accumulo-Pull-Requests/989/ to view the results.

[GitHub] ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r163957225
 
 

 ##
 File path: core/src/test/java/org/apache/accumulo/core/conf/PropertyTest.java
 ##
 @@ -152,4 +153,57 @@ public void validatePropertyKeys() {
   public void testGCDeadServerWaitSecond() {
 assertEquals("1h", Property.GC_WAL_DEAD_SERVER_WAIT.getDefaultValue());
   }
+
+  @SuppressWarnings("deprecation")
 
 Review comment:
   Rather than suppress warnings across the entire method, it's better to only 
suppress the one case in the method that is deprecated. This avoids masking 
future un-triaged warnings.
   
   Here, the `Property.INSTANCE_DFS_URI` can be assigned to a variable of type 
Property just before its corresponding `assertTrue` line, and the assignment 
can be suppressed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r163952193
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java
 ##
 @@ -629,6 +633,11 @@ public String getRawDefaultValue() {
* @return default value
*/
   public String getDefaultValue() {
+Preconditions.checkState(defaultValueComputed);
+return computedDefaultValue;
+  }
+
+  private void precomputeDefaultValue() {
 
 Review comment:
   I wonder if we can't compute the default value in the constructor?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r163950956
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java
 ##
 @@ -589,7 +591,9 @@
 
   ;
 
-  private String key, defaultValue, description;
+  private String key, defaultValue, computedDefaultValue, description;
+  private boolean annotationsComputed = false, defaultValueComputed = false;
+  private boolean isSensitive, isDeprecated, isExperimental, isInterpolated;
 
 Review comment:
   This is a tiny thing, and I realize we had this there before... but I'm not 
generally a fan of multiple declarations. It's cleaner to declare one variable 
per line... it's easier to see which keywords apply.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r163955055
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java
 ##
 @@ -709,7 +727,20 @@ public boolean isSensitive() {
* @return true if property is sensitive
*/
   public static boolean isSensitive(String key) {
 
 Review comment:
   I had thought we had a similar method for Experimental and Deprecated with 
arbitrary keys also, which would check based on the prefix... but perhaps not. 
I don't see it, but it could be useful for the shell config command output.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

ctubbsii commented on a change in pull request #366: ACCUMULO-4779 Speedup 
Property by precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366#discussion_r163950613
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/conf/Property.java
 ##
 @@ -629,6 +633,11 @@ public String getRawDefaultValue() {
* @return default value
*/
   public String getDefaultValue() {
+Preconditions.checkState(defaultValueComputed);
 
 Review comment:
   I prefer throwing AssertionError if this is a check against a future 
programmer error, rather than a protection against a state that the system 
could find itself in for an unspecified reason. But, it's not a blocker.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Accumulo-Pull-Requests - Build # 988 - Still Failing

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-Pull-Requests (build #988)

Status: Still Failing

Check console output at 
https://builds.apache.org/job/Accumulo-Pull-Requests/988/ to view the results.

[GitHub] keith-turner opened a new pull request #366: ACCUMULO-4779 Speedup Property by precomputing and avoiding sync

[GitBox]

keith-turner opened a new pull request #366: ACCUMULO-4779 Speedup Property by 
precomputing and avoiding sync
URL: https://github.com/apache/accumulo/pull/366
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (ACCUMULO-4355) Provide more granular control for bulk import operations

[Michael Hogue (JIRA)]

[ 
https://issues.apache.org/jira/browse/ACCUMULO-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339506#comment-16339506
 ] 

Michael Hogue commented on ACCUMULO-4355:
-

[~ShawnWalker]: I'm interested in what sort of progress has made on this issue 
given your implementation ideas. I'm happy to contribute if there's an avenue 
to do so.

> Provide more granular control for bulk import operations
> 
>
> Key: ACCUMULO-4355
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4355
> Project: Accumulo
>  Issue Type: Wish
>  Components: master, tserver
>Reporter: Shawn Walker
>Assignee: Shawn Walker
>Priority: Major
> Fix For: 2.0.0
>
>
> Accumulo currently provides mechanisms to initiate bulk imports and to list 
> bulk imports in progress.  Scheduling of bulk import requests is not entirely 
> deterministic, and most of the execution of a bulk-import request is done in 
> a non-preemptable manner.  As such, any bulk import which takes very long to 
> complete can block bulk imports with higher operational priority for 
> significant periods.
> To better support bulk-import-heavy applications, it would be nice if 
> Accumulo would offer additional mechanisms for controlling the scheduling and 
> execution of bulk imports, such as the abilities to:
> * Pause/resume bulk import in progress.
> * Prioritize/reprioritize bulk import requests.
> * Cancel bulk import in progress.  If possible, cancelling a partially 
> completed bulk import request should result in a rollback of changes.  That 
> is, a bulk import should either succeed or make no changes.
> Additionally, for multitenant situations, it would be nice if Accumulo would:
> * Provide multiple queues for bulk import requests.  Each queue would have 
> its requests worked serially in priority order.  Requests in separate queues 
> should be worked in parallel, or have time distributed among the queues in 
> some manner as to make work appear roughly parallel.
> 
> Implementation-wise, I'm thinking of rewriting much of the current 
> bulk-loading logic.  While the current logic depends upon multiple threads 
> executing (potentially long-duration) blocking RPC calls, I'd like to move to 
> a more event-driven/message-passing model backed by a persistent state 
> machine.
> Current ideas I'm playing around with (very tentative)
> * Creating a new table {{accumulo.bulk_load_queues}} to keep track of bulk 
> load progress.
> * Distributing bulk load orchestration via a mechanism similar to tablet 
> assignment instead of the current blocking RPC calls (LoadFiles.java:156).
> * Implementing something akin to a two-phase commit to achieve rollback 
> behavior on failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] keith-turner commented on a change in pull request #364: ACCUMULO-4778 Initial feedback for table name to id mapping cache

[GitBox]

keith-turner commented on a change in pull request #364: ACCUMULO-4778 Initial 
feedback for table name to id mapping cache
URL: https://github.com/apache/accumulo/pull/364#discussion_r163900901
 
 

 ##
 File path: core/src/main/java/org/apache/accumulo/core/client/impl/Tables.java
 ##
 @@ -129,12 +85,29 @@ public static String getTableName(Instance instance, 
String tableId) throws Tabl
 return tableName;
   }
 
-  public static SortedMap getNameToIdMap(Instance instance) {
-return getMap(instance, true);
+  public static Map getNameToIdMap(Instance instance) {
+return getTableMap(instance).getNameToIdMap();
   }
 
-  public static SortedMap getIdToNameMap(Instance instance) {
-return getMap(instance, false);
+  public static Map getIdToNameMap(Instance instance) {
+return getTableMap(instance).getIdtoNameMap();
+  }
+
+  /**
+   * Create a new TableMap if needed and cache it locally. The creation of the 
TableMap is synchronized so only one thread will generate the map. Added for
+   * ACCUMULO-4778.
+   */
+  private static TableMap getTableMap(Instance instance) {
+TableMap map = tableMapCache;
+if (map == null || map.getCreationTime() + TABLE_MAP_CACHE_EXPIRATION <= 
System.nanoTime()) {
+  synchronized (Tables.class) {
+map = tableMapCache;
+if (map == null || map.getCreationTime() + TABLE_MAP_CACHE_EXPIRATION 
<= System.nanoTime()) {
+  map = tableMapCache = new TableMap(instance, getZooCache(instance));
 
 Review comment:
   probably need a TableMap per instance


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

Accumulo-Pull-Requests - Build # 987 - Failure

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-Pull-Requests (build #987)

Status: Failure

Check console output at 
https://builds.apache.org/job/Accumulo-Pull-Requests/987/ to view the results.

Accumulo-Pull-Requests - Build # 986 - Still unstable

[Apache Jenkins Server]

The Apache Jenkins build system has built Accumulo-Pull-Requests (build #986)

Status: Still unstable

Check console output at 
https://builds.apache.org/job/Accumulo-Pull-Requests/986/ to view the results.

[jira] [Resolved] (ACCUMULO-4731) Improve exception handling if a key encryption key cannot be loaded

[Nick Felts (JIRA)]

 [ 
https://issues.apache.org/jira/browse/ACCUMULO-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Felts resolved ACCUMULO-4731.
--
Resolution: Fixed

> Improve exception handling if a key encryption key cannot be loaded
> ---
>
> Key: ACCUMULO-4731
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4731
> Project: Accumulo
>  Issue Type: Improvement
>  Components: tserver
>Reporter: Nick Felts
>Assignee: Nick Felts
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 2.0.0
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)