[jira] [Comment Edited] (HADOOP-14312) RetryInvocationHandler may report ANN as SNN in messages.
[
https://issues.apache.org/jira/browse/HADOOP-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969823#comment-15969823
]
Yongjun Zhang edited comment on HADOOP-14312 at 4/15/17 5:40 AM:
-
HI [~szetszwo],
Thanks for your work at
HADOOP-13146. Refactor RetryInvocationHandler.
HADOOP-13226 Support async call retry and failover.
I uploaded a preliminary patch, which introduces a thread local variable to
store proxy info, This would incur some cost.
To avoid the cost, I wish we could incorporate the proxy info in the exception
thrown from the method below, instead of we have to store this info separately.
But this would be changing the exception thrown from this method, and changing
the methods that calls this one, etc. Not clean, if not infeasible.
{code}
protected Object invokeMethod(Method method, Object[] args) throws Throwable {
try {
if (!method.isAccessible()) {
method.setAccessible(true);
}
final Object r = method.invoke(proxyDescriptor.getProxy(), args);
hasSuccessfulCall = true;
return r;
} catch (InvocationTargetException e) {
throw e.getCause();
}
}
{code}
Another way we can solve this problem is, don't report the proxy if there is no
failover, only reports the proxy info when doing failover, which is done in a
synchronized way. But we will lose some info in the log.
I would really appreciate if you have other suggestions.
Thanks a lot.
was (Author: yzhangal):
HI [~szetszwo],
Thanks for your work at
HADOOP-13146. Refactor RetryInvocationHandler.
HADOOP-13226 Support async call retry and failover.
I uploaded a preliminary patch, which may introduces a thread local variable to
store proxy info, This would incur some cost.
To avoid the cost, I wish we could incorporate the proxy info in the exception
thrown from the method below, instead of we have to store this info separately.
But this would be changing the exception thrown from this method, and changing
the methods that calls this one, etc. Not clean, if not infeasible.
{code}
protected Object invokeMethod(Method method, Object[] args) throws Throwable {
try {
if (!method.isAccessible()) {
method.setAccessible(true);
}
final Object r = method.invoke(proxyDescriptor.getProxy(), args);
hasSuccessfulCall = true;
return r;
} catch (InvocationTargetException e) {
throw e.getCause();
}
}
{code}
Another way we can solve this problem is, don't report the proxy if there is no
failover, only reports the proxy info when doing failover, which is done in a
synchronized way. But we will lose some info in the log.
I would really appreciate if you have other suggestions.
Thanks a lot.
> RetryInvocationHandler may report ANN as SNN in messages.
> --
>
> Key: HADOOP-14312
> URL: https://issues.apache.org/jira/browse/HADOOP-14312
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14312.001.patch
>
>
> When multiple threads use the same DFSClient to make RPC calls, they may
> report incorrect NN host name in messages like
> INFO [pool-3-thread-13] retry.RetryInvocationHandler
> (RetryInvocationHandler.java:invoke(148)) - Exception while invoking delete
> of class ClientNamenodeProtocolTranslatorPB over
> hdpb-nn0001.prn.parsec.apple.com/*a.b.c.d*:8020. Trying to fail over
> immediately.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category WRITE is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> where *a.b.c.d* is the active NN, which confuses user to think failover is
> not behaving correctly.
> The reason is that the ProxyDescriptor data field of RetryInvocationHandler
> may be shared by multiple threads that do the RPC calls, the failover done by
> one thread may be visible to other threads when reporting the above kind of
> message.
> As an example,
> # multiple threads start with the same SNN to do RPC calls,
> # all threads discover that a failover is needed,
> # thread X failover first, and changed the ProxyDescriptor's proxyInfo to ANN
> # other threads reports the above message with the proxyInfo changed by
> thread X, and reported ANN instead of SNN in the message.
> Some details:
> RetryInvocationHandler does the following when failing over:
> {code}
> synchronized void failover(long expectedFailoverCount, Method method,
>int callId) {
> // Make sure that concurrent failed invocations only cause a single
> // actual failover.
> if (failoverCount == expectedFailoverCount) {
> fpp.performFailover(proxyInfo.proxy);
> failoverCount++;
> } else
[jira] [Commented] (HADOOP-14312) RetryInvocationHandler may report ANN as SNN in messages.
[
https://issues.apache.org/jira/browse/HADOOP-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969823#comment-15969823
]
Yongjun Zhang commented on HADOOP-14312:
HI [~szetszwo],
Thanks for your work at
HADOOP-13146. Refactor RetryInvocationHandler.
HADOOP-13226 Support async call retry and failover.
I uploaded a preliminary patch, which may introduces a thread local variable to
store proxy info, This would incur some cost.
To avoid the cost, I wish we could incorporate the proxy info in the exception
thrown from the method below, instead of we have to store this info separately.
But this would be changing the exception thrown from this method, and changing
the methods that calls this one, etc. Not clean, if not infeasible.
{code}
protected Object invokeMethod(Method method, Object[] args) throws Throwable {
try {
if (!method.isAccessible()) {
method.setAccessible(true);
}
final Object r = method.invoke(proxyDescriptor.getProxy(), args);
hasSuccessfulCall = true;
return r;
} catch (InvocationTargetException e) {
throw e.getCause();
}
}
{code}
Another way we can solve this problem is, don't report the proxy if there is no
failover, only reports the proxy info when doing failover, which is done in a
synchronized way. But we will lose some info in the log.
I would really appreciate if you have other suggestions.
Thanks a lot.
> RetryInvocationHandler may report ANN as SNN in messages.
> --
>
> Key: HADOOP-14312
> URL: https://issues.apache.org/jira/browse/HADOOP-14312
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14312.001.patch
>
>
> When multiple threads use the same DFSClient to make RPC calls, they may
> report incorrect NN host name in messages like
> INFO [pool-3-thread-13] retry.RetryInvocationHandler
> (RetryInvocationHandler.java:invoke(148)) - Exception while invoking delete
> of class ClientNamenodeProtocolTranslatorPB over
> hdpb-nn0001.prn.parsec.apple.com/*a.b.c.d*:8020. Trying to fail over
> immediately.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category WRITE is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> where *a.b.c.d* is the active NN, which confuses user to think failover is
> not behaving correctly.
> The reason is that the ProxyDescriptor data field of RetryInvocationHandler
> may be shared by multiple threads that do the RPC calls, the failover done by
> one thread may be visible to other threads when reporting the above kind of
> message.
> As an example,
> # multiple threads start with the same SNN to do RPC calls,
> # all threads discover that a failover is needed,
> # thread X failover first, and changed the ProxyDescriptor's proxyInfo to ANN
> # other threads reports the above message with the proxyInfo changed by
> thread X, and reported ANN instead of SNN in the message.
> Some details:
> RetryInvocationHandler does the following when failing over:
> {code}
> synchronized void failover(long expectedFailoverCount, Method method,
>int callId) {
> // Make sure that concurrent failed invocations only cause a single
> // actual failover.
> if (failoverCount == expectedFailoverCount) {
> fpp.performFailover(proxyInfo.proxy);
> failoverCount++;
> } else {
> LOG.warn("A failover has occurred since the start of call #" + callId
> + " " + proxyInfo.getString(method.getName()));
> }
> proxyInfo = fpp.getProxy();
> }
> {code}
> and changed the proxyInfo in the ProxyDescriptor.
> While the log method below report message with ProxyDescriotor's proxyinfo:
> {code}
> private void log(final Method method, final boolean isFailover,
> final int failovers, final long delay, final Exception ex) {
> ..
>final StringBuilder b = new StringBuilder()
> .append(ex + ", while invoking ")
> .append(proxyDescriptor.getProxyInfo().getString(method.getName()));
> if (failovers > 0) {
> b.append(" after ").append(failovers).append(" failover attempts");
> }
> b.append(isFailover? ". Trying to failover ": ". Retrying ");
> b.append(delay > 0? "after sleeping for " + delay + "ms.":
> "immediately.");
> {code}
> and so does {{handleException}} method do
> {code}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Exception while invoking call #" + callId + " "
> + proxyDescriptor.getProxyInfo().getString(method.getName())
> + ". Not retrying because " + retryInfo.action.reason, e);
> }
> {code}
> FailoverProxyProvider
> {code}
>public String
[jira] [Updated] (HADOOP-14312) RetryInvocationHandler may report ANN as SNN in messages.
[
https://issues.apache.org/jira/browse/HADOOP-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-14312:
---
Status: Patch Available (was: Open)
> RetryInvocationHandler may report ANN as SNN in messages.
> --
>
> Key: HADOOP-14312
> URL: https://issues.apache.org/jira/browse/HADOOP-14312
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14312.001.patch
>
>
> When multiple threads use the same DFSClient to make RPC calls, they may
> report incorrect NN host name in messages like
> INFO [pool-3-thread-13] retry.RetryInvocationHandler
> (RetryInvocationHandler.java:invoke(148)) - Exception while invoking delete
> of class ClientNamenodeProtocolTranslatorPB over
> hdpb-nn0001.prn.parsec.apple.com/*a.b.c.d*:8020. Trying to fail over
> immediately.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category WRITE is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> where *a.b.c.d* is the active NN, which confuses user to think failover is
> not behaving correctly.
> The reason is that the ProxyDescriptor data field of RetryInvocationHandler
> may be shared by multiple threads that do the RPC calls, the failover done by
> one thread may be visible to other threads when reporting the above kind of
> message.
> As an example,
> # multiple threads start with the same SNN to do RPC calls,
> # all threads discover that a failover is needed,
> # thread X failover first, and changed the ProxyDescriptor's proxyInfo to ANN
> # other threads reports the above message with the proxyInfo changed by
> thread X, and reported ANN instead of SNN in the message.
> Some details:
> RetryInvocationHandler does the following when failing over:
> {code}
> synchronized void failover(long expectedFailoverCount, Method method,
>int callId) {
> // Make sure that concurrent failed invocations only cause a single
> // actual failover.
> if (failoverCount == expectedFailoverCount) {
> fpp.performFailover(proxyInfo.proxy);
> failoverCount++;
> } else {
> LOG.warn("A failover has occurred since the start of call #" + callId
> + " " + proxyInfo.getString(method.getName()));
> }
> proxyInfo = fpp.getProxy();
> }
> {code}
> and changed the proxyInfo in the ProxyDescriptor.
> While the log method below report message with ProxyDescriotor's proxyinfo:
> {code}
> private void log(final Method method, final boolean isFailover,
> final int failovers, final long delay, final Exception ex) {
> ..
>final StringBuilder b = new StringBuilder()
> .append(ex + ", while invoking ")
> .append(proxyDescriptor.getProxyInfo().getString(method.getName()));
> if (failovers > 0) {
> b.append(" after ").append(failovers).append(" failover attempts");
> }
> b.append(isFailover? ". Trying to failover ": ". Retrying ");
> b.append(delay > 0? "after sleeping for " + delay + "ms.":
> "immediately.");
> {code}
> and so does {{handleException}} method do
> {code}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Exception while invoking call #" + callId + " "
> + proxyDescriptor.getProxyInfo().getString(method.getName())
> + ". Not retrying because " + retryInfo.action.reason, e);
> }
> {code}
> FailoverProxyProvider
> {code}
>public String getString(String methodName) {
> return proxy.getClass().getSimpleName() + "." + methodName
> + " over " + proxyInfo;
> }
> @Override
> public String toString() {
> return proxy.getClass().getSimpleName() + " over " + proxyInfo;
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14312) RetryInvocationHandler may report ANN as SNN in messages.
[
https://issues.apache.org/jira/browse/HADOOP-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HADOOP-14312:
---
Attachment: HADOOP-14312.001.patch
> RetryInvocationHandler may report ANN as SNN in messages.
> --
>
> Key: HADOOP-14312
> URL: https://issues.apache.org/jira/browse/HADOOP-14312
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HADOOP-14312.001.patch
>
>
> When multiple threads use the same DFSClient to make RPC calls, they may
> report incorrect NN host name in messages like
> INFO [pool-3-thread-13] retry.RetryInvocationHandler
> (RetryInvocationHandler.java:invoke(148)) - Exception while invoking delete
> of class ClientNamenodeProtocolTranslatorPB over
> hdpb-nn0001.prn.parsec.apple.com/*a.b.c.d*:8020. Trying to fail over
> immediately.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category WRITE is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> where *a.b.c.d* is the active NN, which confuses user to think failover is
> not behaving correctly.
> The reason is that the ProxyDescriptor data field of RetryInvocationHandler
> may be shared by multiple threads that do the RPC calls, the failover done by
> one thread may be visible to other threads when reporting the above kind of
> message.
> As an example,
> # multiple threads start with the same SNN to do RPC calls,
> # all threads discover that a failover is needed,
> # thread X failover first, and changed the ProxyDescriptor's proxyInfo to ANN
> # other threads reports the above message with the proxyInfo changed by
> thread X, and reported ANN instead of SNN in the message.
> Some details:
> RetryInvocationHandler does the following when failing over:
> {code}
> synchronized void failover(long expectedFailoverCount, Method method,
>int callId) {
> // Make sure that concurrent failed invocations only cause a single
> // actual failover.
> if (failoverCount == expectedFailoverCount) {
> fpp.performFailover(proxyInfo.proxy);
> failoverCount++;
> } else {
> LOG.warn("A failover has occurred since the start of call #" + callId
> + " " + proxyInfo.getString(method.getName()));
> }
> proxyInfo = fpp.getProxy();
> }
> {code}
> and changed the proxyInfo in the ProxyDescriptor.
> While the log method below report message with ProxyDescriotor's proxyinfo:
> {code}
> private void log(final Method method, final boolean isFailover,
> final int failovers, final long delay, final Exception ex) {
> ..
>final StringBuilder b = new StringBuilder()
> .append(ex + ", while invoking ")
> .append(proxyDescriptor.getProxyInfo().getString(method.getName()));
> if (failovers > 0) {
> b.append(" after ").append(failovers).append(" failover attempts");
> }
> b.append(isFailover? ". Trying to failover ": ". Retrying ");
> b.append(delay > 0? "after sleeping for " + delay + "ms.":
> "immediately.");
> {code}
> and so does {{handleException}} method do
> {code}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Exception while invoking call #" + callId + " "
> + proxyDescriptor.getProxyInfo().getString(method.getName())
> + ". Not retrying because " + retryInfo.action.reason, e);
> }
> {code}
> FailoverProxyProvider
> {code}
>public String getString(String methodName) {
> return proxy.getClass().getSimpleName() + "." + methodName
> + " over " + proxyInfo;
> }
> @Override
> public String toString() {
> return proxy.getClass().getSimpleName() + " over " + proxyInfo;
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14312) RetryInvocationHandler may report ANN as SNN in messages.
[
https://issues.apache.org/jira/browse/HADOOP-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang reassigned HADOOP-14312:
--
Assignee: Yongjun Zhang
> RetryInvocationHandler may report ANN as SNN in messages.
> --
>
> Key: HADOOP-14312
> URL: https://issues.apache.org/jira/browse/HADOOP-14312
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> When multiple threads use the same DFSClient to make RPC calls, they may
> report incorrect NN host name in messages like
> INFO [pool-3-thread-13] retry.RetryInvocationHandler
> (RetryInvocationHandler.java:invoke(148)) - Exception while invoking delete
> of class ClientNamenodeProtocolTranslatorPB over
> hdpb-nn0001.prn.parsec.apple.com/*a.b.c.d*:8020. Trying to fail over
> immediately.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category WRITE is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> where *a.b.c.d* is the active NN, which confuses user to think failover is
> not behaving correctly.
> The reason is that the ProxyDescriptor data field of RetryInvocationHandler
> may be shared by multiple threads that do the RPC calls, the failover done by
> one thread may be visible to other threads when reporting the above kind of
> message.
> As an example,
> # multiple threads start with the same SNN to do RPC calls,
> # all threads discover that a failover is needed,
> # thread X failover first, and changed the ProxyDescriptor's proxyInfo to ANN
> # other threads reports the above message with the proxyInfo changed by
> thread X, and reported ANN instead of SNN in the message.
> Some details:
> RetryInvocationHandler does the following when failing over:
> {code}
> synchronized void failover(long expectedFailoverCount, Method method,
>int callId) {
> // Make sure that concurrent failed invocations only cause a single
> // actual failover.
> if (failoverCount == expectedFailoverCount) {
> fpp.performFailover(proxyInfo.proxy);
> failoverCount++;
> } else {
> LOG.warn("A failover has occurred since the start of call #" + callId
> + " " + proxyInfo.getString(method.getName()));
> }
> proxyInfo = fpp.getProxy();
> }
> {code}
> and changed the proxyInfo in the ProxyDescriptor.
> While the log method below report message with ProxyDescriotor's proxyinfo:
> {code}
> private void log(final Method method, final boolean isFailover,
> final int failovers, final long delay, final Exception ex) {
> ..
>final StringBuilder b = new StringBuilder()
> .append(ex + ", while invoking ")
> .append(proxyDescriptor.getProxyInfo().getString(method.getName()));
> if (failovers > 0) {
> b.append(" after ").append(failovers).append(" failover attempts");
> }
> b.append(isFailover? ". Trying to failover ": ". Retrying ");
> b.append(delay > 0? "after sleeping for " + delay + "ms.":
> "immediately.");
> {code}
> and so does {{handleException}} method do
> {code}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Exception while invoking call #" + callId + " "
> + proxyDescriptor.getProxyInfo().getString(method.getName())
> + ". Not retrying because " + retryInfo.action.reason, e);
> }
> {code}
> FailoverProxyProvider
> {code}
>public String getString(String methodName) {
> return proxy.getClass().getSimpleName() + "." + methodName
> + " over " + proxyInfo;
> }
> @Override
> public String toString() {
> return proxy.getClass().getSimpleName() + " over " + proxyInfo;
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Moved] (HADOOP-14312) RetryInvocationHandler may report ANN as SNN in messages.
[
https://issues.apache.org/jira/browse/HADOOP-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang moved HDFS-11656 to HADOOP-14312:
---
Key: HADOOP-14312 (was: HDFS-11656)
Project: Hadoop Common (was: Hadoop HDFS)
> RetryInvocationHandler may report ANN as SNN in messages.
> --
>
> Key: HADOOP-14312
> URL: https://issues.apache.org/jira/browse/HADOOP-14312
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Yongjun Zhang
>
> When multiple threads use the same DFSClient to make RPC calls, they may
> report incorrect NN host name in messages like
> INFO [pool-3-thread-13] retry.RetryInvocationHandler
> (RetryInvocationHandler.java:invoke(148)) - Exception while invoking delete
> of class ClientNamenodeProtocolTranslatorPB over
> hdpb-nn0001.prn.parsec.apple.com/*a.b.c.d*:8020. Trying to fail over
> immediately.
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):
> Operation category WRITE is not supported in state standby. Visit
> https://s.apache.org/sbnn-error
> where *a.b.c.d* is the active NN, which confuses user to think failover is
> not behaving correctly.
> The reason is that the ProxyDescriptor data field of RetryInvocationHandler
> may be shared by multiple threads that do the RPC calls, the failover done by
> one thread may be visible to other threads when reporting the above kind of
> message.
> As an example,
> # multiple threads start with the same SNN to do RPC calls,
> # all threads discover that a failover is needed,
> # thread X failover first, and changed the ProxyDescriptor's proxyInfo to ANN
> # other threads reports the above message with the proxyInfo changed by
> thread X, and reported ANN instead of SNN in the message.
> Some details:
> RetryInvocationHandler does the following when failing over:
> {code}
> synchronized void failover(long expectedFailoverCount, Method method,
>int callId) {
> // Make sure that concurrent failed invocations only cause a single
> // actual failover.
> if (failoverCount == expectedFailoverCount) {
> fpp.performFailover(proxyInfo.proxy);
> failoverCount++;
> } else {
> LOG.warn("A failover has occurred since the start of call #" + callId
> + " " + proxyInfo.getString(method.getName()));
> }
> proxyInfo = fpp.getProxy();
> }
> {code}
> and changed the proxyInfo in the ProxyDescriptor.
> While the log method below report message with ProxyDescriotor's proxyinfo:
> {code}
> private void log(final Method method, final boolean isFailover,
> final int failovers, final long delay, final Exception ex) {
> ..
>final StringBuilder b = new StringBuilder()
> .append(ex + ", while invoking ")
> .append(proxyDescriptor.getProxyInfo().getString(method.getName()));
> if (failovers > 0) {
> b.append(" after ").append(failovers).append(" failover attempts");
> }
> b.append(isFailover? ". Trying to failover ": ". Retrying ");
> b.append(delay > 0? "after sleeping for " + delay + "ms.":
> "immediately.");
> {code}
> and so does {{handleException}} method do
> {code}
> if (LOG.isDebugEnabled()) {
> LOG.debug("Exception while invoking call #" + callId + " "
> + proxyDescriptor.getProxyInfo().getString(method.getName())
> + ". Not retrying because " + retryInfo.action.reason, e);
> }
> {code}
> FailoverProxyProvider
> {code}
>public String getString(String methodName) {
> return proxy.getClass().getSimpleName() + "." + methodName
> + " over " + proxyInfo;
> }
> @Override
> public String toString() {
> return proxy.getClass().getSimpleName() + " over " + proxyInfo;
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14266) S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969811#comment-15969811
]
Mingliang Liu commented on HADOOP-14266:
[~fabbri] your comments are very precise. Thanks for the clear explanation! I
suggest we update the description of this JIRA using most of the above comments
when the patch is final.
After reading your comment, I also have two basic ideas to optimize further
along with your proposed future enhancement.
# For the {{!recursive && isAuthoritative}} case, we can return metadata store
cachedFilesIterator results without asking S3. This will be similar to
{{listLocatedStatus()}}.
# If we have returned value order guarantee from both S3 list object request
and metadata store {{DescendantsIterator}}, in {{FileStatusListingIterator}} we
can maintain two moving iterators and avoid pre-iterating providedStatus, in
which way it uses much less memory (no providedStatus HashSet then).
I will re-visit these two ideas again next week. Perhaps we can address them
later.
> S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
> ---
>
> Key: HADOOP-14266
> URL: https://issues.apache.org/jira/browse/HADOOP-14266
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14266-HADOOP-13345.000.patch,
> HADOOP-14266-HADOOP-13345.001.patch, HADOOP-14266-HADOOP-13345.002.patch,
> HADOOP-14266-HADOOP-13345.003.patch, HADOOP-14266-HADOOP-13345.003.patch,
> HADOOP-14266-HADOOP-13345.004.patch, HADOOP-14266-HADOOP-13345-005.patch,
> HADOOP-14266-HADOOP-13345.005.patch, HADOOP-14266-HADOOP-13345.006.patch
>
>
> Similar to [HADOOP-13926], this is to track the effort of employing
> MetadataStore in {{S3AFileSystem::listFiles()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14266) S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
[
https://issues.apache.org/jira/browse/HADOOP-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969765#comment-15969765
]
Aaron Fabbri commented on HADOOP-14266:
---
Thanks for your patience. The patch looks good to me. I am working through
the test code now and running all the integration tests. Thank you for the
extra list consistency test cases.
I took notes below, mostly for my understanding:
FileStatusListingIterator
- Now accepts an optional "providedStatus" iterator for entries from a
MetadataStore (previous patch added this, but it was an array).
{{FileStatusListingIterator}} will produce a union of the set of paths provided
by the underlying S3 iterator, and the set (if any) supplied via providedStatus
iterator.
listFiles()
- Before creating the S3 listing iterator, create a
{{cachedFileStatusIterator}} which enumerates the results from the
{{MetadataStore}}. In the recursive case, we use {{DescendantsIterator}} which
knows how to recursively enumerate a directory tree in MetadataStore. For
non-recursive case, we simply wrap the MetadataStore's listing in a
{{ProvidedFileStatusIterator}}.
- When creating the {{FileStatusListingIterator}} we can now pass in the
MetadataStore's iterator. {{listFiles()}} returns the
{{FileStatusListingIterator}} which returns the union of the S3 and
MetadataStore results.
- We do not try to optimize the {{recursive=true}} case yet. I agree with
this. I can imagine a scheme where we only demand-fetch S3 subtree listings
when we hit a subtree that is missing in MetadataStore or has
{{isAuthoritative == false}}, but that is complex and could actually perform
worse in limited corner cases. Sounds like future enhancement to me.
listLocatedStatus()
- Same logic, just refactored a bit, and uses the new Iterator (instead of
array) for MetadataStore results.
- This one (previous patch) *does* optimize the {{isAuthoritative}} case: if
the MetadataStore claims the listing is complete, and the S3A client is
configured to allow it ({{fs.s3a.metadatastore.authoritative=true}}), it will
skip the query to s3.
> S3Guard: S3AFileSystem::listFiles() to employ MetadataStore
> ---
>
> Key: HADOOP-14266
> URL: https://issues.apache.org/jira/browse/HADOOP-14266
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: HADOOP-13345
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-14266-HADOOP-13345.000.patch,
> HADOOP-14266-HADOOP-13345.001.patch, HADOOP-14266-HADOOP-13345.002.patch,
> HADOOP-14266-HADOOP-13345.003.patch, HADOOP-14266-HADOOP-13345.003.patch,
> HADOOP-14266-HADOOP-13345.004.patch, HADOOP-14266-HADOOP-13345-005.patch,
> HADOOP-14266-HADOOP-13345.005.patch, HADOOP-14266-HADOOP-13345.006.patch
>
>
> Similar to [HADOOP-13926], this is to track the effort of employing
> MetadataStore in {{S3AFileSystem::listFiles()}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969742#comment-15969742 ] Kazuyuki Tanimura edited comment on HADOOP-14237 at 4/15/17 12:53 AM: -- I will make this an independent credential provider. was (Author: kazuyukitanimura): I will make this an independnet credential provider. > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14237) S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes
[ https://issues.apache.org/jira/browse/HADOOP-14237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969742#comment-15969742 ] Kazuyuki Tanimura commented on HADOOP-14237: I will make this an independnet credential provider. > S3A Support Shared Instance Profile Credentials Across All Hadoop Nodes > --- > > Key: HADOOP-14237 > URL: https://issues.apache.org/jira/browse/HADOOP-14237 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0, 3.0.0-alpha1, 3.0.0-alpha2, 2.8.1 > Environment: EC2, AWS >Reporter: Kazuyuki Tanimura >Assignee: Kazuyuki Tanimura > > When I run a large Hadoop cluster on EC2 instances with IAM Role, it fails > getting the instance profile credentials, eventually all jobs on the cluster > fail. Since a number of S3A clients (all mappers and reducers) try to get the > credentials, the AWS credential endpoint starts responding 5xx and 4xx error > codes. > SharedInstanceProfileCredentialsProvider.java is sort of trying to solve it, > but it still does not share the credentials with other EC2 nodes / JVM > processes. > This issue prevents users from creating Hadoop clusters on EC2 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13689) Do not attach javadoc and sources jars during non-dist build
[ https://issues.apache.org/jira/browse/HADOOP-13689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969721#comment-15969721 ] eddie luo commented on HADOOP-13689: Hello: If you run "mvn clean install -DskipTests -Pdist -Dtar -Psrc -Dmaven.javadoc.skip" then hadoop-common jar is not under ./hadoop-2.8.1-SNAPSHOT/share/hadoop/common/ > Do not attach javadoc and sources jars during non-dist build > > > Key: HADOOP-13689 > URL: https://issues.apache.org/jira/browse/HADOOP-13689 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.8.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HADOOP-13689.001.patch > > > Looking at maven output when running with "-Pdist", the source plugin > "test-jar" and "jar" goals are invoked twice. This is because it's turned on > by both the dist profile and on by default. > Outside of the release context, it's not that important to have javadoc and > source JARs, so I think we can turn it off by default. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969707#comment-15969707 ] Hudson commented on HADOOP-14311: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11588 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11588/]) HADOOP-14311. Add python2.7-dev to Dockerfile (Allen Wittenauer via (arun suresh: rev 0ac17dc644c0429ff8a6f743bf9d3ecdd7458e58) * (edit) dev-support/docker/Dockerfile > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-14311.00.patch > > > Some changes upstream in either Ubuntu or Python now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14311: - Fix Version/s: 3.0.0-alpha3 2.8.1 2.9.0 > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-14311.00.patch > > > Some changes upstream in either Ubuntu or Python now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-14311: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed this to trunk, branch-2 and branch-2.8 > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: HADOOP-14311.00.patch > > > Some changes upstream in either Ubuntu or Python now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969686#comment-15969686 ] Arun Suresh commented on HADOOP-14311: -- Verified that this works by including the fix in the latest YARN-6406 patch. Thanks [~aw] > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14311.00.patch > > > Some changes upstream in either Ubuntu or Python now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969682#comment-15969682 ] Chris Douglas commented on HADOOP-14311: +1 > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14311.00.patch > > > Some changes upstream in either Ubuntu or Python now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14311) Add python2.7-dev to Dockerfile
[
https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969635#comment-15969635
]
Hadoop QA commented on HADOOP-14311:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
29s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 2m
39s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
11s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HADOOP-14311 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863513/HADOOP-14311.00.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux 36200938d533 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11
16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4168805 |
| shellcheck | v0.4.6 |
| modules | C: U: |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12107/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add python2.7-dev to Dockerfile
> ---
>
> Key: HADOOP-14311
> URL: https://issues.apache.org/jira/browse/HADOOP-14311
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha3
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14311.00.patch
>
>
> Some changes upstream in either Ubuntu or Python now require us to do this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14309) PowerShellFencer
[
https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969626#comment-15969626
]
Hadoop QA commented on HADOOP-14309:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
38s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 4s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Found reliance on default encoding in
org.apache.hadoop.ha.PowerShellFencer.buildPSScript(String, String):in
org.apache.hadoop.ha.PowerShellFencer.buildPSScript(String, String): new
java.io.FileWriter(File) At PowerShellFencer.java:[line 109] |
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HADOOP-14309 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863504/HADOOP-14309-000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3145e4a5d21a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a41f8dd |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12105/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12105/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12105/testReport/ |
| modules | C:
[jira] [Updated] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14311: -- Description: Some changes upstream in either Ubuntu or Python now require us to do this. (was: Some changes upstream now require us to do this.) > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14311.00.patch > > > Some changes upstream in either Ubuntu or Python now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14311) Add python2.7-dev to Dockerfile
[
https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969623#comment-15969623
]
Hadoop QA commented on HADOOP-14311:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 2m
51s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
11s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HADOOP-14311 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863513/HADOOP-14311.00.patch
|
| Optional Tests | asflicense shellcheck shelldocs |
| uname | Linux d31bd4878719 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4168805 |
| shellcheck | v0.4.6 |
| modules | C: U: |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12106/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add python2.7-dev to Dockerfile
> ---
>
> Key: HADOOP-14311
> URL: https://issues.apache.org/jira/browse/HADOOP-14311
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha3
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: HADOOP-14311.00.patch
>
>
> Some changes upstream now require us to do this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969605#comment-15969605 ] Hadoop QA commented on HADOOP-14311: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/12107/console in case of problems. > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14311.00.patch > > > Some changes upstream now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969604#comment-15969604 ] Allen Wittenauer commented on HADOOP-14311: --- (Note: I kicked this off manually, so there will likely be double hadoop qa runs) > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14311.00.patch > > > Some changes upstream now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969601#comment-15969601 ] Hadoop QA commented on HADOOP-14311: (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HADOOP-Build/12106/console in case of problems. > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14311.00.patch > > > Some changes upstream now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.
[ https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969596#comment-15969596 ] Ai Deng commented on HADOOP-13453: -- Ok, will test the path05 when I back from holiday. Happy easter everyone! > S3Guard: Instrument new functionality with Hadoop metrics. > -- > > Key: HADOOP-13453 > URL: https://issues.apache.org/jira/browse/HADOOP-13453 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Chris Nauroth >Assignee: Ai Deng > Attachments: HADOOP-13453-HADOOP-13345-001.patch, > HADOOP-13453-HADOOP-13345-002.patch, HADOOP-13453-HADOOP-13345-003.patch, > HADOOP-13453-HADOOP-13345-004.patch, HADOOP-13453-HADOOP-13345-005.patch > > > Provide Hadoop metrics showing operational details of the S3Guard > implementation. > The metrics will be implemented in this ticket: > ● S3GuardRechecksNthPercentileLatency (MutableQuantiles) Percentile time > spent > in rechecks attempting to achieve consistency. Repeated for multiple > percentile values > of N. This metric is an indicator of the additional latency cost of running > S3A with > S3Guard. > ● S3GuardRechecksNumOps (MutableQuantiles) Number of times a consistency > recheck was required while attempting to achieve consistency. > ● S3GuardStoreNthPercentileLatency (MutableQuantiles) Percentile time > spent in > operations against the consistent store, including both write operations > during file system > mutations and read operations during file system consistency checks. Repeated > for > multiple percentile values of N. This metric is an indicator of latency to > the consistent > store implementation. > ● S3GuardConsistencyStoreNumOps (MutableQuantiles) Number of operations > against the consistent store, including both write operations during file > system mutations > and read operations during file system consistency checks. > ● S3GuardConsistencyStoreFailures (MutableCounterLong) Number of failures > during operations against the consistent store implementation. > ● S3GuardConsistencyStoreTimeouts (MutableCounterLong) Number of timeouts > during operations against the consistent store implementation. > ● S3GuardInconsistencies (MutableCounterLong) C ount of times S3Guard > failed to > achieve consistency, even after exhausting all rechecks. A high count may > indicate > unexpected outofband modification of the S3 bucket contents, such as by an > external > tool that does not make corresponding updates to the consistent store. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14311: -- Status: Patch Available (was: Open) > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14311.00.patch > > > Some changes upstream now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14311) Add python2.7-dev to Dockerfile
[ https://issues.apache.org/jira/browse/HADOOP-14311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-14311: -- Attachment: HADOOP-14311.00.patch > Add python2.7-dev to Dockerfile > --- > > Key: HADOOP-14311 > URL: https://issues.apache.org/jira/browse/HADOOP-14311 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.0.0-alpha3 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Attachments: HADOOP-14311.00.patch > > > Some changes upstream now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14311) Add python2.7-dev to Dockerfile
Allen Wittenauer created HADOOP-14311: - Summary: Add python2.7-dev to Dockerfile Key: HADOOP-14311 URL: https://issues.apache.org/jira/browse/HADOOP-14311 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0-alpha3 Reporter: Allen Wittenauer Assignee: Allen Wittenauer Some changes upstream now require us to do this. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14216) Improve Configuration XML Parsing Performance
[ https://issues.apache.org/jira/browse/HADOOP-14216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969581#comment-15969581 ] Jonathan Eagles commented on HADOOP-14216: -- Sorry guys. Have been OOO this week. Will take a look at this soon to understand this fully. The logging for a missing xinclude could be done quite easily. In previous versions there was no logging, so I have to wonder what the logic should be. xinclude with fallback simply means include the file only if it is present, fail otherwise unless a fallback is specified. The above failure seems like a missed case, but the extra logging could be helpful. What do you think? > Improve Configuration XML Parsing Performance > - > > Key: HADOOP-14216 > URL: https://issues.apache.org/jira/browse/HADOOP-14216 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HADOOP-14216.1.patch, HADOOP-14216.2-branch-2.patch, > HADOOP-14216.2.patch, HADOOP-14216.addendum.1.patch > > > JIRA is to improve XML parsing performance through reuse and a change in XML > parser (STAX) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14310) RolloverSignerSecretProvider.LOG should be @VisibleForTesting
Daniel Templeton created HADOOP-14310: - Summary: RolloverSignerSecretProvider.LOG should be @VisibleForTesting Key: HADOOP-14310 URL: https://issues.apache.org/jira/browse/HADOOP-14310 Project: Hadoop Common Issue Type: Improvement Components: security Affects Versions: 2.8.0 Reporter: Daniel Templeton Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14309) PowerShellFencer
[ https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HADOOP-14309: - Status: Patch Available (was: In Progress) > PowerShellFencer > > > Key: HADOOP-14309 > URL: https://issues.apache.org/jira/browse/HADOOP-14309 > Project: Hadoop Common > Issue Type: Improvement > Components: ha >Affects Versions: 2.8.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri >Priority: Minor > Attachments: HADOOP-14309-000.patch > > > Fencer implementation that uses PowerShell to fence a remote component. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14309) PowerShellFencer
[ https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HADOOP-14309: - Status: In Progress (was: Patch Available) > PowerShellFencer > > > Key: HADOOP-14309 > URL: https://issues.apache.org/jira/browse/HADOOP-14309 > Project: Hadoop Common > Issue Type: Improvement > Components: ha >Affects Versions: 2.8.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri >Priority: Minor > Attachments: HADOOP-14309-000.patch > > > Fencer implementation that uses PowerShell to fence a remote component. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14309) PowerShellFencer
[
https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969552#comment-15969552
]
Inigo Goiri commented on HADOOP-14309:
--
To enable the fencing mechanism for HDFS HA, one could set:
{code}
dfs.ha.fencing.methods
powershell(NameNode)
{code}
> PowerShellFencer
>
>
> Key: HADOOP-14309
> URL: https://issues.apache.org/jira/browse/HADOOP-14309
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ha
>Affects Versions: 2.8.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
>Priority: Minor
> Attachments: HADOOP-14309-000.patch
>
>
> Fencer implementation that uses PowerShell to fence a remote component.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14309) PowerShellFencer
[ https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HADOOP-14309: - Attachment: HADOOP-14309-000.patch Fencer method that uses PowerShell to remotely connect to a machine and kill the required process. > PowerShellFencer > > > Key: HADOOP-14309 > URL: https://issues.apache.org/jira/browse/HADOOP-14309 > Project: Hadoop Common > Issue Type: Improvement > Components: ha >Affects Versions: 2.8.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri >Priority: Minor > Attachments: HADOOP-14309-000.patch > > > Fencer implementation that uses PowerShell to fence a remote component. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14309) PowerShellFencer
[ https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HADOOP-14309: - Status: Patch Available (was: Open) > PowerShellFencer > > > Key: HADOOP-14309 > URL: https://issues.apache.org/jira/browse/HADOOP-14309 > Project: Hadoop Common > Issue Type: Improvement > Components: ha >Reporter: Inigo Goiri >Assignee: Inigo Goiri >Priority: Minor > Attachments: HADOOP-14309-000.patch > > > Fencer implementation that uses PowerShell to fence a remote component. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14309) PowerShellFencer
[ https://issues.apache.org/jira/browse/HADOOP-14309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HADOOP-14309: - Affects Version/s: 2.8.0 > PowerShellFencer > > > Key: HADOOP-14309 > URL: https://issues.apache.org/jira/browse/HADOOP-14309 > Project: Hadoop Common > Issue Type: Improvement > Components: ha >Affects Versions: 2.8.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri >Priority: Minor > Attachments: HADOOP-14309-000.patch > > > Fencer implementation that uses PowerShell to fence a remote component. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14309) PowerShellFencer
Inigo Goiri created HADOOP-14309: Summary: PowerShellFencer Key: HADOOP-14309 URL: https://issues.apache.org/jira/browse/HADOOP-14309 Project: Hadoop Common Issue Type: Improvement Components: ha Reporter: Inigo Goiri Assignee: Inigo Goiri Priority: Minor Fencer implementation that uses PowerShell to fence a remote component. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969481#comment-15969481
]
stack commented on HADOOP-14284:
HBase cheats. It has an unofficial pre-build step, the product of which gets
checked in so it is available build time (pre-build does stuff like generate
class files from protos, shade and patch protobufs so we can run with
pb3.Changes requiring rerun of pre-build are rare). This is messy. We are
discussing formalizing pre-build by starting an ancillary project run by the
HBase PMC. We'd freight this hbase-3rdparty w/ all of our unofficial pre-build
malarky. We also have the 'guava-problem' (and the netty-problem, etc.) and
need a soln. Current intent is that hbase-3rdparty includes shaded versions of
critical libs (guava, netty, protobuf). Mainline hbase then just includes the
hbase-3rdparty artifact This is a WIP.
That referenced Curator TN is an interesting read. Curator made an unfortunate
mistake (been there). Propagating their incomplete fix here is unfortunate (Can
we depend on a Curator that has complete 'fix'' in hadoop3 or just kick out
Curator?).
bq. Shading Guava inside hadoop-client-modules to shade all Guava unlike
hadoop-shaded-thirdparty.
Does this mean we'd have guava in hadoop-client-modules and in
hadoop-shaded-thirdparty? What you thinking [~ozawa]? Thanks.
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969466#comment-15969466
]
Steve Moist commented on HADOOP-14305:
--
{quote}
Steve Moist : have you been using this feature yet? In particular, using it
with writers to the same bucket with different policies? It may be that you
just can't do this —something the release notes will need to cover.
{quote}
I've mainly been using it during testing with a cluster and parts of a demo.
Most of it has been fairly light usage, except when I ran terragen a few times.
I haven't seen this kind of issue. During a demo I did change from SSE-S3 to
SSE-KMS to SSE-C to write to the same bucket, but I used Cloudera Manager to
push those changes to the cluster and restarted the services. So I never had
multiple threads running with different encryption types.
> S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
> ---
>
> Key: HADOOP-14305
> URL: https://issues.apache.org/jira/browse/HADOOP-14305
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14305-001.patch
>
>
> The S3a encryption tests all run in parallel (they were interfering with each
> other, apparently). This adds ~1 min to the test runs.
> They should run in serial. That they fail when this is attempted due to Bad
> Auth problems must be considered a serious problem, as is indicates issues
> related to working with SSE encryption from hadoop
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969451#comment-15969451
]
Sean Busbey commented on HADOOP-14284:
--
This is a problem of build ordering. shading doesn't happen normally until the
package phase. But the update to the source means it's needed _before then_ at
compile on the dependent modules. Maven doesn't have a notion of this, aside
from either a) manually rearranging when shading happens (which will break some
assumptions in maven) or b) being different projects rather than just different
modules.
HBase gets around this painfully; we're considering just moving to its own repo
and release cycle. maybe [~saint@gmail.com] can explain it succinctly.
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14291) S3a "No auth" message to include diagnostics
[ https://issues.apache.org/jira/browse/HADOOP-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969423#comment-15969423 ] Steve Loughran commented on HADOOP-14291: - HADOOP-14305 implies that SSE key mismatch can raise bad auth problems too. The wiki entry must mention that, the diagnostics string must include the encryption mode of the FS client. > S3a "No auth" message to include diagnostics > > > Key: HADOOP-14291 > URL: https://issues.apache.org/jira/browse/HADOOP-14291 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > There's a whole section in s3a troubleshooting because requests can get auth > failures for many reasons, including > * no credentials > * wrong credentials > * right credentials, wrong bucket > * wrong endpoint for v4 auth > * trying to use private S3 server without specifying endpoint, so AWS being > hit > * clock out > * joda time > > We can aid with debugging this by including as much as we can in in the > message and a URL To a new S3A bad auth wiki page. > Info we could include > * bucket > * fs.s3a.endpoint > * nslookup of endpoint > * Anything else relevant but not a security risk > Goal; people stand a chance of working out what is failing within a bounded > time period -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14291) S3a "No auth" message to include diagnostics
[ https://issues.apache.org/jira/browse/HADOOP-14291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969420#comment-15969420 ] Steve Loughran commented on HADOOP-14291: - + [~ehiggs] [~Thomas Demoor] > S3a "No auth" message to include diagnostics > > > Key: HADOOP-14291 > URL: https://issues.apache.org/jira/browse/HADOOP-14291 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran > > There's a whole section in s3a troubleshooting because requests can get auth > failures for many reasons, including > * no credentials > * wrong credentials > * right credentials, wrong bucket > * wrong endpoint for v4 auth > * trying to use private S3 server without specifying endpoint, so AWS being > hit > * clock out > * joda time > > We can aid with debugging this by including as much as we can in in the > message and a URL To a new S3A bad auth wiki page. > Info we could include > * bucket > * fs.s3a.endpoint > * nslookup of endpoint > * Anything else relevant but not a security risk > Goal; people stand a chance of working out what is failing within a bounded > time period -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-14305: Description: The S3a encryption tests all run in parallel (they were interfering with each other, apparently). This adds ~1 min to the test runs. They should run in serial. That they fail when this is attempted due to Bad Auth problems must be considered a serious problem, as is indicates issues related to working with SSE encryption from hadoop was: The S3a encryption tests all run in parallel (they were interfering with each other, apparently). This adds ~1 min to the test runs. it should be straighforward for them to use unique names. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14283) S3A may hang due to bug in AWS SDK 1.11.86
[ https://issues.apache.org/jira/browse/HADOOP-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969327#comment-15969327 ] Aaron Fabbri commented on HADOOP-14283: --- This has been fixed in 1.11.119 ([github issue|https://github.com/aws/aws-sdk-java/issues/1102]), so we just need to upgrade our SDK version to resolve this. > S3A may hang due to bug in AWS SDK 1.11.86 > -- > > Key: HADOOP-14283 > URL: https://issues.apache.org/jira/browse/HADOOP-14283 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.0.0-alpha2 >Reporter: Aaron Fabbri >Assignee: Aaron Fabbri >Priority: Critical > > We hit a hang bug when testing S3A with parallel renames. > I narrowed this down to the newer AWS Java SDK. It only happens under load, > and appears to be a failure to wake up a waiting thread on timeout/error. > I've created a github issue here: > https://github.com/aws/aws-sdk-java/issues/1102 > I can post a Hadoop scale test which reliably reproduces this after some > cleanup. I have posted an SDK-only test here which reproduces the issue > without Hadoop: > https://github.com/ajfabbri/awstest > I have a support ticket open and am working with Amazon on this bug so I'll > take this issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-8590) Backport HADOOP-7318 (MD5Hash factory should reset the digester it returns) to branch-1
[ https://issues.apache.org/jira/browse/HADOOP-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969303#comment-15969303 ] Andras Bokor commented on HADOOP-8590: -- Is it still a valid request? Do we still support branch-1? > Backport HADOOP-7318 (MD5Hash factory should reset the digester it returns) > to branch-1 > --- > > Key: HADOOP-8590 > URL: https://issues.apache.org/jira/browse/HADOOP-8590 > Project: Hadoop Common > Issue Type: Bug > Components: io >Affects Versions: 1.0.3 >Reporter: Todd Lipcon > > I ran into this bug on branch-1 today, it seems like we should backport it. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11794) Enable distcp to copy blocks in parallel
[ https://issues.apache.org/jira/browse/HADOOP-11794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969300#comment-15969300 ] Yongjun Zhang commented on HADOOP-11794: Thanks much [~omkarksa] and [~steve_l]. Sorry I was out for a few days. I just committed to branch-2. > Enable distcp to copy blocks in parallel > > > Key: HADOOP-11794 > URL: https://issues.apache.org/jira/browse/HADOOP-11794 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Affects Versions: 0.21.0 >Reporter: dhruba borthakur >Assignee: Yongjun Zhang > Attachments: HADOOP-11794.001.patch, HADOOP-11794.002.patch, > HADOOP-11794.003.patch, HADOOP-11794.004.patch, HADOOP-11794.005.patch, > HADOOP-11794.006.patch, HADOOP-11794.007.patch, HADOOP-11794.008.patch, > HADOOP-11794.009.patch, HADOOP-11794.010.branch2.patch, > HADOOP-11794.010.patch, MAPREDUCE-2257.patch > > > The minimum unit of work for a distcp task is a file. We have files that are > greater than 1 TB with a block size of 1 GB. If we use distcp to copy these > files, the tasks either take a long long long time or finally fails. A better > way for distcp would be to copy all the source blocks in parallel, and then > stich the blocks back to files at the destination via the HDFS Concat API > (HDFS-222) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14306) TestLocalFileSystem tests have very low timeouts
[
https://issues.apache.org/jira/browse/HADOOP-14306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969136#comment-15969136
]
Hadoop QA commented on HADOOP-14306:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 56s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HADOOP-14306 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863468/HADOOP-14306.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f6c2520f7bff 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0cab572 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12104/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestLocalFileSystem tests have very low timeouts
>
>
> Key: HADOOP-14306
> URL: https://issues.apache.org/jira/browse/HADOOP-14306
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>
[jira] [Updated] (HADOOP-14306) TestLocalFileSystem tests have very low timeouts
[ https://issues.apache.org/jira/browse/HADOOP-14306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HADOOP-14306: - Attachment: HADOOP-14306.002.patch [~aw], [~ste...@apache.org], thanks for the review/advice. My initial thinking was to get rid of the specific timeouts on the tests so that they would just use the maven-wide timeout. But I agree that a test suite timeout is a better solution. Uploading a new patch that gets rid of the specific timeouts on each test, but adds a test suite timeout of 60 seconds. This is a very conservative timeout as the largest timeout of the tests beforehand was 10 seconds. So let me know if you think that it should be more aggressive. > TestLocalFileSystem tests have very low timeouts > > > Key: HADOOP-14306 > URL: https://issues.apache.org/jira/browse/HADOOP-14306 > Project: Hadoop Common > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-14306.001.patch, HADOOP-14306.002.patch > > > Most tests have a timeout of 1 second, which is much too low, especially if > there is a spinning disk involved. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968944#comment-15968944 ] Steve Loughran commented on HADOOP-13075: - For people following this, HADOOP-14305 is covering what appears to be a serious issue here: try running the encryption tests in parallel and things fail on getFileStatus() of directories. That's potentially a blocker: I don't want to ship this feature until we understand what's up and how make it go away. If its not fixable. we can discuss on that JIRA what to do next...maybe diagnostics and docs are the solution. Step 1 though: work out what's going wrong. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch, > HADOOP-13075-003.patch, HADOOP-13075-branch2.002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14306) TestLocalFileSystem tests have very low timeouts
[ https://issues.apache.org/jira/browse/HADOOP-14306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968936#comment-15968936 ] Steve Loughran commented on HADOOP-14306: - +Allen: maven itself has test run timeouts. > TestLocalFileSystem tests have very low timeouts > > > Key: HADOOP-14306 > URL: https://issues.apache.org/jira/browse/HADOOP-14306 > Project: Hadoop Common > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HADOOP-14306.001.patch > > > Most tests have a timeout of 1 second, which is much too low, especially if > there is a spinning disk involved. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14306) TestLocalFileSystem tests have very low timeouts
[
https://issues.apache.org/jira/browse/HADOOP-14306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968935#comment-15968935
]
Steve Loughran commented on HADOOP-14306:
-
-1
you've actually turned off all the tests
Also, like AW notes, hadoop tests mandate a timeout, though we nowadays use
per-suite timeout for easier maintenance. Look at
{{AbstractFSContractTestBase}} to see how to use a test rule for that.
> TestLocalFileSystem tests have very low timeouts
>
>
> Key: HADOOP-14306
> URL: https://issues.apache.org/jira/browse/HADOOP-14306
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: HADOOP-14306.001.patch
>
>
> Most tests have a timeout of 1 second, which is much too low, especially if
> there is a spinning disk involved.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14295) Authentication proxy filter may fail authorization because of getRemoteAddr
[
https://issues.apache.org/jira/browse/HADOOP-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968853#comment-15968853
]
Hadoop QA commented on HADOOP-14295:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 50s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HADOOP-14295 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863439/HADOOP-14295.004.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c914fea3013a 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0cab572 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12103/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12103/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12103/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Authentication proxy filter may fail authorization because of getRemoteAddr
> ---
>
> Key: HADOOP-14295
> URL: https://issues.apache.org/jira/browse/HADOOP-14295
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.4, 3.0.0-alpha2,
[jira] [Comment Edited] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968813#comment-15968813
]
Tsuyoshi Ozawa edited comment on HADOOP-14284 at 4/14/17 9:06 AM:
--
Compilation failure is caused by not refering maven local repository - how can
we install newly added components on test machines? Does anyone know how to do
it?
{quote}
\[ERROR\] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-auth: Compilation failure: Compilation failure:
\[ERROR\]
/testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:\[36,61\]
error: package org.apache.hadoop.shaded.com.google.common.annotations does not
exist
{quote}
was (Author: ozawa):
Compilation failure is caused by not refering maven local repository - how can
we install newly added components on test machines? Does anyone know how to do
it?
{quote}
\[ERROR\] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-auth: Compilation failure: Compilation failure:
\[ERROR\]
/testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:\[36,61\]
error: package org.apache.hadoop.shaded.com.google.common.annotations does not
exist
{quote]
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968813#comment-15968813
]
Tsuyoshi Ozawa edited comment on HADOOP-14284 at 4/14/17 9:06 AM:
--
Compilation failure is caused by not refering maven local repository - how can
we install newly added components on test machines? Does anyone know how to do
it?
{quote}
\[ERROR\] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-auth: Compilation failure: Compilation failure:
\[ERROR\]
/testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:\[36,61\]
error: package org.apache.hadoop.shaded.com.google.common.annotations does not
exist
{quote]
was (Author: ozawa):
Compilation failure is caused by not refering maven local repository - how can
we install newly added components on test machines? Does anyone know how to do
it?
{quote}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-auth: Compilation failure: Compilation failure:
[ERROR]
/testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:[36,61]
error: package org.apache.hadoop.shaded.com.google.common.annotations does not
exist
{quote]
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968813#comment-15968813
]
Tsuyoshi Ozawa commented on HADOOP-14284:
-
Compilation failure is caused by not refering maven local repository - how can
we install newly added components on test machines? Does anyone know how to do
it?
{quote}]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-auth: Compilation failure: Compilation failure:
[ERROR]
/testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:[36,61]
error: package org.apache.hadoop.shaded.com.google.common.annotations does not
exist
{quote]
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14284) Shade Guava everywhere
[
https://issues.apache.org/jira/browse/HADOOP-14284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968813#comment-15968813
]
Tsuyoshi Ozawa edited comment on HADOOP-14284 at 4/14/17 9:05 AM:
--
Compilation failure is caused by not refering maven local repository - how can
we install newly added components on test machines? Does anyone know how to do
it?
{quote}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-auth: Compilation failure: Compilation failure:
[ERROR]
/testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:[36,61]
error: package org.apache.hadoop.shaded.com.google.common.annotations does not
exist
{quote]
was (Author: ozawa):
Compilation failure is caused by not refering maven local repository - how can
we install newly added components on test machines? Does anyone know how to do
it?
{quote}]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-auth: Compilation failure: Compilation failure:
[ERROR]
/testptch/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java:[36,61]
error: package org.apache.hadoop.shaded.com.google.common.annotations does not
exist
{quote]
> Shade Guava everywhere
> --
>
> Key: HADOOP-14284
> URL: https://issues.apache.org/jira/browse/HADOOP-14284
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0-alpha3
>Reporter: Andrew Wang
>Assignee: Tsuyoshi Ozawa
>Priority: Blocker
> Attachments: HADOOP-14238.pre001.patch, HADOOP-14284.002.patch,
> HADOOP-14284.004.patch, HADOOP-14284.007.patch, HADOOP-14284.010.patch
>
>
> HADOOP-10101 upgraded the guava version for 3.x to 21.
> Guava is broadly used by Java projects that consume our artifacts.
> Unfortunately, these projects also consume our private artifacts like
> {{hadoop-hdfs}}. They also are unlikely on the new shaded client introduced
> by HADOOP-11804, currently only available in 3.0.0-alpha2.
> We should shade Guava everywhere to proactively avoid breaking downstreams.
> This isn't a requirement for all dependency upgrades, but it's necessary for
> known-bad dependencies like Guava.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14308) GraphiteSink gives up reconnecting to the graphite server
Damien Claveau created HADOOP-14308: --- Summary: GraphiteSink gives up reconnecting to the graphite server Key: HADOOP-14308 URL: https://issues.apache.org/jira/browse/HADOOP-14308 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.7.3, 2.7.2, 2.7.1, 2.7.0 Reporter: Damien Claveau Assignee: Damien Claveau Priority: Minor Since https://issues.apache.org/jira/browse/HADOOP-11400 the GraphiteSink now handles the reconnection, but it gives up silently after 5 failed attempts. As discussed here https://issues.apache.org/jira/browse/HADOOP-11400?focusedCommentId=14977496=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14977496, the sink should better throw an exception then rely on the MetricsSinkAdapter logic to retry with the configured retryDelay and retryBackoff parameters. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14295) Authentication proxy filter may fail authorization because of getRemoteAddr
[
https://issues.apache.org/jira/browse/HADOOP-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HADOOP-14295:
Attachment: HADOOP-14295.004.patch
> Authentication proxy filter may fail authorization because of getRemoteAddr
> ---
>
> Key: HADOOP-14295
> URL: https://issues.apache.org/jira/browse/HADOOP-14295
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.4, 3.0.0-alpha2, 2.8.1
>Reporter: Jeffrey E Rodriguez
>Assignee: Jeffrey E Rodriguez
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: hadoop-14295.001.patch, HADOOP-14295.002.patch,
> HADOOP-14295.003.patch, HADOOP-14295.004.patch
>
>
> When we turn on Hadoop UI Kerberos and try to access Datanode /logs the proxy
> (Knox) would get an Authorization failure and it hosts would should as
> 127.0.0.1 even though Knox wasn't in local host to Datanode, error message:
> {quote}
> "2017-04-08 07:01:23,029 ERROR security.AuthenticationWithProxyUserFilter
> (AuthenticationWithProxyUserFilter.java:getRemoteUser(94)) - Unable to verify
> proxy user: Unauthorized connection for super-user: knox from IP 127.0.0.1"
> {quote}
> We were able to figure out that Datanode have Jetty listening on localhost
> and that Netty is used to server request to DataNode, this was a measure to
> improve performance because of Netty Async NIO design.
> I propose to add a check for x-forwarded-for header since proxys usually
> inject that header before we do a getRemoteAddr
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14295) Authentication proxy filter may fail authorization because of getRemoteAddr
[
https://issues.apache.org/jira/browse/HADOOP-14295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968786#comment-15968786
]
Yuanbo Liu commented on HADOOP-14295:
-
[~jojochuang] Thanks for your review.
Attach v4 patch for this JIRA.
> Authentication proxy filter may fail authorization because of getRemoteAddr
> ---
>
> Key: HADOOP-14295
> URL: https://issues.apache.org/jira/browse/HADOOP-14295
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.7.4, 3.0.0-alpha2, 2.8.1
>Reporter: Jeffrey E Rodriguez
>Assignee: Jeffrey E Rodriguez
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: hadoop-14295.001.patch, HADOOP-14295.002.patch,
> HADOOP-14295.003.patch, HADOOP-14295.004.patch
>
>
> When we turn on Hadoop UI Kerberos and try to access Datanode /logs the proxy
> (Knox) would get an Authorization failure and it hosts would should as
> 127.0.0.1 even though Knox wasn't in local host to Datanode, error message:
> {quote}
> "2017-04-08 07:01:23,029 ERROR security.AuthenticationWithProxyUserFilter
> (AuthenticationWithProxyUserFilter.java:getRemoteUser(94)) - Unable to verify
> proxy user: Unauthorized connection for super-user: knox from IP 127.0.0.1"
> {quote}
> We were able to figure out that Datanode have Jetty listening on localhost
> and that Netty is used to server request to DataNode, this was a measure to
> improve performance because of Netty Async NIO design.
> I propose to add a check for x-forwarded-for header since proxys usually
> inject that header before we do a getRemoteAddr
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
[
https://issues.apache.org/jira/browse/HADOOP-14300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968778#comment-15968778
]
Damien Claveau commented on HADOOP-14300:
-
Hi Wei-Chiu, thank you for the precommit check.
I don't think the failed junit tests on hadoop.security.TestKDiag can be
related to the patch.
> GraphiteSink reports metrics containing undesirable whitespaces
> ---
>
> Key: HADOOP-14300
> URL: https://issues.apache.org/jira/browse/HADOOP-14300
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.7.2, 2.7.3
>Reporter: Damien Claveau
>Assignee: Damien Claveau
>Priority: Trivial
> Attachments: HADOOP-14300, HADOOP-14300.001.patch
>
>
> According to the Graphite documentation, the data sent must be in the
> following format:
> But some tag values that are flattened in the metricpath string sometimes
> include whitespaces and violate the carbon/graphite format.
> For example, the dirpath enumeration from the datanode :
> FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current,
> /data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609
> Currently, none of the aggregators available in the Graphite ecosystem can
> handle these malformed metrics. These whitespaces should be replaced by
> underscores.
> I will be happy to attach a small patch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14307) GraphiteSink can specify which tags to export into the metrics prefix
Damien Claveau created HADOOP-14307: --- Summary: GraphiteSink can specify which tags to export into the metrics prefix Key: HADOOP-14307 URL: https://issues.apache.org/jira/browse/HADOOP-14307 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 2.7.3, 2.7.2, 2.7.1, 2.7.0 Reporter: Damien Claveau Assignee: Damien Claveau Priority: Minor Fix For: 2.7.3, 2.7.2, 2.7.1, 2.7.0 This Jira is a feature proposal to add the ability in GraphiteSink to specify which Tag (Name/Value) pairs are to be flattened in the metric prefix. The motivation for this is that currently, all the tags are included in the prefix like this : graphite_prefix.Context=$context.Process=$process.Tag1=$value1.Tag2=$value2..Tag9=$value9.metric This requires a bunch of rewriting rules (and complex regexp) in the metric aggregation servers (carbon-relay, carbon-aggregator, ...) . The feature would be exactly the same as the solution implemented in the GangliaSink : https://issues.apache.org/jira/browse/HADOOP-7507 See also the commit : https://github.com/apache/hadoop/commit/2ca9c8d926a8eeb871b2868e6eb4dfb97d7dc63d -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
[
https://issues.apache.org/jira/browse/HADOOP-14300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968743#comment-15968743
]
Hadoop QA commented on HADOOP-14300:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 60 new + 230 unchanged - 26 fixed = 290 total (was 256) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 48s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.security.TestKDiag |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | HADOOP-14300 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863429/HADOOP-14300.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0468dc69fa6f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0cab572 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> GraphiteSink reports metrics containing undesirable whitespaces
> ---
>
>
[jira] [Updated] (HADOOP-14300) GraphiteSink reports metrics containing undesirable whitespaces
[
https://issues.apache.org/jira/browse/HADOOP-14300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Damien Claveau updated HADOOP-14300:
Attachment: HADOOP-14300.001.patch
New patch attached with full test coverage (and minor refactoring in the unit
test).
test-patch execution fully succeeded locally.
> GraphiteSink reports metrics containing undesirable whitespaces
> ---
>
> Key: HADOOP-14300
> URL: https://issues.apache.org/jira/browse/HADOOP-14300
> Project: Hadoop Common
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.7.2, 2.7.3
>Reporter: Damien Claveau
>Assignee: Damien Claveau
>Priority: Trivial
> Attachments: HADOOP-14300, HADOOP-14300.001.patch
>
>
> According to the Graphite documentation, the data sent must be in the
> following format:
> But some tag values that are flattened in the metricpath string sometimes
> include whitespaces and violate the carbon/graphite format.
> For example, the dirpath enumeration from the datanode :
> FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset{dirpath='[/data/1/hdfs/data/current,
> /data/2/hdfs/data/current]'}.Hostname=worker1.Capacity 78436466688 1491986609
> Currently, none of the aggregators available in the Graphite ecosystem can
> handle these malformed metrics. These whitespaces should be replaced by
> underscores.
> I will be happy to attach a small patch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
