[jira] [Resolved] (HADOOP-14497) Logs for KMS delegation token lifecycle
[ https://issues.apache.org/jira/browse/HADOOP-14497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen resolved HADOOP-14497. Resolution: Done Fix Version/s: 3.0.0-alpha4 2.9.0 All sub-tasks are done and the items mentioned in the description is complete. Specifically: #1 is improved by subtask 4 #4 is added by subtask 2 #2 and #3 already exists. So I'm closing this jira as done. Thank you for reporting, [~yzhangal], and feel free to reopen of comment if you think there's anything else we should do here! > Logs for KMS delegation token lifecycle > --- > > Key: HADOOP-14497 > URL: https://issues.apache.org/jira/browse/HADOOP-14497 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Yongjun Zhang >Assignee: Xiao Chen > Fix For: 2.9.0, 3.0.0-alpha4 > > > We run into quite some customer cases about authentication failures related > to KMS delegation token. It would be nice to see a log for each stage of the > token: > 1. creation > 2. renewal > 3. removal upon cancel > 4. remove upon expiration > So that when we correlate the logs for the same DT, we can have a good > picture about what's going on, and what could have caused the authentication > failure. > The same is applicable to other delegation tokens. > NOTE: When log info about delagation token, we don't want leak user's secret > info. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14519) Client$Connection#waitForWork may suffer spurious wakeup
John Zhuge created HADOOP-14519:
---
Summary: Client$Connection#waitForWork may suffer spurious wakeup
Key: HADOOP-14519
URL: https://issues.apache.org/jira/browse/HADOOP-14519
Project: Hadoop Common
Issue Type: Bug
Components: ipc
Affects Versions: 2.8.0
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Critical
{{Client$Connection#waitForWork}} may suffer spurious wakeup because the
{{wait}} is not surrounded by a loop. See
[https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#wait()].
{code:title=Client$Connection#waitForWork}
if (calls.isEmpty() && !shouldCloseConnection.get() && running.get()) {
long timeout = maxIdleTime-
(Time.now()-lastActivity.get());
if (timeout>0) {
try {
wait(timeout); << spurious wakeup
} catch (InterruptedException e) {}
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14518) Customize User-Agent header sent in HTTP/HTTPS requests by WASB.
Georgi Chalakov created HADOOP-14518: Summary: Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Key: HADOOP-14518 URL: https://issues.apache.org/jira/browse/HADOOP-14518 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.0.0-alpha3 Reporter: Georgi Chalakov Priority: Minor WASB passes a User-Agent header to the Azure back-end. Right now, it uses the default value set by the Azure Client SDK, so Hadoop traffic doesn't appear any different from general Blob traffic. If we customize the User-Agent header, then it will enable better troubleshooting and analysis by Azure service. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14517) Fix TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure failure
Lei (Eddy) Xu created HADOOP-14517:
--
Summary: Fix
TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure failure
Key: HADOOP-14517
URL: https://issues.apache.org/jira/browse/HADOOP-14517
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0-alpha3
Reporter: Lei (Eddy) Xu
TestDFSStripedInputStreamWithRandomECPolicy#testPreadWithDNFailure fails on
trunk:
{code}
Running org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 10.99 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
testPreadWithDNFailure(org.apache.hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy)
Time elapsed: 1.265 sec <<< FAILURE!
org.junit.internal.ArrayComparisonFailure: arrays first differed at element
[327680]; expected:<-36> but was:<2>
at
org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
at org.junit.Assert.internalArrayEquals(Assert.java:473)
at org.junit.Assert.assertArrayEquals(Assert.java:294)
at org.junit.Assert.assertArrayEquals(Assert.java:305)
at
org.apache.hadoop.hdfs.TestDFSStripedInputStream.testPreadWithDNFailure(TestDFSStripedInputStream.java:306)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14516) Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients
[ https://issues.apache.org/jira/browse/HADOOP-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu resolved HADOOP-14516. Resolution: Duplicate Closing as duplicates. Please see [HADOOP-14490] and comment there. Thanks, > Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft > Azure Storage Clients > --- > > Key: HADOOP-14516 > URL: https://issues.apache.org/jira/browse/HADOOP-14516 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.0.0-alpha3 >Reporter: Georgi Chalakov > > Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft > Azure Storage Clients. We are currently using version 4.2.0 of the SDK. > Azure Storage Clients changes between 4.2 and 5.2: > * Fixed Exists() calls on Shares and Directories to now populate metadata. > This was already being done for Files. > * Changed blob constants to support up to 256 MB on put blob for block > blobs. The default value for put blob threshold has also been updated to half > of the maximum, or 128 MB currently. > * Fixed a bug that prevented setting content MD5 to true when creating a new > file. > * Fixed a bug where access conditions, options, and operation context were > not being passed when calling openWriteExisting() on a page blob or a file. > * Fixed a bug where an exception was being thrown on a range get of a blob > or file when the options disableContentMD5Validation is set to false and > useTransactionalContentMD5 is set to true and there is no overall MD5. > * Fixed a bug where retries were happening immediately if a socket exception > was thrown. > * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. > This check has been moved to create() and uploadProperties() in > CloudFileShare. > * Prefix support for listing files and directories. > * Added support for setting public access when creating a blob container > * The public access setting on a blob container is now a container property > returned from downloadProperties. > * Add Message now modifies the PopReceipt, Id, NextVisibleTime, > InsertionTime, and ExpirationTime properties of its CloudQueueMessage > parameter. > * Populate content MD5 for range gets on Blobs and Files. > * Added support in Page Blob for incremental copy. > * Added large BlockBlob upload support. Blocks can now support sizes up to > 100 MB. > * Added a new, memory-optimized upload strategy for the upload* APIs. This > algorithm only applies for blocks greater than 4MB and when > storeBlobContentMD5 and Client-Side Encryption are disabled. > * getQualifiedUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedUri() instead. This new function will return the blob > including the snapshot (if present) and no SAS token. > * getQualifiedStorageUri() has been deprecated for Blobs. Please use > getSnapshotQualifiedStorageUri() instead. This new function will return the > blob including the snapshot (if present) and no SAS token. > * Fixed a bug where copying from a blob that included a SAS token and a > snapshot ommitted the SAS token. > * Fixed a bug in client-side encryption for tables that was preventing the > Java client from decrypting entities encrypted with the .NET client, and vice > versa. > * Added support for server-side encryption. > * Added support for getBlobReferenceFromServer methods on CloudBlobContainer > to support retrieving a blob without knowing its type. > * Fixed a bug in the retry policies where 300 status codes were being > retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14516) Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients
Georgi Chalakov created HADOOP-14516: Summary: Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients Key: HADOOP-14516 URL: https://issues.apache.org/jira/browse/HADOOP-14516 Project: Hadoop Common Issue Type: Improvement Components: fs/azure Affects Versions: 3.0.0-alpha3 Reporter: Georgi Chalakov Update WASB driver to use the latest version (5.2.0) of SDK for Microsoft Azure Storage Clients. We are currently using version 4.2.0 of the SDK. Azure Storage Clients changes between 4.2 and 5.2: * Fixed Exists() calls on Shares and Directories to now populate metadata. This was already being done for Files. * Changed blob constants to support up to 256 MB on put blob for block blobs. The default value for put blob threshold has also been updated to half of the maximum, or 128 MB currently. * Fixed a bug that prevented setting content MD5 to true when creating a new file. * Fixed a bug where access conditions, options, and operation context were not being passed when calling openWriteExisting() on a page blob or a file. * Fixed a bug where an exception was being thrown on a range get of a blob or file when the options disableContentMD5Validation is set to false and useTransactionalContentMD5 is set to true and there is no overall MD5. * Fixed a bug where retries were happening immediately if a socket exception was thrown. * In CloudFileShareProperties, setShareQuota() no longer asserts in bounds. This check has been moved to create() and uploadProperties() in CloudFileShare. * Prefix support for listing files and directories. * Added support for setting public access when creating a blob container * The public access setting on a blob container is now a container property returned from downloadProperties. * Add Message now modifies the PopReceipt, Id, NextVisibleTime, InsertionTime, and ExpirationTime properties of its CloudQueueMessage parameter. * Populate content MD5 for range gets on Blobs and Files. * Added support in Page Blob for incremental copy. * Added large BlockBlob upload support. Blocks can now support sizes up to 100 MB. * Added a new, memory-optimized upload strategy for the upload* APIs. This algorithm only applies for blocks greater than 4MB and when storeBlobContentMD5 and Client-Side Encryption are disabled. * getQualifiedUri() has been deprecated for Blobs. Please use getSnapshotQualifiedUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token. * getQualifiedStorageUri() has been deprecated for Blobs. Please use getSnapshotQualifiedStorageUri() instead. This new function will return the blob including the snapshot (if present) and no SAS token. * Fixed a bug where copying from a blob that included a SAS token and a snapshot ommitted the SAS token. * Fixed a bug in client-side encryption for tables that was preventing the Java client from decrypting entities encrypted with the .NET client, and vice versa. * Added support for server-side encryption. * Added support for getBlobReferenceFromServer methods on CloudBlobContainer to support retrieving a blob without knowing its type. * Fixed a bug in the retry policies where 300 status codes were being retried when they shouldn't be. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14515) Specifically configure org.apache.zookeeper.ClientCnxn in KMS log4j
Xiao Chen created HADOOP-14515:
--
Summary: Specifically configure org.apache.zookeeper.ClientCnxn in
KMS log4j
Key: HADOOP-14515
URL: https://issues.apache.org/jira/browse/HADOOP-14515
Project: Hadoop Common
Issue Type: Improvement
Components: kms
Affects Versions: 2.6.0
Reporter: Xiao Chen
Assignee: Xiao Chen
When investigating a case, we tried to turn on KMS DEBUG by setting the root
logger in the log4j to DEBUG. This ends up making
{{org.apache.zookeeper.ClientCnxn}} to generate 199.2M out of a 200M log file,
which made the kms.log rotate very quickly.
We should keep zookeeper's log unaffected by the root logger, and only turn it
on when interested.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/ [Jun 8, 2017 5:08:08 PM] (templedf) YARN-6679. Reduce Resource instance overhead via non-PBImpl (Contributed [Jun 8, 2017 9:15:09 PM] (jzhuge) HDFS-11851. getGlobalJNIEnv() may deadlock if exception is thrown. [Jun 8, 2017 9:36:47 PM] (liuml07) HDFS-11945. Internal lease recovery may not be retried for a long time. [Jun 8, 2017 10:06:30 PM] (liuml07) HADOOP-14283. Upgrade AWS SDK to 1.11.134. Contributed by Aaron Fabbri [Jun 9, 2017 4:34:15 AM] (xiao) HADOOP-13174. Add more debug logs for delegation tokens and [Jun 9, 2017 5:35:31 AM] (xiao) HADOOP-13854. KMS should log error details in KMSExceptionsProvider. [Jun 9, 2017 5:40:13 AM] (naganarasimha_gr) YARN-6682. Improve performance of AssignmentInformation datastructures. -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.sftp.TestSFTPFileSystem hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.mover.TestMover hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.namenode.ha.TestHASafeMode hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.yarn.sls.nodemanager.TestNMSimulator Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-mvninstall-root.txt [492K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [144K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [416K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/340/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [72K]
[jira] [Created] (HADOOP-14513) A little performance improvement of HarFileSystem
hu xiaodong created HADOOP-14513:
Summary: A little performance improvement of HarFileSystem
Key: HADOOP-14513
URL: https://issues.apache.org/jira/browse/HADOOP-14513
Project: Hadoop Common
Issue Type: Improvement
Reporter: hu xiaodong
Assignee: hu xiaodong
Priority: Trivial
In the Java source of HarFileSystem.java:
{code:title=HarFileSystem.java|borderStyle=solid}
...
...
private Path archivePath(Path p) {
Path retPath = null;
Path tmp = p;
// I think p.depth() need not be loop many times, depth() is a complex
calculation
for (int i=0; i< p.depth(); i++) {
if (tmp.toString().endsWith(".har")) {
retPath = tmp;
break;
}
tmp = tmp.getParent();
}
return retPath;
}
...
...
{code}
I think the fellow is more suitable:
{code:title=HarFileSystem.java|borderStyle=solid}
...
...
private Path archivePath(Path p) {
Path retPath = null;
Path tmp = p;
// just loop once
for (int i=0,depth=p.depth(); i< depth; i++) {
if (tmp.toString().endsWith(".har")) {
retPath = tmp;
break;
}
tmp = tmp.getParent();
}
return retPath;
}
...
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
