[jira] [Created] (HDFS-13623) getContentSummary to return ContentSummary without hidden files

[Miklos Szurap (JIRA)]

Miklos Szurap created HDFS-13623:


 Summary: getContentSummary to return ContentSummary without hidden 
files
 Key: HDFS-13623
 URL: https://issues.apache.org/jira/browse/HDFS-13623
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs, namenode
Affects Versions: 3.1.0
Reporter: Miklos Szurap


Improve the 
[FileSystem.getContentSummary()|http://hadoop.apache.org/docs/r3.1.0/api/org/apache/hadoop/fs/FileSystem.html#getContentSummary-org.apache.hadoop.fs.Path-]
 method to return ContentSummary object with "getFileCountWithoutHiddenFiles()" 
and "getLengthWithoutHiddenFiles()".

That two new counter should not include hidden files and hidden directories 
(and it's sub-contents).
{code:java}
public static final PathFilter HIDDEN_FILES_PATH_FILTER = new PathFilter() {
  public boolean accept(Path p) {
   String name = p.getName();
   return !name.startsWith("_") && !name.startsWith(".");
  }
};{code}
This would be especially useful for Hive: to compute table statistics with a 
single {{contentSummary}} call instead of {{globStatus}} (multiple 
{{listStatus}} calls) and iterating over multiple thousand of objects on client 
side.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10389) LoadBalancingKMSClientProvider ClassCastException on AuthenticationException

[Miklos Szurap (JIRA)]

Miklos Szurap created HDFS-10389:


 Summary: LoadBalancingKMSClientProvider ClassCastException on 
AuthenticationException
 Key: HDFS-10389
 URL: https://issues.apache.org/jira/browse/HDFS-10389
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Miklos Szurap


An Oozie job with a single shell action fails (may not be important, but if you 
needs the exact details I can provide them) with an error message coming from 
NodeManager:
{code}
2016-05-10 11:10:14,290 ERROR 
org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread 
Thread[LogAggregationService #652,5,main] threw an Exception.
java.lang.ClassCastException: 
org.apache.hadoop.security.authentication.client.AuthenticationException cannot 
be cast to java.security.GeneralSecurityException
at 
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:189)
at 
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
at 
org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1419)
at 
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1521)
at org.apache.hadoop.fs.Hdfs.createInternal(Hdfs.java:108)
at org.apache.hadoop.fs.Hdfs.createInternal(Hdfs.java:59)
at org.apache.hadoop.fs.AbstractFileSystem.create(AbstractFileSystem.java:577)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:683)
at org.apache.hadoop.fs.FileContext$3.next(FileContext.java:679)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.create(FileContext.java:679)
at 
org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogWriter$1.run(AggregatedLogFormat.java:382)
at 
org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogWriter$1.run(AggregatedLogFormat.java:377)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at 
org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogWriter.(AggregatedLogFormat.java:376)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.uploadLogsForContainers(AppLogAggregatorImpl.java:246)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.doAppLogAggregation(AppLogAggregatorImpl.java:456)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl.run(AppLogAggregatorImpl.java:421)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService$2.run(LogAggregationService.java:384)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}

The unsafe cast is here:
https://github.com/apache/hadoop/blob/2e1d0ff4e901b8313c8d71869735b94ed8bc40a0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/LoadBalancingKMSClientProvider.java#L189

Because of this ClassCastException:
- an uncaught exception is raised
- we do not see the exact "caused by" exception/message
- the oozie job fails
- YARN logs are not reported/saved



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org