[jira] [Created] (MAPREDUCE-6876) FileInputFormat.listStatus should not fetch delegation tokens
Michael Gummelt created MAPREDUCE-6876:
--
Summary: FileInputFormat.listStatus should not fetch delegation
tokens
Key: MAPREDUCE-6876
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6876
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Michael Gummelt
{{FileInputFormat.listStatus}} fetches delegation tokens:
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L213
AFAICT, this is unnecessary. {{listStatus}} doesn't delegate those tokens to
another process. This is causing issues described in the attached Spark
Kerberos ticket, because {{TokenCache.obtainTokensForNameNodes}}, which is used
to fetch the delegation tokens, assumes that certain MapReduce configuration
variables are set, which isn't true in the Spark calling code. This is a
separate problem, but nonetheless it wouldn't have arisen if {{listStatus}}
weren't fetching delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-5641) History for failed Application Masters should be made available to the Job History Server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter reassigned MAPREDUCE-5641: Assignee: (was: Robert Kanter) > History for failed Application Masters should be made available to the Job > History Server > - > > Key: MAPREDUCE-5641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5641 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, jobhistoryserver >Affects Versions: 2.2.0 >Reporter: Robert Kanter > Attachments: MAPREDUCE-5641.patch, MAPREDUCE-5641.patch > > > Currently, the JHS has no information about jobs whose AMs have failed. This > is because the History is written by the AM to the intermediate folder just > before finishing, so when it fails for any reason, this information isn't > copied there. However, it is not lost as its in the AM's staging directory. > To make the History available in the JHS, all we need to do is have another > mechanism to move the History from the staging directory to the intermediate > directory. The AM also writes a "Summary" file before exiting normally, > which is also unavailable when the AM fails. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5907) Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967303#comment-15967303 ] Steve Loughran commented on MAPREDUCE-5907: --- seems good. don't feel bad about pinging me every week or so for a review: I know how frustrating it is for things to be left, and this patch is an example of how neglecting patch review is bad: the patch ages and the effort to pull it in increases. Alongside this work, a test for S3A can go in: created HADOOP-14302. It's a separate JIRA so this patch doesn't depend on it. [~blue_impala_48d6] might have some insights here, being as he'd benefit from this. > Improve getSplits() performance for fs implementations that can utilize > performance gains from recursive listing > > > Key: MAPREDUCE-5907 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5907 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.4.0 >Reporter: Sumit Kumar >Assignee: Sumit Kumar > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5907-2.patch, MAPREDUCE-5907-3.patch, > MAPREDUCE-5907.patch > > > FileInputFormat (both mapreduce and mapred implementations) use recursive > listing while calculating splits. They however do this by doing listing level > by level. That means to discover files in /foo/bar means they do listing at > /foo/bar first to get the immediate children, then make the same call on all > immediate children for /foo/bar to discover their immediate children and so > on. This doesn't scale well for object store based fs implementations like s3 > and swift because every listStatus call ends up being a webservice call to > backend. In cases where large number of files are considered for input, this > makes getSplits() call slow. > This patch adds a new set of recursive list apis that gives opportunity to > the fs implementations to optimize. The behavior remains the same for other > implementations (that is a default implementation is provided for other fs so > they don't have to implement anything new). However for objectstore based fs > implementations it provides a simple change to include recursive flag as true > (as shown in the patch) to improve listing performance. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
