[jira] [Commented] (MAPREDUCE-7307) Potential thread leak in LocatedFileStatusFetcher

[ https://issues.apache.org/jira/browse/MAPREDUCE-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237782#comment-17237782 ] Zhihua Deng commented on MAPREDUCE-7307: Thanks very much for the help and review. >

[jira] [Commented] (MAPREDUCE-7307) Potential thread leak in LocatedFileStatusFetcher

[ https://issues.apache.org/jira/browse/MAPREDUCE-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235088#comment-17235088 ] Zhihua Deng commented on MAPREDUCE-7307: [~ste...@apache.org], [~jlowe]  could you please

[jira] [Assigned] (MAPREDUCE-7307) Potential thread leak in LocatedFileStatusFetcher

[ https://issues.apache.org/jira/browse/MAPREDUCE-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng reassigned MAPREDUCE-7307: -- Assignee: Zhihua Deng > Potential thread leak in LocatedFileStatusFetcher >

[jira] [Updated] (MAPREDUCE-7307) Potential thread leak in LocatedFileStatusFetcher

[ https://issues.apache.org/jira/browse/MAPREDUCE-7307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7307: --- Component/s: job submission > Potential thread leak in LocatedFileStatusFetcher >

[jira] [Created] (MAPREDUCE-7307) Potential thread leak in LocatedFileStatusFetcher

Zhihua Deng created MAPREDUCE-7307: -- Summary: Potential thread leak in LocatedFileStatusFetcher Key: MAPREDUCE-7307 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7307 Project: Hadoop

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073229#comment-17073229 ] Zhihua Deng commented on MAPREDUCE-7241: Thanks for reviewing, [~jlowe]! > FileInputFormat

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.06.patch > FileInputFormat listStatus with less memory

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070242#comment-17070242 ] Zhihua Deng commented on MAPREDUCE-7241: Comments are added and refine unit test >

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.05.patch > FileInputFormat listStatus with less memory

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069234#comment-17069234 ] Zhihua Deng commented on MAPREDUCE-7241: Refine the code style and update the patch >

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.04.patch > FileInputFormat listStatus with less memory

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069186#comment-17069186 ] Zhihua Deng commented on MAPREDUCE-7241: Add a unit test for this patch > FileInputFormat

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.03.patch > FileInputFormat listStatus with less memory

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: (was: MAPREDUCE-7241.03.patch) > FileInputFormat listStatus with less

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.03.patch > FileInputFormat listStatus with less memory

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068493#comment-17068493 ] Zhihua Deng commented on MAPREDUCE-7241: [~ste...@apache.org], [~jlowe], [~bkarthikk] Can

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus with less memory footprint

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Summary: FileInputFormat listStatus with less memory footprint (was: FileInputFormat

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus oom caused by lots of unwanted block infos

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967366#comment-16967366 ] Zhihua Deng commented on MAPREDUCE-7241:  Would anyone have any ideas about this improvement? 

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus oom caused by lots of unwanted block infos

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937505#comment-16937505 ] Zhihua Deng commented on MAPREDUCE-7241: [~ste...@apache.org] a PR has been opened for this, 

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus oom caused by lots of unwanted block infos

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: (was: MAPREDUCE-7241.01.patch) > FileInputFormat listStatus oom caused

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus oom caused by lots of unwanted block infos

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.trunk.02.patch > FileInputFormat listStatus oom caused by

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus oom caused by lots of unwanted block infos

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Summary: FileInputFormat listStatus oom caused by lots of unwanted block infos (was:

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.trunk.patch > FileInputFormat listStatus causes oom when

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: (was: MAPREDUCE-7241.trunk.patch) > FileInputFormat listStatus causes

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933143#comment-16933143 ] Zhihua Deng commented on MAPREDUCE-7241: [~ste...@apache.org] can you take a look at the

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.trunk.patch > FileInputFormat listStatus causes oom when

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: (was: MAPREDUCE-7241.02.patch) > FileInputFormat listStatus causes oom

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.02.patch > FileInputFormat listStatus causes oom when there

[jira] [Comment Edited] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932482#comment-16932482 ] Zhihua Deng edited comment on MAPREDUCE-7241 at 9/18/19 2:10 PM: -  

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932482#comment-16932482 ] Zhihua Deng commented on MAPREDUCE-7241:  When would you do the copy? Before the returned 

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: MAPREDUCE-7241.01.patch Status: Patch Available (was: Open) >

[jira] [Commented] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931994#comment-16931994 ] Zhihua Deng commented on MAPREDUCE-7241: Changing listLocatedStatus like that is hard to

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files in HDFS

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Description: This case sometimes sees in hive when user issues queries over all

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Description: This case sometimes sees in hive when user issues queries over all

[jira] [Created] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files

Zhihua Deng created MAPREDUCE-7241: -- Summary: FileInputFormat listStatus causes oom when there are lots of files Key: MAPREDUCE-7241 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7241

[jira] [Updated] (MAPREDUCE-7241) FileInputFormat listStatus causes oom when there are lots of files

[ https://issues.apache.org/jira/browse/MAPREDUCE-7241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihua Deng updated MAPREDUCE-7241: --- Attachment: filestatus.png > FileInputFormat listStatus causes oom when there are lots