[jira] [Assigned] (HIVE-16003) Blobstores should use fs.listFiles(path, recursive=true) rather than FileUtils.listStatusRecursively

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman reassigned HIVE-16003:
-

Assignee: (was: Janaki Lahorani)

> Blobstores should use fs.listFiles(path, recursive=true) rather than 
> FileUtils.listStatusRecursively
> 
>
> Key: HIVE-16003
> URL: https://issues.apache.org/jira/browse/HIVE-16003
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sahil Takiar
>Priority: Major
>
> {{FileUtils.listStatusRecursively}} can be slow on blobstores because 
> {{listStatus}} calls are applied recursively to a given directory. This can 
> be especially bad on tables with multiple levels of partitioning.
> The {{FileSystem}} API provides an optimized API called {{listFiles(path, 
> recursive)}} that can be used to invoke an optimized recursive directory 
> listing.
> The problem is that the {{listFiles(path, recursive)}} API doesn't provide a 
> option to pass in a {{PathFilter}}, while {{FileUtils.listStatusRecursively}} 
> uses a custom HIDDEN_FILES_PATH_FILTER.
> To fix this we could either:
> 1: Modify the FileSystem API to provide a {{listFiles(path, recursive, 
> PathFilter)}} method (probably the cleanest solution)
> 2: Add conditional logic so that blobstores invoke {{listFiles(path, 
> recursive)}} and the rest of the code uses the current implementation of 
> {{FileUtils.listStatusRecursively}}
> 3: Replace the implementation of {{FileUtils.listStatusRecursively}} with 
> {{listFiles(path, recursive)}} and apply the {{PathFilter}} on the results 
> (not sure what optimizations can be made if {{PathFilter}} objects are passed 
> into {{FileSystem}} methods - maybe {{PathFilter}} objects are pushed to the 
> NameNode?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-16003) Blobstores should use fs.listFiles(path, recursive=true) rather than FileUtils.listStatusRecursively

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani reassigned HIVE-16003:
--

Assignee: Janaki Lahorani  (was: Sahil Takiar)

> Blobstores should use fs.listFiles(path, recursive=true) rather than 
> FileUtils.listStatusRecursively
> 
>
> Key: HIVE-16003
> URL: https://issues.apache.org/jira/browse/HIVE-16003
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>
> {{FileUtils.listStatusRecursively}} can be slow on blobstores because 
> {{listStatus}} calls are applied recursively to a given directory. This can 
> be especially bad on tables with multiple levels of partitioning.
> The {{FileSystem}} API provides an optimized API called {{listFiles(path, 
> recursive)}} that can be used to invoke an optimized recursive directory 
> listing.
> The problem is that the {{listFiles(path, recursive)}} API doesn't provide a 
> option to pass in a {{PathFilter}}, while {{FileUtils.listStatusRecursively}} 
> uses a custom HIDDEN_FILES_PATH_FILTER.
> To fix this we could either:
> 1: Modify the FileSystem API to provide a {{listFiles(path, recursive, 
> PathFilter)}} method (probably the cleanest solution)
> 2: Add conditional logic so that blobstores invoke {{listFiles(path, 
> recursive)}} and the rest of the code uses the current implementation of 
> {{FileUtils.listStatusRecursively}}
> 3: Replace the implementation of {{FileUtils.listStatusRecursively}} with 
> {{listFiles(path, recursive)}} and apply the {{PathFilter}} on the results 
> (not sure what optimizations can be made if {{PathFilter}} objects are passed 
> into {{FileSystem}} methods - maybe {{PathFilter}} objects are pushed to the 
> NameNode?)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-16003) Blobstores should use fs.listFiles(path, recursive=true) rather than FileUtils.listStatusRecursively

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar reassigned HIVE-16003:
---


> Blobstores should use fs.listFiles(path, recursive=true) rather than 
> FileUtils.listStatusRecursively
> 
>
> Key: HIVE-16003
> URL: https://issues.apache.org/jira/browse/HIVE-16003
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> {{FileUtils.listStatusRecursively}} can be slow on blobstores because 
> {{listStatus}} calls are applied recursively to a given directory. This can 
> be especially bad on tables with multiple levels of partitioning.
> The {{FileSystem}} API provides an optimized API called {{listFiles(path, 
> recursive)}} that can be used to invoke an optimized recursive directory 
> listing.
> The problem is that the {{listFiles(path, recursive)}} API doesn't provide a 
> option to pass in a {{PathFilter}}, while {{FileUtils.listStatusRecursively}} 
> uses a custom HIDDEN_FILES_PATH_FILTER.
> To fix this we could either:
> 1: Modify the FileSystem API to provide a {{listFiles(path, recursive, 
> PathFilter)}} method
> 2: Add conditional logic so that blobstores invoke {{listFiles(path, 
> recursive)}} and the rest of the code uses the current implementation of 
> {{FileUtils.listStatusRecursively}}
> 3: Replace the implementation of {{FileUtils.listStatusRecursively}} with 
> {{listFiles(path, recursive)}} and apply the {{PathFilter}} on the results



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)