[ 
https://issues.apache.org/jira/browse/HIVE-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392428#comment-15392428
 ] 

Sergey Shelukhin commented on HIVE-12050:
-----------------------------------------

.

> change ORC split generation to use a different model
> ----------------------------------------------------
>
>                 Key: HIVE-12050
>                 URL: https://issues.apache.org/jira/browse/HIVE-12050
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Sergey Shelukhin
>
> With directory listing, ETL vs BI decision, local cache, metastore cache, 
> with PPD, file footers, and combination thereof (e.g. most splits are 
> processed via metastore PPD but some files are not cached and we need to make 
> ETL vs BI decision), some of which are blocking and some not, -I want to 
> write ORC split generation in Erlang- strategies are no longer the best model 
> to organize all the work. Some messaging or task queue based model might be 
> better where each worker that does a blocking operation (dir listing, file 
> read, metastore call, etc.) generates a list of things, and things are 
> further processed by other workers until all things are splits and there are 
> no more other things to process.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to