ORC does not currently expose a primary key to the user, though we have
talked of having it do that. As Mich says the indexing on ORC is
oriented towards statistics that help the optimizer plan the query.
This can be very important in split generation (determining which parts
of the input will be read by which tasks) as well as on the fly input
pruning (deciding not to read a section of the file because the stats
show that no rows in that section will match a predicate). Either of
these can help joins. But as there is not a user visible primary key
there's no ability to rewrite the join as an index based join, which I
think is what you were asking about in your original email.
Alan.
Philip Lee <mailto:philjj...@gmail.com>
February 1, 2016 at 7:27
Also,
when making ORC from CSV,
for indexing every key on each coulmn is made, or a primary on a table
is made ?
If keys are made on each column in a table, accessing any column in
some functions like filtering should be faster.
--
==========================================================
/*Hae Joon Lee*/
Now, in Germany,
M.S. Candidate, Interested in Distributed System, Iterative Processing
Dept. of Computer Science, Informatik in German, TUB
Technical University of Berlin
In Korea,
M.S. Candidate, Computer Architecture Laboratory
Dept. of Computer Science, KAIST
Rm# 4414 CS Dept. KAIST
373-1 Guseong-dong, Yuseong-gu, Daejon, South Korea (305-701)
Mobile) 49) 015-251-448-278 in Germany, no cellular in Korea
==========================================================
Philip Lee <mailto:philjj...@gmail.com>
February 1, 2016 at 7:21
Hello,
I experiment the performance of some systems between ORC and CSV file.
I read about ORC documentation on Hive website, but still curious of
some things.
I know ORC format is faster on filtering or reading because it has
indexing.
Has it advantage of joining two tables of ORC dataset as well?
Could you explain about it in detail?
When experimenting, it seems like it has some advantages of joining in
some aspect, but not quite sure what characteristic of ORC make this
happening rather than CSV.
Best,
Phil
