Thanks guys for you quick responses, Yes in my case the account is a folder and not a file, so i need to find another design that can support my variable retention per account and hive is not a viable solution, we still need to use impala as the query engine.
On Sat, Dec 15, 2018 at 2:12 PM Zoltán Borók-Nagy <borokna...@cloudera.com> wrote: > Hi, > > Yes, and the account column must be present in the data files, otherwise > Impala won't see it. > If that's the case you'll need to write a bit more complex job than a copy. > > BR, > Zoltan > > > > On Sat, Dec 15, 2018 at 12:19 PM Quanlong Huang <huangquanl...@gmail.com> > wrote: > >> Yes if those are file (not directory) names. >> >> However, if /tmp/table1/year/month/day/account=aaaa is a directory and >> your partition location is /tmp/table1/year/month/day, Impala can't read >> the underlying files recursively. There's a JIRA for support recursively >> reading: https://issues.apache.org/jira/browse/IMPALA-4596 >> >> On Sat, Dec 15, 2018 at 5:46 PM Fawze Abujaber <fawz...@gmail.com> wrote: >> >>> Thanks Quanlong for you response, I cereated a code who create this >>> partitions in order to be able to manage and define variable retention by >>> account. >>> >>> Can i conclude if i do my files structure like this, it will works for >>> me with partition by year,month and day? >>> /tmp/table1/year/month/day/account=aaaa >>> /tmp/table1/year/month/day/account=bbbb >>> >>> -- Take Care Fawze Abujaber
