There are multiple ways 1. Query the HiveMetaStore directly. 2. Use sys.* tables or better even information_schema to get the same.
On Fri, Nov 8, 2019 at 9:14 PM <m...@datameer.com> wrote: > Hi, > I have a question about how to get the location for a bunch of partitions. > My answer is: using the hive query `DESCRIBE EXTENDED <tablename> > PARTITION(<partition_name>)` > > I'm getting back a json response (if i set return type to JSON) which has > the > HDFS location in it. > > But. If I have, lets say 1000 partitions and every query needs 0.5 sec, > I have to wait 500 sec. > > So my question is, do you have a single query to gather all locations? > Or do you have a workaround to get the locations faster? > I think about to query the metadatore RDS directly. Similar to > http://www.openkb.info/2015/04/how-to-list-table-or-partition-location.html > But in an enterprise environment I'm pretty sure this approach would not be > the best because the RDS (mysql or derby) is maybe not reachable or > I don't have the permission to it. > > Any other hint or idea how to get all partition locations from lets > say external table with custom partition locations in a performant way. > > Thanks > Marko > > > -- -- Best Wishes, Ashutosh Bapat
