I'm looking to write SerDe records that will essentially "lay down" log files according to partitions, buckets, clusters declared in the DDL...
the strategy is that Hive can just run with this data reasonably efficiently in its queries. Does the metastore need to get updated to know how many files I've written? (currently fighting registration of a new binarysortable SerDe - and building Hive in debug, but that's for the Developers' list...) thanks! Chris ________________________________ From: Paul Yang <[email protected]> To: "[email protected]" <[email protected]> Sent: Wed, July 21, 2010 12:05:29 PM Subject: RE: Managed Vs External tables Yeah, the metastore still holds the definition for external tables. As you mentioned, for an external table, hive doesn’t delete the data when you drop the table nor renames the directory when the table is renamed. Also, external tables can’t be archived. In general, hive will not do any operations that affect the underlying files if the tables is declared external. From:Pradeep Kamath [mailto:[email protected]] Sent: Wednesday, July 21, 2010 10:10 AM To: [email protected] Subject: Managed Vs External tables Hi, I am trying to understand the differences between managed Vs external tables. From http://wiki.apache.org/hadoop/Hive/StorageHandlers#Terminology: “A managed table is one for which the definition is primarily managed in Hive's metastore, and for whose data storage Hive is responsible. An external table is one whose definition is managed in some external catalog, and whose data Hive does not own (i.e. it will not be deleted when the table is dropped).” I am a little confused by the “external table is one whose definition is managed in some external catalog” – I thought the definition for external tables is still managed by the metastore (and not an external catalog) no? I thought the only difference between managed and external tables is that the data is not dropped when you drop an external table – are there any other differences? Thanks, Pradeep
