lirui-apache commented on a change in pull request #8353: [FLINK-12233][hive]
Support table related operations in HiveCatalog
URL: https://github.com/apache/flink/pull/8353#discussion_r282050516
##########
File path:
flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/GenericHiveMetastoreCatalog.java
##########
@@ -42,23 +46,35 @@
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.TableType;
-import org.apache.hadoop.hive.metastore.api.AlreadyExistsException;
import org.apache.hadoop.hive.metastore.api.Database;
-import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
import org.apache.hadoop.hive.metastore.api.Table;
-import org.apache.hadoop.hive.metastore.api.UnknownDBException;
-import org.apache.thrift.TException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.ArrayList;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
/**
* A catalog that persists all Flink streaming and batch metadata by using
Hive metastore as a persistent storage.
*/
public class GenericHiveMetastoreCatalog extends HiveCatalogBase {
private static final Logger LOG =
LoggerFactory.getLogger(GenericHiveMetastoreCatalog.class);
+ // Prefix used to distinguish properties created by Hive and Flink,
+ // as Hive metastore has its own properties created upon table creation
and migration between different versions of metastore.
+ private static final String FLINK_PROPERTY_PREFIX = "flink.";
+
+ // Flink tables should be stored as 'external' tables in Hive metastore
Review comment:
Since we don't specify a location for the external table, will hive create a
folder for it automatically? If so, I think that folder won't be deleted when
the table is dropped, and thus leaving many empty folders as we create more and
more flink tables.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
