[
https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=799666&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-799666
]
ASF GitHub Bot logged work on HIVE-26012:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Aug/22 09:16
Start Date: 10/Aug/22 09:16
Worklog Time Spent: 10m
Work Description: DanielZhu58 commented on code in PR #3477:
URL: https://github.com/apache/hive/pull/3477#discussion_r942220131
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -3958,12 +4003,16 @@ private Partition append_partition_common(RawStore ms,
String catName, String db
throw new AlreadyExistsException("Partition already exists:" + part);
}
- if (!wh.isDir(partLocation)) {
- if (!wh.mkdirs(partLocation)) {
- throw new MetaException(partLocation
- + " is not a directory or unable to create one");
+ if (!skipFSWrites) {
+ if (!wh.isDir(partLocation)) {
+ if (!wh.mkdirs(partLocation)) {
+ throw new MetaException(partLocation
+ + " is not a directory or unable to create one");
+ }
+ madeDir = true;
}
- madeDir = true;
+ } else {
+ LOG.warn("Because skipFSWrites is true, skip creating directories for
partitions.");
Review Comment:
Yes, we can leave this as LOG.info.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -2361,32 +2402,36 @@ private void create_table_core(final RawStore ms, final
CreateTableRequest req)
firePreEvent(new PreCreateTableEvent(tbl, db, this));
- if (!TableType.VIRTUAL_VIEW.toString().equals(tbl.getTableType())) {
- if (tbl.getSd().getLocation() == null
- || tbl.getSd().getLocation().isEmpty()) {
- tblPath = wh.getDefaultTablePath(db, tbl.getTableName() +
getTableSuffix(tbl), isExternal(tbl));
- } else {
- if (!isExternal(tbl) && !MetaStoreUtils.isNonNativeTable(tbl)) {
- LOG.warn("Location: " + tbl.getSd().getLocation()
- + " specified for non-external table:" + tbl.getTableName());
- }
- tblPath = wh.getDnsPath(new Path(tbl.getSd().getLocation()));
- // ignore suffix if it's already there (direct-write CTAS)
- if (!tblPath.getName().matches("(.*)" + SOFT_DELETE_TABLE_PATTERN)) {
- tblPath = new Path(tblPath + getTableSuffix(tbl));
+ if (!skipFSWrites) {
+ if (!TableType.VIRTUAL_VIEW.toString().equals(tbl.getTableType())) {
+ if (tbl.getSd().getLocation() == null
+ || tbl.getSd().getLocation().isEmpty()) {
+ tblPath = wh.getDefaultTablePath(db, tbl.getTableName() +
getTableSuffix(tbl), isExternal(tbl));
+ } else {
+ if (!isExternal(tbl) && !MetaStoreUtils.isNonNativeTable(tbl)) {
+ LOG.warn("Location: " + tbl.getSd().getLocation()
+ + " specified for non-external table:" +
tbl.getTableName());
+ }
+ tblPath = wh.getDnsPath(new Path(tbl.getSd().getLocation()));
+ // ignore suffix if it's already there (direct-write CTAS)
+ if (!tblPath.getName().matches("(.*)" +
SOFT_DELETE_TABLE_PATTERN)) {
+ tblPath = new Path(tblPath + getTableSuffix(tbl));
+ }
}
+ tbl.getSd().setLocation(tblPath.toString());
}
- tbl.getSd().setLocation(tblPath.toString());
- }
- if (tblPath != null) {
- if (!wh.isDir(tblPath)) {
- if (!wh.mkdirs(tblPath)) {
- throw new MetaException(tblPath
- + " is not a directory or unable to create one");
+ if (tblPath != null) {
+ if (!wh.isDir(tblPath)) {
+ if (!wh.mkdirs(tblPath)) {
+ throw new MetaException(tblPath
+ + " is not a directory or unable to create one");
+ }
+ madeDir = true;
}
- madeDir = true;
}
+ } else {
+ LOG.warn("Because skipFSWrites is true, skip the creation of
directories for tables.");
Review Comment:
Yes, we can leave this as LOG.info.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java:
##########
@@ -1399,6 +1403,42 @@ public void create_database(final Database db)
}
}
+ public void create_database_req(final CreateDatabaseRequest req)
Review Comment:
Yes. `@Override` is added.
Issue Time Tracking
-------------------
Worklog Id: (was: 799666)
Time Spent: 3.5h (was: 3h 20m)
> HMS APIs to be enhanced for metadata replication
> ------------------------------------------------
>
> Key: HIVE-26012
> URL: https://issues.apache.org/jira/browse/HIVE-26012
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Affects Versions: 3.1.0
> Reporter: Naveen Gangam
> Assignee: Hongdan Zhu
> Priority: Major
> Labels: pull-request-available
> Attachments: HMS APIs to be enhanced for metadata replication.docx
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> HMS currently has APIs like these that automatically create/delete the
> directories on the associated DFS.
> [create/drop]_database
> [create/drop]_table*
> [add/append/drop]_partition*
> This is expected and should be this way when query processors use this APIs.
> However, when tools that replicate hive metadata use this APIs on the target
> cluster, creating these dirs on target side which cause the replication of
> DFS-snapshots to fail.
> So we if provide an option to bypass this creation of dirs, dfs replications
> will be smoother. In the future we will need to restrict users that can use
> these APIs. So we will have some sort of an authorization policy.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
