[
https://issues.apache.org/jira/browse/HIVE-26793?focusedWorklogId=841367&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-841367
]
ASF GitHub Bot logged work on HIVE-26793:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 24/Jan/23 12:12
Start Date: 24/Jan/23 12:12
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3822:
URL: https://github.com/apache/hive/pull/3822#discussion_r1085219185
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSMetricsListener.java:
##########
@@ -98,29 +100,39 @@ public void onAddPartition(AddPartitionEvent
partitionEvent) throws MetaExceptio
@Override
public void onAllocWriteId(AllocWriteIdEvent allocWriteIdEvent, Connection
dbConn, SQLGenerator sqlGenerator) throws MetaException {
if (MetastoreConf.getBoolVar(getConf(),
MetastoreConf.ConfVars.METASTORE_ACIDMETRICS_EXT_ON)) {
- Table table = getTable(allocWriteIdEvent);
- // In the case of CTAS, the table is created after write ids are
allocated, so we'll skip metrics collection.
- if (table != null &&
MetaStoreUtils.isNoAutoCompactSet(table.getParameters())) {
- int noAutoCompactSet =
-
Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
- if (noAutoCompactSet >=
- MetastoreConf.getIntVar(getConf(),
-
MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD))
{
- LOGGER.warn("There has been a write to table " + table.getDbName() +
"." + table.getTableName() +
- " where auto-compaction is disabled (tblproperties
(\"no_auto_compact\"=\"true\")).");
+ if (isNoAutoCompactSet(allocWriteIdEvent)) {
+ int numOfWritesToDisabledCompactionTable =
Metrics.getOrCreateGauge(MetricsConstants.WRITES_TO_DISABLED_COMPACTION_TABLE).incrementAndGet();
+ if (numOfWritesToDisabledCompactionTable >=
MetastoreConf.getIntVar(getConf(),
MetastoreConf.ConfVars.COMPACTOR_NUMBER_OF_DISABLED_COMPACTION_TABLES_THRESHOLD))
{
+ LOGGER.warn(
+ "There has been a write to table " +
allocWriteIdEvent.getDbName() + "." + allocWriteIdEvent.getTableName() + "
where auto-compaction is disabled \"no_auto_compact\"=\"true\".");
}
}
}
}
- private Table getTable(AllocWriteIdEvent allocWriteIdEvent) throws
MetaException {
+ private Boolean isNoAutoCompactSet(AllocWriteIdEvent allocWriteIdEvent)
throws MetaException {
String catalog = MetaStoreUtils.getDefaultCatalog(getConf());
String dbName = allocWriteIdEvent.getDbName();
String tableName = allocWriteIdEvent.getTableName();
- if (allocWriteIdEvent.getIHMSHandler() != null) {
- return allocWriteIdEvent.getIHMSHandler().getMS().getTable(catalog,
dbName, tableName);
- } else {
- return HMSHandler.getMSForConf(getConf()).getTable(catalog, dbName,
tableName);
+ Map<String, String> dbParameters;
+ Table table;
+ try {
Review Comment:
try scope is unclear, maybe:
````
private Boolean isNoAutoCompactSet(AllocWriteIdEvent allocWriteIdEvent)
throws MetaException {
String catalog = MetaStoreUtils.getDefaultCatalog(getConf());
String dbName = allocWriteIdEvent.getDbName();
String tableName = allocWriteIdEvent.getTableName();
RawStore rawStore;
if (allocWriteIdEvent.getIHMSHandler() != null) {
rawStore = allocWriteIdEvent.getIHMSHandler().getMS();
} else {
rawStore = HMSHandler.getMSForConf(getConf());
}
Map<String, String> dbParameters;
try {
dbParameters = rawStore.getDatabase(catalog, dbName).getParameters();
} catch (NoSuchObjectException e) {
LOGGER.error("Unable to find database " + dbName + ", " +
e.getMessage());
throw new MetaException(String.valueOf(e));
}
Table table = rawStore.getTable(catalog, dbName, tableName);
// In the case of CTAS, the table is created after write ids are
allocated, so we'll skip metrics collection.
if (table != null) {
return MetaStoreUtils.isNoAutoCompactSet(dbParameters,
table.getParameters());
}
return false;
}
````
Issue Time Tracking
-------------------
Worklog Id: (was: 841367)
Time Spent: 4.5h (was: 4h 20m)
> Create a new configuration to override "no compaction" for tables
> -----------------------------------------------------------------
>
> Key: HIVE-26793
> URL: https://issues.apache.org/jira/browse/HIVE-26793
> Project: Hive
> Issue Type: Improvement
> Reporter: Kokila N
> Assignee: Kokila N
> Priority: Major
> Labels: pull-request-available
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Currently a simple user can create a table with
> {color:#6a8759}no_auto_compaction=true{color} table property and create an
> aborted write transaction writing to this table. This way a malicious user
> can prevent cleaning up data for the aborted transaction, creating
> performance degradation.
> This configuration should be allowed to overridden on a database level:
> adding {color:#6a8759}no_auto_compaction=false{color} should override the
> table level setting forcing the initiator to schedule compaction for all
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
