[
https://issues.apache.org/jira/browse/KYLIN-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17928037#comment-17928037
]
Guoliang Sun commented on KYLIN-6002:
-------------------------------------
h3. Design
When deleting a model, the corresponding records in the CC <-> model mapping
table will also be deleted. If a CC is no longer associated with any model, the
orphaned CC will be removed from the CC metadata table.
Considering this as a design flaw and that the current version has not yet
entered production, we will not introduce overly complex logic to handle such
dirty data. How to handle the current dirty data? Please refer to the following
SQL. Replace `\{metadata_url}_compute_column` and
`\{metadata_url}_cc_model_relation` with the actual names of your metadata
tables, then execute the SQL to clean up.
{code:java}
delete from pfzhan_open_kylin_compute_column where uuid not in
(select distinct cc_uuid from pfzhan_open_kylin_cc_model_relation) {code}
> When recommending CC, it should check if there are reusable CCs in the CC
> table.
> --------------------------------------------------------------------------------
>
> Key: KYLIN-6002
> URL: https://issues.apache.org/jira/browse/KYLIN-6002
> Project: Kylin
> Issue Type: Bug
> Affects Versions: 5.0.0
> Reporter: Guoliang Sun
> Priority: Major
>
> Error see following
> {code:java}
> org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique
> constraint \"step_cdh632_azure_all_5_compute_column_uk4\"\n 详细：Key (project,
> table_identity, expression_md5)=(project_34948, SSB_STEP.LINEORDER,
> 4af65bfef97a50f00557d3d037a1bc63) already exists.
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:190)
> at
> org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:177)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at
> org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
> at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:59)
> at com.sun.proxy.$Proxy203.execute(Unknown Source)
> at
> org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
> at
> org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
> at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:50)
> at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
> at
> org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197)
> at
> org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184)
> at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62)
> at
> org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152)
> at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85)
> at com.sun.proxy.$Proxy213.insert(Unknown Source)
> at
> org.mybatis.dynamic.sql.util.mybatis3.MyBatis3Utils.insert(MyBatis3Utils.java:114)
> at
> org.apache.kylin.common.persistence.metadata.mapper.ComputeColumnMapper.insert(ComputeColumnMapper.java:73)
> at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
> at
> org.apache.ibatis.binding.MapperProxy$DefaultMethodInvoker.invoke(MapperProxy.java:166)
> at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85)
> at com.sun.proxy.$Proxy213.insert(Unknown Source)
> at
> org.apache.kylin.common.persistence.metadata.mapper.ComputeColumnMapper.insert(ComputeColumnMapper.java:46)
> at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
> at
> org.apache.ibatis.binding.MapperProxy$DefaultMethodInvoker.invoke(MapperProxy.java:166)
> at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85)
> at com.sun.proxy.$Proxy213.insert(Unknown Source)
> at
> org.apache.kylin.common.persistence.metadata.mapper.BasicMapper.insertOne(BasicMapper.java:144)
> at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
> at
> org.apache.ibatis.binding.MapperProxy$DefaultMethodInvoker.invoke(MapperProxy.java:166)
> at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85)
> at com.sun.proxy.$Proxy213.insertOne(Unknown Source)
> at
> org.apache.kylin.common.persistence.metadata.JdbcMetadataStore.save(JdbcMetadataStore.java:182)
> at
> org.apache.kylin.common.persistence.TransparentResourceStore.checkAndPutResource(TransparentResourceStore.java:135)
> at
> org.apache.kylin.common.persistence.TransparentResourceStore.checkAndPutResource(TransparentResourceStore.java:123)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:331)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:273)
> at org.apache.kylin.metadata.Manager.save(Manager.java:123)
> at org.apache.kylin.metadata.Manager.createAS(Manager.java:141)
> at
> org.apache.kylin.metadata.model.ComputedColumnManager.saveCCWithCheck(ComputedColumnManager.java:75)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> at
> org.apache.kylin.metadata.model.NDataModelManager.updateCcAndRelations(NDataModelManager.java:346)
> at
> org.apache.kylin.metadata.model.NDataModelManager.saveModel(NDataModelManager.java:334)
> at
> org.apache.kylin.metadata.model.NDataModelManager.updateDataModel(NDataModelManager.java:308)
> at
> org.apache.kylin.rest.service.OptRecApproveService$RecApproveContext.rewriteModel(OptRecApproveService.java:250)
> at
> org.apache.kylin.rest.service.OptRecApproveService$RecApproveContext.lambda$approveRawRecItems$1(OptRecApproveService.java:139)
> at
> org.apache.kylin.common.persistence.transaction.UnitOfWork.doInTransactionWithRetry(UnitOfWork.java:99)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:49)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:44)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:39)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:30)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:26)
> at
> org.apache.kylin.rest.service.OptRecApproveService$RecApproveContext.approveRawRecItems(OptRecApproveService.java:135)
> at
> org.apache.kylin.rest.service.OptRecApproveService.approveRecItemsToAddLayout(OptRecApproveService.java:842)
> at
> org.apache.kylin.rest.service.OptRecApproveService.approveAllRecItems(OptRecApproveService.java:811)
> at
> org.apache.kylin.rest.service.OptRecApproveService.lambda$batchApprove$2(OptRecApproveService.java:774)
> at
> org.apache.kylin.common.persistence.transaction.UnitOfWork.doTransaction(UnitOfWork.java:150)
> at
> org.apache.kylin.common.persistence.transaction.UnitOfWork.doInTransactionWithRetry(UnitOfWork.java:115)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:49)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:44)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:39)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:30)
> at
> org.apache.kylin.metadata.project.EnhancedUnitOfWork.doInTransactionWithCheckAndRetry(EnhancedUnitOfWork.java:26)
> at
> org.apache.kylin.rest.service.OptRecApproveService.batchApprove(OptRecApproveService.java:765)
> at
> org.apache.kylin.rest.controller.open.OpenModelRecController.batchApproveRecommendations(OpenModelRecController.java:97)
> at
> org.apache.kylin.rest.controller.open.OpenModelRecController$$FastClassBySpringCGLIB$$bc6647f8.invoke(<generated>)
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
