[
https://issues.apache.org/jira/browse/KYLIN-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17250841#comment-17250841
]
ASF GitHub Bot commented on KYLIN-4818:
---------------------------------------
zzcclp commented on a change in pull request #1514:
URL: https://github.com/apache/kylin/pull/1514#discussion_r544857040
##########
File path:
kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/kylin/engine/spark/builder/CubeSnapshotBuilder.scala
##########
@@ -183,6 +185,22 @@ class CubeSnapshotBuilder extends Logging {
}
import org.apache.kylin.engine.spark.utils.SparkDataSource._
+ def checkDupKey() = {
+ val joinDescs = seg.joindescs
+ joinDescs.foreach {
+ joinDesc =>
+ val tableInfo = joinDesc.lookupTable
+ val lookupTableName = tableInfo.tableName
+ val df = ss.table(tableInfo)
+ val countColumn = df.count()
+ val lookupTablePKS = joinDesc.PKS.map(lookupTablePK =>
lookupTablePK.columnName)
+ val countDistinctColumn =
df.agg(countDistinct(joinDesc.PKS(0).columnName, lookupTablePKS:
_*)).collect().map(_(0)).toList(0)
Review comment:
better change to : `df.agg(countDistinct(lookupTablePKS(0),
lookupTablePKS.tail : _*)).collect().map(_.getLong(0)).head`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Calculate cuboid statistics in Kylin 4
> --------------------------------------
>
> Key: KYLIN-4818
> URL: https://issues.apache.org/jira/browse/KYLIN-4818
> Project: Kylin
> Issue Type: Sub-task
> Components: Spark Engine
> Reporter: Xiaoxiang Yu
> Assignee: Xiaoxiang Yu
> Priority: Major
> Fix For: v4.0.0-beta
>
>
> Refer to SparkFactDistinct.java in Kylin 3, I will try to use spark to
> calculate(estimate) rowcount/size for cuboid candidate. Rowcount/size of
> cuboid si the input for cubeplanner phase one and phase two.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
