bhat-vinay commented on code in PR #10860:
URL: https://github.com/apache/hudi/pull/10860#discussion_r1524160977
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/command/index/TestFunctionalIndex.scala:
##########
@@ -253,6 +253,47 @@ class TestFunctionalIndex extends HoodieSparkSqlTestBase {
}
}
+ test("Test functional index update after initialization") {
+ withTempDir(tmp => {
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""create table $tableName (
+ id int,
+ name string,
+ price double,
+ ts long
+ ) using hudi
+ options (
+ primaryKey ='id',
+ type = 'mor',
+ preCombineField = 'ts',
+ hoodie.metadata.record.index.enable = 'true',
+ hoodie.datasource.write.recordkey.field = 'id'
+ )
+ partitioned by(ts)
+ location '$basePath'""".stripMargin)
+ spark.sql(s"insert into $tableName values(1, 'a1', 10, 1000)")
+ spark.sql(s"insert into $tableName values(2, 'a2', 10, 1001)")
+ spark.sql(s"insert into $tableName values(3, 'a3', 10, 1002)")
+
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts,
'yyyy-MM-dd') = '1970-01-01'")(
+ Seq(1, "a1"),
+ Seq(2, "a2"),
+ Seq(3, "a3")
+ )
+ // create functional index
+ val createIndexSql = s"create index idx_datestr on $tableName using
column_stats(ts) options(func='from_unixtime', format='yyyy-MM-dd')"
+ spark.sql(createIndexSql)
+ // do another insert after initializing the index
+ spark.sql(s"insert into $tableName values(4, 'a4', 10, 10000000)")
+ // check query result
+ checkAnswer(s"select id, name from $tableName where from_unixtime(ts,
'yyyy-MM-dd') = '1970-04-26'")(
+ Seq(4, "a4")
+ )
Review Comment:
Might also want to check that the functional-index is available in
tableConfig's metadata partition lists
(`assert(metaClient.getTableConfig.isMetadataPartitionAvailable("func_index_idx_datestr"))`)
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -737,11 +738,12 @@ public boolean isMetadataTableAvailable() {
/**
* Checks if metadata table is enabled and the specified partition has been
initialized.
*
- * @param partition The partition to check
+ * @param metadataPartitionType The metadata table partition type to check
* @returns true if the specific partition has been initialized, else
returns false.
*/
- public boolean isMetadataPartitionAvailable(MetadataPartitionType partition)
{
- return getMetadataPartitions().contains(partition.getPartitionPath());
+ public boolean isMetadataPartitionAvailable(MetadataPartitionType
metadataPartitionType) {
Review Comment:
Is this sufficient or should we actually check the whole partition-path.
Something like `public boolean isMetadataPartitionPathAvailable(String
partitionPath)`. Mostly asking for the case when there could be multiple
functional-index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
