tandonraghav commented on issue #2151:
URL: https://github.com/apache/hudi/issues/2151#issuecomment-706067650
@bvaradar SLA ~ 15m for fresh data... and we can have around 10-30M
updates/client..But this varies from client to client.
But we want to test for 50M/client and there can be X clients like that...
So,COW is not an option.
Also, i just tried with this config on HUDI 0.6.0
````
ds
.write().format("org.apache.hudi").
options(QuickstartUtils.getQuickstartWriteConfigs()).
//option("hoodie.avro.schema",SchemaUtils.schemaMapper.get(dbName)).
option(DataSourceWriteOptions.OPERATION_OPT_KEY(),DataSourceWriteOptions.UPSERT_OPERATION_OPT_VAL()).
option(DataSourceWriteOptions.TABLE_TYPE_OPT_KEY(),DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL()).
option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY(),
"ts_ms").
//option("hoodie.combine.before.upsert",false).
option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY(),
"_id").
option(DataSourceWriteOptions.PAYLOAD_CLASS_OPT_KEY(),
MergeHudiPayload.class.getName()).
option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY(),
"db_name").
option(HoodieWriteConfig.TABLE_NAME, dbName).
**option(HoodieCompactionConfig.INLINE_COMPACT_PROP,false).**
option(DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY(),"true").
option(DataSourceWriteOptions.HIVE_TABLE_OPT_KEY(), dbName).
option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY(),"db_name").
option(DataSourceWriteOptions.DEFAULT_HIVE_ASSUME_DATE_PARTITION_OPT_VAL(),"false").
**option(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS_PROP,1).**
option(DataSourceWriteOptions.ASYNC_COMPACT_ENABLE_OPT_KEY(),false).
option(DataSourceWriteOptions.HIVE_PARTITION_EXTRACTOR_CLASS_OPT_KEY(),
MultiPartKeysValueExtractor.class.getName()).
mode(SaveMode.Append).
save(tablePath);
````
And i dont see any compactionTimeline requested (I checked inside folders).
Am i missing something?
````
-rw-r--r-- 1 raghav wheel 0B Oct 9 14:33
20201009143322.deltacommit.requested
-rw-r--r-- 1 raghav wheel 1.6K Oct 9 14:33
20201009143322.deltacommit.inflight
-rw-r--r-- 1 raghav wheel 1.9K Oct 9 14:33 20201009143322.deltacommit
-rw-r--r-- 1 raghav wheel 0B Oct 9 14:34
20201009143417.deltacommit.requested
-rw-r--r-- 1 raghav wheel 1.6K Oct 9 14:34
20201009143417.deltacommit.inflight
-rw-r--r-- 1 raghav wheel 1.9K Oct 9 14:34 20201009143417.deltacommit
````
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
