JingsongLi commented on code in PR #6024:
URL: https://github.com/apache/paimon/pull/6024#discussion_r2258711532
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/compact/AppendTableCompactor.java:
##########
@@ -199,4 +210,41 @@ public List<Committable> prepareCommit(boolean
waitCompaction, long checkpointId
public Iterable<Future<CommitMessage>> result() {
return result;
}
+
+ private void replace(FileStoreTable newTable) throws Exception {
+ if (commitUser == null) {
+ return;
+ }
+
+ List<? extends FileStoreWrite.State<?>> states = write.checkpoint();
+ write.close();
+ write = (BaseAppendFileStoreWrite)
newTable.store().newWrite(commitUser);
+ write.restore((List) states);
+ }
+
+ protected void updateWriteWithNewSchema() {
+ // the configs to be refreshed
+ String refreshedConfigs =
+ Options.fromMap(table.options())
+
.get(FlinkConnectorOptions.SINK_WRITER_REFRESH_DETECT_OPTIONS);
+
+ if (!StringUtils.isNullOrWhitespaceOnly(refreshedConfigs)) {
+ Optional<TableSchema> lastestSchema =
table.schemaManager().latest();
+ if (lastestSchema.isPresent() && lastestSchema.get().id() >
table.schema().id()) {
+ LOG.info(
+ "table schema has changed, current schema-id:{}, try
to update write with new schema-id:{}.",
+ table.schema().id(),
+ lastestSchema.get().id());
+ try {
+ CoreOptions newCoreOptions = new
CoreOptions(lastestSchema.get().options());
+ table =
+ table.copy(
+
newCoreOptions.getSpecificOptions(refreshedConfigs.split(",")));
+ replace(table);
Review Comment:
Just inline replace method
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/compact/AppendTableCompactor.java:
##########
@@ -199,4 +210,41 @@ public List<Committable> prepareCommit(boolean
waitCompaction, long checkpointId
public Iterable<Future<CommitMessage>> result() {
return result;
}
+
+ private void replace(FileStoreTable newTable) throws Exception {
+ if (commitUser == null) {
+ return;
+ }
+
+ List<? extends FileStoreWrite.State<?>> states = write.checkpoint();
+ write.close();
+ write = (BaseAppendFileStoreWrite)
newTable.store().newWrite(commitUser);
+ write.restore((List) states);
+ }
+
+ protected void updateWriteWithNewSchema() {
Review Comment:
Can you extract a class to do `WriterRefresh`?
##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/compact/AppendTableCompactor.java:
##########
@@ -199,4 +210,41 @@ public List<Committable> prepareCommit(boolean
waitCompaction, long checkpointId
public Iterable<Future<CommitMessage>> result() {
return result;
}
+
+ private void replace(FileStoreTable newTable) throws Exception {
+ if (commitUser == null) {
+ return;
+ }
+
+ List<? extends FileStoreWrite.State<?>> states = write.checkpoint();
+ write.close();
+ write = (BaseAppendFileStoreWrite)
newTable.store().newWrite(commitUser);
+ write.restore((List) states);
+ }
+
+ protected void updateWriteWithNewSchema() {
+ // the configs to be refreshed
+ String refreshedConfigs =
+ Options.fromMap(table.options())
+
.get(FlinkConnectorOptions.SINK_WRITER_REFRESH_DETECT_OPTIONS);
+
+ if (!StringUtils.isNullOrWhitespaceOnly(refreshedConfigs)) {
+ Optional<TableSchema> lastestSchema =
table.schemaManager().latest();
+ if (lastestSchema.isPresent() && lastestSchema.get().id() >
table.schema().id()) {
+ LOG.info(
+ "table schema has changed, current schema-id:{}, try
to update write with new schema-id:{}.",
+ table.schema().id(),
+ lastestSchema.get().id());
+ try {
+ CoreOptions newCoreOptions = new
CoreOptions(lastestSchema.get().options());
+ table =
+ table.copy(
+
newCoreOptions.getSpecificOptions(refreshedConfigs.split(",")));
Review Comment:
Only refreshedConfigs changed, you can do refresh.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@paimon.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
