rdblue commented on PR #4795:
URL: https://github.com/apache/iceberg/pull/4795#issuecomment-1138712534
@CodingCat, I don't think there is currently an option to add metadata to a
commit like that. Even if there were, I'm not sure it would help in this
situation. The problem is that Spark has no way to pass properties to SQL
commands like DELETE and MERGE. That only works with the DataFrame methods. The
approach you linked to doesn't work because that is in the SQLConf, which is at
the session level. You'd need a separate session for every writer thread. If
that's what you'd like to do, then we can help you get that feature in. Is that
what you want?
I think that we may want to introduce a different way than using write
properties to set additional metadata, because of the SQL problem. We could add
a way to pass this information, like this:
```java
public class CommitMetadata {
private static final ThreadLocal<Map<String, String>> COMMIT_PROPERTIES
= ThreadLocal.withInitial(ImmutableMap::of);
public static <R> R withCommitProperties(Map<String, String> properties,
Callable<R> callable) throws Exception {
COMMIT_PROPERTIES.set(properties);
try {
return callable.call();
} finally {
COMMIT_PROPERTIES.set(ImmutableMap.of());
}
}
static Map<String, String> commitProperties() {
return COMMIT_PROPERTIES.get();
}
}
```
Then we would update the Spark commit path to use add the properties from
that class. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
