nsivabalan commented on code in PR #8724:
URL: https://github.com/apache/hudi/pull/8724#discussion_r1201317230
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -310,12 +312,16 @@ private static Properties
getOrderedPropertiesWithTableChecksum(Properties props
*/
private static String storeProperties(Properties props, FSDataOutputStream
outputStream) throws IOException {
final String checksum;
- if (isValidChecksum(props)) {
+ final boolean isValidChecksum = isValidChecksum(props);
+ final String comment = String.format("Date=%s, host=%s, #properties=%d,
hudi_version=%s",
+ Instant.now(), NetworkUtils.getHostname(), isValidChecksum ?
props.size() : props.size() + 1, HoodieVersion.get());
Review Comment:
if you modified NetworkUtils.getHostname() just for this purpose, lets keep
the existing method and add a new one.
##########
hudi-client/hudi-java-client/src/main/java/org/apache/hudi/client/common/HoodieJavaEngineContext.java:
##########
@@ -161,4 +162,11 @@ public List<Integer> getCachedDataIds(HoodieDataCacheKey
cacheKey) {
public List<Integer> removeCachedDataIds(HoodieDataCacheKey cacheKey) {
return Collections.emptyList();
}
+
+ @Override
+ public Map<String, String> getInfo() {
+ final Map<String, String> info = new HashMap<>();
+ System.getProperties().stringPropertyNames().forEach(property ->
info.put(property, System.getProperty(property)));
Review Comment:
yes. we should be judicious here
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -214,4 +214,17 @@ public List<Integer>
removeCachedDataIds(HoodieDataCacheKey cacheKey) {
return removed == null ? Collections.emptyList() : removed;
}
}
+
+ @Override
+ public Map<String, String> getInfo() {
+ Map<String, String> info = new HashMap<>();
+ info.put("spark.user", javaSparkContext.sparkUser());
+ info.put("spark.master", javaSparkContext.master());
+ info.put("spark.application", javaSparkContext.appName());
+ info.put("spark.version", javaSparkContext.version());
+ info.put("spark.defaultParallelism",
String.valueOf(javaSparkContext.defaultParallelism()));
+ info.put("spark.defaultMinPartitions",
String.valueOf(javaSparkContext.defaultMinPartitions()));
+ info.put("spark.executor.instances",
String.valueOf(javaSparkContext.getConf().get("spark.executor.instances")));
+ return info;
Review Comment:
should we also let users configure any hoodie write configs to be added to
the extra metadata. since we don't serialize them anywhere, sometimes might
come in handy during investigations. by default we don't need to add any hoodie
write configs. but if user configures them, we can add them as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
