sv2000 commented on a change in pull request #3226:
URL: https://github.com/apache/gobblin/pull/3226#discussion_r574206234
##########
File path:
gobblin-core/src/main/java/org/apache/gobblin/writer/PartitionedDataWriter.java
##########
@@ -130,7 +130,8 @@ public PartitionedDataWriter(DataWriterBuilder<S, D>
builder, final State state)
this.state.setProp(WRITER_LATEST_SCHEMA, builder.getSchema());
}
long cacheExpiryInterval =
this.state.getPropAsLong(PARTITIONED_WRITER_CACHE_TTL_SECONDS,
DEFAULT_PARTITIONED_WRITER_CACHE_TTL_SECONDS);
- this.writeTimeoutInterval = cacheExpiryInterval / 3;
+ // Increase the timeout value to make it less sensitive to HDFS slow writer
+ this.writeTimeoutInterval = cacheExpiryInterval / 3 * 2;
Review comment:
Do we want to derive this from Cache TTL value or can this be an
independent config? The reason being the right value for write timeout may vary
depending on the destination being written to. And if the config exceeds the
cache ttl, we can bound the value to the TTL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
