vinothchandar commented on code in PR #13399:
URL: https://github.com/apache/hudi/pull/13399#discussion_r2136051316
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/marker/DirectWriteMarkers.java:
##########
@@ -189,26 +188,17 @@ public Option<StoragePath>
createWithEarlyConflictDetection(String partitionPath
private Option<StoragePath> create(StoragePath markerPath, boolean
checkIfExists) {
HoodieTimer timer = HoodieTimer.start();
- StoragePath dirPath = markerPath.getParent();
- try {
- if (!storage.exists(dirPath)) {
- storage.createDirectory(dirPath); // create a new partition as needed.
Review Comment:
direct markers are written from executors.. and we need this to create the
same folder hierarchy as files being created. no?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/embedded/EmbeddedTimelineService.java:
##########
@@ -176,20 +176,20 @@ private void startServer(TimelineServiceCreator
timelineServiceCreator) throws I
this.serviceConfig = timelineServiceConfBuilder.build();
- server = timelineServiceCreator.create(context, storageConf.newInstance(),
serviceConfig, viewManager);
Review Comment:
can we still pass in a `context` ; may be make it a
`HoodieLocalEngineContext` to have thread parallelism on the driver.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/MarkerUtils.java:
##########
@@ -103,6 +103,15 @@ public static boolean
doesMarkerTypeFileExist(HoodieStorage storage, String mark
return storage.exists(new StoragePath(markerDir, MARKER_TYPE_FILENAME));
}
+ /**
+ * @param storage {@link HoodieStorage} to use.
+ * @param markerDir marker directory.
+ * @return {@code true} if the MARKERS.type file exists; {@code false}
otherwise.
+ */
+ public static boolean doesMarkerTypeFileExist(HoodieStorage storage,
StoragePath markerDir) throws IOException {
Review Comment:
can we remove/change the `string` arg method? instead of overloading
##########
hudi-common/src/main/java/org/apache/hudi/timeline/TimelineServiceClient.java:
##########
@@ -61,7 +61,7 @@ protected Response executeRequest(Request request) throws
IOException {
String url = builder.toString();
LOG.debug("Sending request : ({})", url);
org.apache.http.client.fluent.Response response = get(request.getMethod(),
url, timeoutMs);
- return new Response(response.returnContent().asString());
+ return new Response(response.returnContent().asStream());
Review Comment:
are these other places
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerDirState.java:
##########
@@ -94,12 +92,11 @@ public class MarkerDirState implements Serializable {
public MarkerDirState(String markerDirPath, int markerBatchNumThreads,
Option<TimelineServerBasedDetectionStrategy>
conflictDetectionStrategy,
- HoodieStorage storage, Registry metricsRegistry,
- HoodieEngineContext hoodieEngineContext, int
parallelism) {
- this.markerDirPath = markerDirPath;
+ HoodieStorage storage, Registry metricsRegistry, int
parallelism) {
+ this.markerDirPath = new StoragePath(markerDirPath);
this.storage = storage;
this.metricsRegistry = metricsRegistry;
- this.hoodieEngineContext = hoodieEngineContext;
+ this.hoodieEngineContext = new HoodieLocalEngineContext(storage.getConf());
Review Comment:
okay, you are using a local context explicitly here. which addresses my
parallelism concern
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/MarkerHandler.java:
##########
@@ -274,7 +269,7 @@ private CompletableFuture<String>
finishCreateMarkerFuture(Context context, Stri
MarkerCreationFuture future = new MarkerCreationFuture(context, markerDir,
markerName);
try {
future.complete(jsonifyResult(
- future.getContext(), future.isSuccessful(), metricsRegistry, new
ObjectMapper(), LOG));
+ future.getContext(), future.isSuccessful(), metricsRegistry, LOG));
Review Comment:
nit: passing LOG is kind of odd.. should we pass a logging lambda instead..
which take a string and logs it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
