ravjot28 opened a new issue, #20585:
URL: https://github.com/apache/hudi/issues/20585

   ### Bug Description
   
   **What happened:**
   
   `CALL archive_commits(...)` with `hoodie.embed.timeline.server=true` in its 
`options` fails while removing the leftover
   marker directories of the archived instants, and no commit is archived:
   
   ```
   org.apache.hudi.exception.HoodieCommitException: Failed to write commits
     ...
   Caused by: org.apache.hudi.exception.HoodieRemoteException: Failed to delete 
marker directory file:/.../.hoodie/.temp/<instant>
   Connect to localhost:26754 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] 
failed: Connection refused
     at 
org.apache.hudi.table.marker.TimelineServerBasedWriteMarkers.deleteMarkerDir(...)
     at 
org.apache.hudi.client.timeline.TimelineArchiverV2.deleteAnyLeftOverMarkers(...)
     at org.apache.hudi.cli.ArchiveExecutorUtils.archive(...)
     at 
org.apache.spark.sql.hudi.command.procedures.ArchiveCommitsProcedure.call(...)
   ```
   
   On a table at table version 6 the same happens through `TimelineArchiverV1` 
("Failed to archive commits").
   
   **What you expected:**
   
   The procedure archives the commits and removes the leftover marker 
directories, whatever `hoodie.embed.timeline.server`
   says in its options. The procedure never starts a timeline server, so with 
the default local file system view
   timeline-server-based markers should fall back to direct markers.
   
   **Steps to reproduce:**
   
   1. Create a copy-on-write table and write six commits.
   2. Leave a marker directory under `.hoodie/.temp/` for one of the commits 
that will be archived (or let a failed write
      leave one).
   3. Run:
      ```sql
      call archive_commits(table => 't', min_commits => 2, max_commits => 3, 
retain_commits => 1,
        enable_metadata => false, options => 
'hoodie.embed.timeline.server=true');
      ```
   
   `hoodie.write.markers.type` defaults to `TIMELINE_SERVER_BASED`, so 
`hoodie.embed.timeline.server=true` alone is
   enough. Any caller of `ArchiveExecutorUtils.archive` that passes properties 
which keep the embedded timeline server on
   (for example reused writer properties) hits the same failure.
   
   **Root cause (master d5230b69e22):**
   
   - 
`hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/cli/ArchiveExecutorUtils.java:74-76`:
 the builder
     sets `.withEmbeddedTimelineServerEnabled(false)` and then applies 
`.withProps(options)` last (added in #18437 so that
     user options win over sub-config defaults). 
`hoodie.embed.timeline.server=true` in `options` turns the server back on
     in the write config.
   - 
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/marker/WriteMarkersFactory.java:49`:
     with the default local file system view, timeline-server-based markers 
fall back to direct markers when the embedded
     timeline server is off (explicit remote-view settings can still select 
them). With it on,
     `TimelineArchiverV2.java:393-394` (`TimelineArchiverV1.java:487-488` for 
table version 6) creates
     `TimelineServerBasedWriteMarkers`, which call `localhost:26754`. 
`ArchiveExecutorUtils` creates the table and the
     archiver without a write client, so no timeline server is running.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT at d5230b69e22), 1.2.1. 1.2.0 is 
not affected (the procedure has no `options`
   argument there).
   **Query engine:** Spark 3.5
   **Relevant configs:** `hoodie.embed.timeline.server=true` in the procedure's 
`options`; `hoodie.write.markers.type` at its
   default `TIMELINE_SERVER_BASED`.
   
   ### Logs and Stack Trace
   
   See "What happened" above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to