C-Loftus commented on issue #1096: URL: https://github.com/apache/iceberg-go/issues/1096#issuecomment-4670432240
Thank you very much for your response @tanmayrauth . Sorry but I am not following a few things. Could you please clarify a few comments? > model it like the reference — Stat (exists → AlreadyExists) then create — **drop the dedicated Mkdir from the interface.** Hmmm, from what I can see in the Java source, the code uses `fs.mkdirs`. Am I misinterpreting? https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/hadoop/HadoopCatalog.java#L294 Correct me if I am wrong, but doesn't there need to be a mkdir operation in our implementation as well to ensure that the directory will get created if using the filesystem hadoop catalog? Could you clarify how you want to preserve the ability to create directories for the localfs implementation if we are dropping the `MkdirIO` interface? I get that we can abstract the checks if a directory exists by listing files by prefix but since `Stat`/`WriteFile`/etc. never creates directories, what call in localfs will perform that `Mkdir` operation? > That second point decides your option 2: with no marker to fall back on, a blob Mkdir that checks-but-creates-nothing leaves the namespace non-existent — List/Drop/Check read the prefix afterward and find nothing. Passes local tests, breaks only on the backend we can't easily test. I guess what I was trying to suggest here is to keep the Mkdir interface, that way it can be called for the localfs but just becomes a simple no-op for blob storage. I follow your comment (it makes sense that creating a directory as a no-op in S3 could lead to mistakes) but I suppose I am a bit confused how we are to handle functions like `CreateNamespace` which don't create any files only directories. Since blob stores just have lists of files with prefixes, not a notion of empty directories, I am not sure how we are supposed to model that in the same interface without keeping `mkdir` in some capacity for the localfs. Should I be creating some sort of dummy file in this path? Correct me if I am overlooking something though. Thank you very much for your guidance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
