HariprasadAllaka1612 edited a comment on issue #905: S3 folder paths messed up when running from Windows URL: https://github.com/apache/incubator-hudi/issues/905#issuecomment-532742261 @vinothchandar I think i confused you here. When the data is being written to S3 in Hudi format. The file structure is totally fine /gat-datalake-raw-dev/Games2/.hoodie/.temp But when trying to clean-up the .temp folder is being moved out from above mentioned directory and directly going to based path as /gat-datalake-raw-dev/Games2\.hoodie\.temp\20190918173239 My understanding is The code below public String getTempFolderPath() { return basePath + File.separator + TEMPFOLDER_NAME; } /** * Returns Marker folder path * @param instantTs Instant Timestamp * @return */ public String getMarkerFolderPath(String instantTs) { return String.format("%s%s%s", getTempFolderPath(), File.separator, instantTs); } is taking File.seperator based on the local machine not based on DFS and forming a seperator \ for windows This might be the issue i am changing it now to below code to see if it works public String getTempFolderPath() { return basePath + "/" + TEMPFOLDER_NAME; } /** * Returns Marker folder path * @param instantTs Instant Timestamp * @return */ public String getMarkerFolderPath(String instantTs) { return String.format("%s%s%s", getTempFolderPath(), "/", instantTs); }
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
