[jira] [Commented] (HUDI-2005) Audit and remove references of fs.listStatus() and fs.getFileStatus() or fs.exists()

[sivabalan narayanan (Jira)]

    [ 
https://issues.apache.org/jira/browse/HUDI-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408907#comment-17408907
 ] 

sivabalan narayanan commented on HUDI-2005:
-------------------------------------------

{code:java}
grep -irl ".exists" hudi-*/* | grep -v Test | grep .java
nsb$ grep -irl "fs.exists" hudi-*/* | grep -v Test | grep .java
hudi-cli/src/main/java/org/apache/hudi/cli/commands/CompactionCommand.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/AbstractUpgradeDowngrade.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/marker/DirectWriteMarkers.java
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/heartbeat/HoodieHeartbeatClient.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeHandle.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkCreateHandle.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowDataCreateHandle.java
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeAndReplaceHandle.java
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/row/HoodieRowCreateHandle.java
hudi-common/src/test/java/org/apache/hudi/common/testutils/FileCreateUtils.java
hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBootstrapIndex.java
hudi-common/src/main/java/org/apache/hudi/common/util/TablePathUtils.java
hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
hudi-common/src/main/java/org/apache/hudi/common/model/HoodiePartitionMetadata.java
hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
hudi-examples/src/main/java/org/apache/hudi/examples/java/HoodieJavaWriteClientExample.java
hudi-examples/src/main/java/org/apache/hudi/examples/spark/HoodieWriteClientExample.java
hudi-flink/src/main/java/org/apache/hudi/util/StreamerUtil.java
hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/profile/WriteProfiles.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotExporter.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/HiveIncrementalPuller.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieSnapshotCopier.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/UtilHelpers.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/HDFSParquetImporter.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/BootstrapExecutor.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieMultiTableDeltaStreamer.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/S3EventsHoodieIncrSource.java
hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieCompactionAdminTool.java
{code}
 

1. HoodieWriteHandle.
{code:java}
public Path makeNewPath(String partitionPath) {
  Path path = FSUtils.getPartitionPath(config.getBasePath(), partitionPath);
  try {
    if (!fs.exists(path)) {
      fs.mkdirs(path); // create a new partition as needed.
    }
  } catch (IOException e) {
    throw new HoodieIOException("Failed to make dir " + path, e);
  }
{code}
2. HoodieLogFormatWriter. don't think we can replace this. 
{code:java}
private FSDataOutputStream getOutputStream() throws IOException, 
InterruptedException {
  if (this.output == null) {
    Path path = logFile.getPath();
    if (fs.exists(path)) {
      boolean isAppendSupported = 
StorageSchemes.isAppendSupported(fs.getScheme());
      if (isAppendSupported) {
.
.

{code}
3. FSUtils. this is invoked from 
AbstractTableFileSystemView#ensurePartitionLoadedCorrectly
{code:java}
public static void createPathIfNotExists(FileSystem fs, Path partitionPath) 
throws IOException {
  if (!fs.exists(partitionPath)) {
    fs.mkdirs(partitionPath);
  }
}
{code}
 

 

> Audit and remove references of fs.listStatus() and fs.getFileStatus() or 
> fs.exists()
> ------------------------------------------------------------------------------------
>
>                 Key: HUDI-2005
>                 URL: https://issues.apache.org/jira/browse/HUDI-2005
>             Project: Apache Hudi
>          Issue Type: Sub-task
>            Reporter: Nishith Agarwal
>            Assignee: sivabalan narayanan
>            Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)