[
https://issues.apache.org/jira/browse/CARBONDATA-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158972#comment-17158972
]
Kunal Kapoor commented on CARBONDATA-3904:
------------------------------------------
What is the warehouse location? HDFS/S3?
> insert into data got Failed to create directory path /d
> -------------------------------------------------------
>
> Key: CARBONDATA-3904
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3904
> Project: CarbonData
> Issue Type: Improvement
> Components: core
> Affects Versions: 2.0.0
> Environment: spark-2.4.5
> hadoop 2.7.3
> carbondata2.0.1
> Reporter: XiaoWen
> Priority: Minor
>
> insert data
> {code:java}
> spark.sql("INSERT OVERWRITE TABLE ods.test_table SELECT * FROM
> ods.socol_cmdinfo")
> {code}
> check logs from spark application on yarn
> $ yarn logs -applicationId application_1592787941917_4116
> found a lot this error messages
> {code:java}
> 20/07/15 16:59:45 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 16:59:45 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 16:59:51 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 16:59:51 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:00:00 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:00:35 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:02:47 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:03:36 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:03:36 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:09:55 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:09:55 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:10:05 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:11:08 ERROR FileFactory: Failed to create directory path /d
> 20/07/15 17:12:45 ERROR FileFactory: Failed to create directory path /d
> {code}
> {code:java}
> core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
> {code}
> {code:java}
> public static void createDirectoryAndSetPermission(String directoryPath,
> FsPermission permission)
> throws IOException {
> FileFactory.FileType fileType = FileFactory.getFileType(directoryPath);
> switch (fileType) {
> case S3:
> case HDFS:
> case ALLUXIO:
> case VIEWFS:
> case CUSTOM:
> case HDFS_LOCAL:
> try {
> Path path = new Path(directoryPath);
> FileSystem fs = path.getFileSystem(getConfiguration());
> if (!fs.exists(path)) {
> fs.mkdirs(path);
> fs.setPermission(path, permission);
> }
> } catch (IOException e) {
> LOGGER.error("Exception occurred : " + e.getMessage(), e);
> throw e;
> }
> return;
> case LOCAL:
> default:
> directoryPath = FileFactory.getUpdatedFilePath(directoryPath);
> File file = new File(directoryPath);
> if (!file.mkdirs()) {
> LOGGER.error(" Failed to create directory path " + directoryPath);
> } }
> }
> {code}
>
> I output the variable directoryPath and fileType
> {code:java}
> if (!file.mkdirs()) {
> // check variables
> LOGGER.info("directoryPath = [" + directoryPath + "], fileType = ["
> + fileType.toString() + "]");
> LOGGER.error(" Failed to create directory path " + directoryPath);
> }
> {code}
> add line
> LOGGER.info("directoryPath = [" + directoryPath + "], fileType = [" +
> fileType.toString() + "]");
> got echo on yarn logs
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> 2020-07-15 10:48:56 INFO directoryPath = [/d], fileType = [LOCAL]
> ....
>
> Why fileType is LOCAL?
> I have never set this value: directoryPath = [/d]?
> But the data is inserted normally and can be queried normally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
