shengkui opened a new issue #3079:
URL: https://github.com/apache/iceberg/issues/3079
I met a exception " Mkdirs failed to create file" while using Iceberg
(v0.12.0) + Flink (v1.12.5) + hive metastore (v3.0.0) + s3a (ceph) storage.
```
Flink SQL> CREATE CATALOG hive_catalog WITH (
> 'type'='iceberg',
> 'catalog-type'='hive',
> 'uri'='thrift://172.21.92.171:32141',
> 'clients'='5',
> 'property-version'='1',
> 'warehouse'='s3a://lsk/'
> );
2021-09-07 11:11:10,425 INFO org.apache.hadoop.hive.conf.HiveConf
[] - Found configuration file null
2021-09-07 11:11:15,701 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Trying to
connect to metastore with URI thrift://172.21.92.171:32141
2021-09-07 11:11:15,751 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Opened a
connection to metastore, current connections: 1
2021-09-07 11:11:15,795 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Connected to
metastore.
[INFO] Catalog has been created.
Flink SQL> use catalog hive_catalog;
Flink SQL> show databases;
default
Flink SQL> show tables;
[INFO] Result was empty.
Flink SQL> CREATE TABLE bench (
> id INT,
> val INT,
> num INT,
> location INT)
> WITH (
> 'connector' = 'iceberg',
> 'path' = 's3a://lsk/demo',
> 'fs.s3a.access.key' = 'OC4ECY5P75I3NP14763',
> 'fs.s3a.secret.key' = 'jhgKEU9k5uKu2huQk6ycLAZBWNaVkYUFASQDk4KP',
> 'fs.s3a.endpoint' = 'http://172.21.92.176:7480',
> 'fs.s3a.aws.credentials.provider' =
'org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider',
> 'fs.s3a.impl' = 'org.apache.hadoop.fs.s3a.S3AFileSystem',
> 'fs.s3a.path.style.access' = 'true',
> 'hive.metastore.uris' = 'thrift://172.21.92.171:32141',
> 'hive.metastore.warehouse.dir' = 's3a://lsk/'
> );
[ERROR] Could not execute SQL statement. Reason:
java.io.IOException: Mkdirs failed to create
file:/user/hive/warehouse/bench/metadata (exists=false,
cwd=file:/home/shengkui.leng)
Flink SQL>
```
The log of flink sql-client:
```
2021-09-07 11:11:15,701 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Trying to
connect to metastore with URI thrift://172.21.92.171:32141
2021-09-07 11:11:15,751 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Opened a
connection to metastore, current connections: 1
2021-09-07 11:11:15,795 INFO
org.apache.hadoop.hive.metastore.HiveMetaStoreClient [] - Connected to
metastore.
2021-09-07 11:11:32,542 INFO org.apache.flink.table.catalog.CatalogManager
[] - Set the current default catalog as [hive_catalog] and the
current default database as [default].
2021-09-07 11:12:23,598 WARN org.apache.flink.table.client.cli.CliClient
[] - Could not execute SQL statement.
org.apache.flink.table.client.gateway.SqlExecutionException: Could not
execute statement: CREATE TABLE bench (
id INT,
val INT,
num INT,
location INT)
WITH (
'connector' = 'iceberg',
'path' = 's3a://lsk/demo',
'fs.s3a.access.key' = 'OC4ECY5P75I3NP14763',
'fs.s3a.secret.key' = 'jhgKEU9k5uKu2huQk6ycLAZBWNaVkYUFASQDk4KP',
'fs.s3a.endpoint' = 'http://172.21.92.176:7480',
'fs.s3a.aws.credentials.provider' =
'org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider',
'fs.s3a.impl' = 'org.apache.hadoop.fs.s3a.S3AFileSystem',
'fs.s3a.path.style.access' = 'true',
'hive.metastore.uris' = 'thrift://172.21.92.171:32141',
'hive.metastore.warehouse.dir' = 's3a://lsk/'
)
at
org.apache.flink.table.client.gateway.local.LocalExecutor.executeSql(LocalExecutor.java:317)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.cli.CliClient.callDdl(CliClient.java:739)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.cli.CliClient.callDdl(CliClient.java:734)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:330)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at java.util.Optional.ifPresent(Optional.java:159) [?:1.8.0_282]
at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:214)
[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:144)
[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:115)
[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:201)
[flink-sql-client_2.12-1.12.5.jar:1.12.5]
Caused by: org.apache.flink.table.api.TableException: Could not execute
CreateTable in path `hive_catalog`.`default`.`bench`
at
org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:796)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.catalog.CatalogManager.createTable(CatalogManager.java:632)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:776)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:666)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeSql$1(LocalExecutor.java:315)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:256)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.LocalExecutor.executeSql(LocalExecutor.java:315)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
... 8 more
Caused by: org.apache.iceberg.exceptions.RuntimeIOException: Failed to
create file:
file:/user/hive/warehouse/bench/metadata/00000-0a8000ba-e302-4f3e-ba54-7126120b8e94.metadata.json
at
org.apache.iceberg.hadoop.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:87)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:119)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:109)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:154)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:206)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:126)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:216)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.CachingCatalog$CachingTableBuilder.lambda$create$0(CachingCatalog.java:212)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2344)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[?:1.8.0_282]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2342)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2325)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.CachingCatalog$CachingTableBuilder.create(CachingCatalog.java:210)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.CachingCatalog.createTable(CachingCatalog.java:106)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.flink.FlinkCatalog.createTable(FlinkCatalog.java:379)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.flink.table.catalog.CatalogManager.lambda$createTable$10(CatalogManager.java:633)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:790)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.catalog.CatalogManager.createTable(CatalogManager.java:632)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:776)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:666)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeSql$1(LocalExecutor.java:315)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:256)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.LocalExecutor.executeSql(LocalExecutor.java:315)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
... 8 more
Caused by: java.io.IOException: Mkdirs failed to create
file:/user/hive/warehouse/bench/metadata (exists=false,
cwd=file:/home/shengkui.leng)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:458)
~[flink-s3-fs-hadoop-1.12.5.jar:1.12.5]
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:443)
~[flink-s3-fs-hadoop-1.12.5.jar:1.12.5]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1169)
~[flink-s3-fs-hadoop-1.12.5.jar:1.12.5]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1149)
~[flink-s3-fs-hadoop-1.12.5.jar:1.12.5]
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1038)
~[flink-s3-fs-hadoop-1.12.5.jar:1.12.5]
at
org.apache.iceberg.hadoop.HadoopOutputFile.createOrOverwrite(HadoopOutputFile.java:85)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.TableMetadataParser.internalWrite(TableMetadataParser.java:119)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.TableMetadataParser.overwrite(TableMetadataParser.java:109)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.BaseMetastoreTableOperations.writeNewMetadata(BaseMetastoreTableOperations.java:154)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.hive.HiveTableOperations.doCommit(HiveTableOperations.java:206)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.BaseMetastoreTableOperations.commit(BaseMetastoreTableOperations.java:126)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:216)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.CachingCatalog$CachingTableBuilder.lambda$create$0(CachingCatalog.java:212)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2344)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[?:1.8.0_282]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2342)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2325)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:108)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.shaded.com.github.benmanes.caffeine.cache.LocalManualCache.get(LocalManualCache.java:62)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.CachingCatalog$CachingTableBuilder.create(CachingCatalog.java:210)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.CachingCatalog.createTable(CachingCatalog.java:106)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.iceberg.flink.FlinkCatalog.createTable(FlinkCatalog.java:379)
~[iceberg-flink-runtime-0.12.0.jar:?]
at
org.apache.flink.table.catalog.CatalogManager.lambda$createTable$10(CatalogManager.java:633)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.catalog.CatalogManager.execute(CatalogManager.java:790)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.catalog.CatalogManager.createTable(CatalogManager.java:632)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeOperation(TableEnvironmentImpl.java:776)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:666)
~[flink-table_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$executeSql$1(LocalExecutor.java:315)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:256)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
at
org.apache.flink.table.client.gateway.local.LocalExecutor.executeSql(LocalExecutor.java:315)
~[flink-sql-client_2.12-1.12.5.jar:1.12.5]
... 8 more
```
The path of metadata in the error message is
"/user/hive/warehouse/bench/metadata", but it's not my real path. I've took a
look at the source code of iceberg, it's the default value of warehouse. I've
not found any setting about this in the document of Iceberg, have I missed
something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
