Ayush Saxena created HIVE-27447:
-----------------------------------
Summary: Iceberg: Queries failing due to FileSystem close errors
due to column statistics
Key: HIVE-27447
URL: https://issues.apache.org/jira/browse/HIVE-27447
Project: Hive
Issue Type: Bug
Reporter: Ayush Saxena
Assignee: Ayush Saxena
During fetching column statistics we are closing the filesystem in the HS2, FS
is shared and cached, so closing it ain't a good idea.
HS2 Logs show
{noformat}
2023-06-16T16:57:09,515 DEBUG
[45c1a38d-23c0-4e3a-96b7-c398bb215b2245c1a38d-23c0-4e3a-96b7-c398bb215b22
HiveServer2-Handler-Pool: Thread-114] fs.FileSystem: FileSystem.close() by
method:
org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1511));
Key: (boroknagyz (auth:SIMPLE))@hdfs://localhost:20500; URI:
hdfs://localhost:20500; Object Identity Hash: 14ec8672
2023-06-16T16:57:09,515 TRACE
[45c1a38d-23c0-4e3a-96b7-c398bb215b2245c1a38d-23c0-4e3a-96b7-c398bb215b22
HiveServer2-Handler-Pool: Thread-114] fs.FileSystem: FileSystem.close() full
stack trace:
java.lang.Throwable: null
at
org.apache.hadoop.fs.FileSystem.debugLogFileSystemClose(FileSystem.java:627)
~[hadoop-common-3.1.1.7.2.16.0-287.jar:?]
at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2599)
~[hadoop-common-3.1.1.7.2.16.0-287.jar:?]
at
org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1511)
~[hadoop-hdfs-client-3.1.1.7.2.16.0-287.jar:?]
at
org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.canProvideColStatistics(HiveIcebergStorageHandler.java:444)
~[hive-iceberg-handler-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]
at
org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:1074)
~[hive-exec-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]
at
org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:298)
~[hive-exec-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
