[
https://issues.apache.org/jira/browse/KYLIN-5177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5177:
--------------------------------
Fix Version/s: v4.0.2
(was: v4.1.0)
> Kylin4.0.1使用S3存储无法清理数据
> ----------------------
>
> Key: KYLIN-5177
> URL: https://issues.apache.org/jira/browse/KYLIN-5177
> Project: Kylin
> Issue Type: Bug
> Components: Storage - Parquet
> Affects Versions: v4.0.1
> Environment: aws emr5.33.1
> Reporter: peilun.zheng
> Assignee: mukvin
> Priority: Major
> Fix For: v4.0.2
>
>
> kylin4.0.1在aws emr5.33.1版本上使用bin/kylin.sh
> org.apache.kylin.tool.StorageCleanupJob
> 无法清理S3上的存储
> log:
> {code:java}
> Retrieving hadoop conf dir...
> Retrieving Spark dependency...
> OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support
> was removed in 8.0
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/hadoop/kylin/tool/kylin-tool-4.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/hadoop/kylin/ext/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/home/hadoop/kylin/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 2022-04-12 07:35:09,765 INFO [main] common.KylinConfig:117 : Loading
> kylin-defaults.properties from
> file:/home/hadoop/kylin/tool/kylin-tool-4.0.1.jar!/kylin-defaults.properties
> 2022-04-12 07:35:09,776 INFO [main] common.KylinConfig:369 : Use
> KYLIN_HOME=/home/hadoop/kylin
> 2022-04-12 07:35:09,819 INFO [main] common.KylinConfigBase:263 : Kylin
> Config was updated with kylin.metadata.url.identifier : kylin_instance
> 2022-04-12 07:35:09,822 INFO [main] common.KylinConfigBase:263 : Kylin
> Config was updated with kylin.log.spark-executor-properties-file :
> /home/hadoop/kylin/conf/spark-executor-log4j.properties
> 2022-04-12 07:35:09,822 INFO [main] common.KylinConfig:160 : Initialized a
> new KylinConfig from getInstanceFromEnv : 1845904670
> 2022-04-12 07:35:10,011 WARN [main] util.NativeCodeLoader:62 : Unable to
> load native-hadoop library for your platform... using builtin-java classes
> where applicable
> Exception in thread "main" java.lang.RuntimeException:
> org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme
> "s3"
> at
> org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:313)
> at
> org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:100)
> at
> org.apache.kylin.rest.job.StorageCleanupJob.<init>(StorageCleanupJob.java:121)
> at org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:26)
> Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem
> for scheme "s3"
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3273)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3293)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:120)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3344)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3312)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:310)
> ... 3 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
