[
https://issues.apache.org/jira/browse/KYLIN-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-4298.
-----------------------------
Resolution: Fixed
> Issue with shrunken dictionary on S3
> ------------------------------------
>
> Key: KYLIN-4298
> URL: https://issues.apache.org/jira/browse/KYLIN-4298
> Project: Kylin
> Issue Type: Bug
> Affects Versions: v3.0.0-alpha2
> Reporter: Andras Istvan Nagy
> Assignee: Xiaoxiang Yu
> Priority: Major
> Fix For: v3.1.0
>
>
> We have run into an issue when using Kylin on S3. After moving
> {{kylin.env.hdfs-working-dir}} to S3, we got this exception:
> {code:java}
> Error: java.lang.IllegalArgumentException: Wrong FS:
> s3://kylin-XXXXX/kylin-test/hdfs-rootdir/kylin_metadata/kylin-330f6073-7123-75f6-ea28-09daab247d0a/vds_crosswalks/dictionary_shrunken/OBJECT_MOVEMENT_EVENTS_OLAP_VIEW.OBJECT_ID,
> expected: hdfs://ip-24-0-2-235.us-west-2.compute.internal:8020 at
> org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:669) at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:214)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1440)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1437)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1452)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1440) at
> org.apache.kylin.engine.mr.steps.ExtractDictionaryFromGlobalMapper.doCleanup(ExtractDictionaryFromGlobalMapper.java:142)
> at org.apache.kylin.engine.mr.KylinMapper.cleanup(KylinMapper.java:103) at
> org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:149) at
> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169){code}
> (I masked the real S3 bucket name in the above exception)
> The problem ceased after disabling the shrunken dictionary feature (setting
> {{kylin.dictionary.shrunken-from-global-enabled=false}})
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
