[
https://issues.apache.org/jira/browse/KYLIN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3626:
--------------------------------
Description:
Due to the difference of HDFS and object store like S3/blob store, Kylin needs
to allow different path patterns for the files when putting to the underlying
storage.
For example, to HDFS, this pattern is good:
/kylin/metadata_name/cube_name/segment_name/file1.parquet
But, the above pattern is not good for S3, since S3 uses the first couple bytes
of the path to do the hashing, so we'd better add salt ahead of the original
pathname:
/salt/kylin/metadata_name/cube_name/segment_name/file1.parquet
or do a reverse order:
/segment_name/cube_name/metadata_name/kylin/file1.parquet
To get the flexibility, we should make this extensible.
> Allow customization for Parquet path
> ------------------------------------
>
> Key: KYLIN-3626
> URL: https://issues.apache.org/jira/browse/KYLIN-3626
> Project: Kylin
> Issue Type: Sub-task
> Reporter: Shaofeng SHI
> Priority: Major
>
> Due to the difference of HDFS and object store like S3/blob store, Kylin
> needs to allow different path patterns for the files when putting to the
> underlying storage.
>
> For example, to HDFS, this pattern is good:
> /kylin/metadata_name/cube_name/segment_name/file1.parquet
>
> But, the above pattern is not good for S3, since S3 uses the first couple
> bytes of the path to do the hashing, so we'd better add salt ahead of the
> original pathname:
> /salt/kylin/metadata_name/cube_name/segment_name/file1.parquet
> or do a reverse order:
> /segment_name/cube_name/metadata_name/kylin/file1.parquet
>
> To get the flexibility, we should make this extensible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
