[
https://issues.apache.org/jira/browse/HUDI-5989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
voon updated HUDI-5989:
-----------------------
Description:
For partition filter like date column, spark internally will convert it to day
numbers after timestamp 0, but hudi lazy fetching doesn't respect this
behavior, this pr tries to fix it.
{code:java}
INFO DataSourceStrategy: Pruning directories with: isnotnull(country#80),
isnotnull(par_date#81),(country#80 = ID),(par_date#81=19415)
...
INFO AbstractTableFileSystemView: Building file system view for partition
(country=ID/par_date=19415)
{code}
This will fix the partition pruning bug introduced in:
[https://github.com/apache/hudi/pull/6680]
h3.
was:
For partition filter like date column, spark internally will convert it to day
numbers after timestamp 0, but hudi lazy fetching doesn't respect this
behavior, this pr tries to fix it.
{code:java}
INFO DataSourceStrategy: Pruning directories with: isnotnull(country#80),
isnotnull(par_date#81),(country#80 = ID),(par_date#81=19415)
INFO AbstractTableFileSystemView: Building file system view for partition
(country=ID/par_date=19415)
{code}
This will fix the partition pruning bug introduced in:
[https://github.com/apache/hudi/pull/6680]
h3.
> Fix the date convert issue when lazy fetching partition path & file slice for
> HoodieFileIndex is used
> -----------------------------------------------------------------------------------------------------
>
> Key: HUDI-5989
> URL: https://issues.apache.org/jira/browse/HUDI-5989
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: voon
> Assignee: voon
> Priority: Major
> Labels: pull-request-available
>
> For partition filter like date column, spark internally will convert it to
> day numbers after timestamp 0, but hudi lazy fetching doesn't respect this
> behavior, this pr tries to fix it.
>
> {code:java}
> INFO DataSourceStrategy: Pruning directories with: isnotnull(country#80),
> isnotnull(par_date#81),(country#80 = ID),(par_date#81=19415)
> ...
> INFO AbstractTableFileSystemView: Building file system view for partition
> (country=ID/par_date=19415)
> {code}
>
> This will fix the partition pruning bug introduced in:
> [https://github.com/apache/hudi/pull/6680]
> h3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
