[
https://issues.apache.org/jira/browse/HUDI-2494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
sivabalan narayanan updated HUDI-2494:
--------------------------------------
Description:
With [sync metadata patch|https://github.com/apache/hudi/pull/3590/], when
metadata is enabled by default, some spark datasource tests failed which were
using timestamp based key gen and custom key gen. Metadata table's records are
getting picked up when we do
{code:java}
spark.read.format(hudi).load(basePath + "/*/*")
{code}
For now, I have disabled metadata for these tests.
testSparkPartitonByWithTimestampBasedKeyGenerator
testSparkPartitonByWithCustomKeyGenerator
I was looking at
[options|https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html]
to ignore certain path, but looks like there is none.
was:
With [sync metadata patch|https://github.com/apache/hudi/pull/3590/], when
metadata is enabled by default, some spark datasource tests failed which were
using timestamp based key gen and custom key gen. Metadata table's records are
getting picked up when we do
{code:java}
spark.read.format(hudi).load(basePath + "/*/*")
{code}
For now, I have disabled metadata for these tests.
testSparkPartitonByWithTimestampBasedKeyGenerator
testSparkPartitonByWithCustomKeyGenerator
> Fix usage of different key generators with metadata enabled
> -----------------------------------------------------------
>
> Key: HUDI-2494
> URL: https://issues.apache.org/jira/browse/HUDI-2494
> Project: Apache Hudi
> Issue Type: Sub-task
> Components: Spark Integration
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Major
> Labels: sev:critical
> Fix For: 0.10.0
>
>
> With [sync metadata patch|https://github.com/apache/hudi/pull/3590/], when
> metadata is enabled by default, some spark datasource tests failed which were
> using timestamp based key gen and custom key gen. Metadata table's records
> are getting picked up when we do
>
> {code:java}
> spark.read.format(hudi).load(basePath + "/*/*")
> {code}
>
> For now, I have disabled metadata for these tests.
> testSparkPartitonByWithTimestampBasedKeyGenerator
> testSparkPartitonByWithCustomKeyGenerator
>
> I was looking at
> [options|https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html]
> to ignore certain path, but looks like there is none.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
