This is an automated email from the ASF dual-hosted git repository.
dependabot[bot] pushed a change to branch
dependabot/pip/hudi-agent-gateway/pyarrow-23.0.1
in repository https://gitbox.apache.org/repos/asf/hudi.git
omit 461a1d6cb06f chore(deps-dev): bump pyarrow in /hudi-agent-gateway
add ba4615b864f0 fix(spark): support saveAsTable append to existing Hudi
tables (#20423)
add b0d16f94b3b5 docs(rfc): close RFC-99 with a reduced scope (#20488)
add 99ecb90bcaf5 fix(spark): reject order-breaking casts and negative
factors in data ... (#19475)
add a9dcd6807105 fix(spark): reject collation-changing casts in data
skipping (#20458)
add 316c7df2839e fix(flink): respect data skipping settings for record
index pruning (#20502)
add 5107bee3176a test(spark): add repeatable SDP phase one acceptance
scenario (#20493)
add 302040ed46ea fix(utilities): surface async table service failures in
HoodieStreamer (#20429)
add 1217eaaf11d9 fix(flink): avoid partition stats pruning for incremental
reads (#20481)
add c58c54e068b5 fix(spark): reject negative factors in data skipping
multiply/divide (#20457)
add 7d54e1410043 test(spark): verify sink cleanup and commit checkpoint
recovery (#20495)
add d42148b54638 fix(spark): tolerate metadata column reconciliation in
HoodieCatalog (#20492)
add 8e5608ecaf6e fix(spark): skip partition stats pruning for time travel
queries (#20451)
add 60f77e9cc9c4 fix(spark): rewrite time travel inside CTE definitions
(#20414)
add fff794d4b384 fix(flink): keep reporting hive sync metrics after a
coordinator restart (#20499)
add 9bf3b2f5e7d9 feat(agent): Add Flink cross path validation and agent
evaluations to Hudi Architect (#20514)
add b3318ae4cf94 test(spark): cover repeated SDP materialized view
refreshes (#20494)
add 4d22ee5ff2d3 refactor(spark): split data skipping into Shape, Leaf,
Range and Verdict stages (#20524)
add ed224fda0f06 chore(ci): make Java CI merge-queue-ready (#20473)
add 10658fb268a8 chore(ci): enable the merge queue on master (#20474)
add c4f0c4c2a80d fix(spark): make the data skipping cast allowlist
deny-by-default (#20513)
add 94677a7c1e10 fix(spark): require a fixed-offset zone for
from/to_utc_timestamp data skipping (#20519)
add 9c75d0b4273c fix(spark): guard date/time data skipping and restore
to_date pruning (#19474)
add 4cad2902ff15 fix(spark): guard date_format/from_unixtime data skipping
against DST in the session zone (#20521)
add f5fe1dce2720 feat(lakehouse): add Superset and Zeppelin Spark Connect
integration (#20467)
add 30249020e2fb fix(spark): guard data skipping through overflowing
arithmetic (#20532)
add 0b696f915b67 fix(spark): retain matching files with missing column
statistics (#20328)
add 399a3406a582 fix(spark): validate partitions and cover keyed SDP flows
(#20496)
add 140ad68dc13e fix(flink): preserve IN matches when literal list
contains NULL (#20517)
add 0a7db9598de6 test(storage-format): cover point lookups and
incompressible values in the multi-block HFile scan test (#20547)
add 4f07924bc7d1 chore(ci): drop the secret committers team from the merge
queue bypass (#20552)
add 23a122a1f645 chore(deps): bump pyarrow in
/hudi-lakehouse/local-dev/example (#20463)
add 840cba6ee617 chore(deps-dev): bump pyarrow in /hudi-agent-gateway
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (461a1d6cb06f)
\
N -- N -- N
refs/heads/dependabot/pip/hudi-agent-gateway/pyarrow-23.0.1 (840cba6ee617)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 53 +-
.github/workflows/README.md | 6 +-
.github/workflows/agent_gateway_ci.yml | 18 +
.github/workflows/bot.yml | 100 +-
.github/workflows/java_ci_engines.yml | 72 +-
.github/workflows/validate_source.yml | 10 +-
README.md | 4 +-
.../combined_safety_gates.json} | 31 +-
.../flink_pr8/cross_path_scenarios.toml | 134 ++
.../existing_multi_writer_out_of_scope.json | 14 +
.../flink_pr8/malformed_cross_path.json | 78 ++
.../tests/test_hudi_architect_cross_path.py | 460 +++++++
.../apache/hudi/async/AsyncClusteringService.java | 6 +-
.../org/apache/hudi/async/AsyncCompactService.java | 6 +-
.../org/apache/hudi/async/HoodieAsyncService.java | 27 +-
.../hudi/metadata/HoodieMetadataWriteUtils.java | 10 +-
.../partitionstats/PartitionStatsIndexer.java | 67 +-
.../hudi/async/TestAsyncTableServiceFailure.java | 142 +++
.../hudi/async/TestHoodieAsyncTableService.java | 56 +
.../metadata/TestHoodieMetadataWriteUtils.java | 70 +
.../partitionstats/TestPartitionStatsIndexer.java | 91 ++
.../spark/sql/HoodieCatalystExpressionUtils.scala | 14 -
.../spark/sql/HoodieCatalystPlansUtils.scala | 14 +-
.../hudi/metadata/HoodieTableMetadataUtil.java | 95 +-
.../hudi/metadata/TestHoodieTableMetadataUtil.java | 75 +-
.../apache/hudi/metrics/FlinkHiveSyncMetrics.java | 71 +-
.../hudi/sink/StreamWriteOperatorCoordinator.java | 3 +-
.../apache/hudi/source/ExpressionEvaluators.java | 22 +-
.../java/org/apache/hudi/source/FileIndex.java | 2 +-
.../org/apache/hudi/table/HoodieTableSource.java | 5 +-
.../java/org/apache/hudi/util/ExpressionUtils.java | 8 +-
.../hudi/metrics/TestFlinkHiveSyncMetrics.java | 59 +
.../sink/TestStreamWriteOperatorCoordinator.java | 19 +
.../hudi/source/TestExpressionEvaluators.java | 48 +-
.../java/org/apache/hudi/source/TestFileIndex.java | 50 +
.../hudi/source/TestIncrementalInputSplits.java | 10 +-
.../apache/hudi/table/ITTestHoodieDataSource.java | 47 +-
.../apache/hudi/table/TestHoodieTableSource.java | 27 +
.../org/apache/hudi/util/TestExpressionUtils.java | 2 +
.../hudi/utils/SharedCoordinatorMetricGroup.java | 100 ++
.../hudi/metadata/TestHoodieTableMetadataUtil.java | 30 +
hudi-integ-test/spark-pipelines/ACCEPTANCE.md | 68 +
hudi-integ-test/spark-pipelines/KEYED_FLOWS.md | 54 +
.../spark-pipelines/MATERIALIZED_VIEWS.md | 48 +
hudi-integ-test/spark-pipelines/keyed_flows.py | 145 +++
.../spark-pipelines/materialized_views.py | 109 ++
.../spark-pipelines/phase1_acceptance.py | 180 +++
.../hudi/io/hfile/TestHFileMultiBlockScan.java | 40 +-
hudi-lakehouse/README.md | 7 +-
hudi-lakehouse/charts/hudi-spark-connect/README.md | 70 +-
.../local-dev/example/connect-requirements.txt | 2 +-
.../superset-spark-connect/Dockerfile | 18 +-
.../superset-spark-connect/QUICKSTART.md | 163 +++
hudi-lakehouse/superset-spark-connect/README.md | 146 +++
.../hudi_spark_connect}/__init__.py | 0
.../hudi_spark_connect/dialect.py | 180 +++
.../hudi_spark_connect/engine_spec.py | 72 ++
.../superset-spark-connect/pyproject.toml | 21 +-
.../superset-spark-connect/test_dialect.py | 137 ++
.../superset-spark-connect/test_engine_spec.py | 83 ++
hudi-lakehouse/superset-spark-connect/test_live.py | 138 ++
.../superset-spark-connect/test_superset_live.py | 196 +++
.../superset-spark-connect/test_zeppelin_live.py | 93 ++
.../scala/org/apache/hudi/HoodieFileIndex.scala | 9 +-
.../org/apache/hudi/HoodieStreamingSink.scala | 4 +
.../apache/hudi/PartitionStatsIndexSupport.scala | 77 +-
.../org/apache/hudi/SparkBaseIndexSupport.scala | 68 +-
.../sql/BaseHoodieCatalystExpressionUtils.scala | 74 +-
.../apache/spark/sql/HoodieSparkTypeUtils.scala | 45 -
.../apache/spark/sql/hudi/DataSkippingUtils.scala | 503 +-------
.../spark/sql/hudi/catalog/HoodieCatalog.scala | 54 +-
.../apache/spark/sql/hudi/dataskipping/Leaf.scala | 126 ++
.../spark/sql/hudi/dataskipping/RangeRule.scala | 133 ++
.../hudi/dataskipping/StatsFilterCompiler.scala | 81 ++
.../spark/sql/hudi/dataskipping/TermRange.scala | 155 +++
.../spark/sql/hudi/dataskipping/TimeZones.scala | 29 +-
.../spark/sql/hudi/dataskipping/Verdict.scala | 78 ++
.../sql/hudi/dataskipping/rules/CastRule.scala | 109 ++
.../dataskipping/rules/ConstantOperandRule.scala | 238 ++++
.../dataskipping/rules/DateTimeFormatRule.scala | 135 ++
.../hudi/dataskipping/rules/DateTimeOrdering.scala | 128 ++
.../dataskipping/rules/MonotonicFunctionRule.scala | 43 +
.../sql/hudi/dataskipping/rules/PrefixRule.scala | 42 +
.../dataskipping/rules/TimeZoneShiftRule.scala | 50 +
.../spark/sql/hudi/analysis/HoodieAnalysis.scala | 45 +-
.../spark/sql/parser/HoodieCommonSqlParser.scala | 6 +-
.../data-skipping/golden-index-filters.txt | 1334 ++++++++++++++++++++
.../org/apache/hudi/DataSkippingGoldenInputs.scala | 231 ++++
.../hudi/TestDataSkippingGoldenFilters.scala | 63 +
.../org/apache/hudi/TestDataSkippingStages.scala | 397 ++++++
.../org/apache/hudi/TestDataSkippingUtils.scala | 756 ++++++++++-
.../hudi/TestHoodieCatalystExpressionUtils.scala | 314 ++++-
.../hudi/TestPartitionStatsIndexSupport.scala | 145 +++
.../hudi/functional/TestBucketIndexSupport.scala | 11 +-
.../hudi/functional/TestPartitionStatsIndex.scala | 15 +-
.../functional/TestStreamingSinkRecovery.scala | 171 +++
.../hudi/functional/TestTimeTravelQuery.scala | 62 +-
.../testutils/HoodieDummyExpressionHolder.scala} | 24 +-
.../sql/hudi/catalog/TestHoodieCatalogDDL.scala | 100 +-
.../spark/sql/hudi/ddl/TestCreateTable.scala | 248 +++-
.../sql/hudi/dml/others/TestTimeTravelInCte.scala | 87 ++
.../sql/hudi/feature/TestDataSkippingQuery.scala | 367 ++++++
.../spark/sql/HoodieSpark3CatalystPlanUtils.scala | 8 +-
.../command/CreateHoodieTableAsSelectCommand.scala | 24 +-
.../command/InsertIntoHoodieTableCommand.scala | 11 +-
.../sql/HoodieSpark33CatalystExpressionUtils.scala | 9 +-
.../sql/HoodieSpark34CatalystExpressionUtils.scala | 9 +-
.../sql/HoodieSpark35CatalystExpressionUtils.scala | 9 +-
.../sql/HoodieSpark4CatalystExpressionUtils.scala | 9 +-
.../command/CreateHoodieTableAsSelectCommand.scala | 24 +-
.../command/InsertIntoHoodieTableCommand.scala | 11 +-
.../spark/sql/HoodieSpark40CatalystPlanUtils.scala | 8 +-
.../spark/sql/HoodieSpark41CatalystPlanUtils.scala | 8 +-
.../spark/sql/HoodieSpark42CatalystPlanUtils.scala | 8 +-
.../hudi/utilities/streamer/HoodieStreamer.java | 27 +-
.../apache/hudi/utilities/streamer/StreamSync.java | 16 +-
...TestHoodieStreamerAsyncTableServiceFailure.java | 131 ++
rfc/README.md | 2 +-
rfc/rfc-99/rfc-99.md | 37 +-
119 files changed, 9908 insertions(+), 1126 deletions(-)
copy
hudi-agent-gateway/tests/fixtures/hudi_architect/{flink_pr2/stable_key.json =>
flink_pr8/combined_safety_gates.json} (60%)
create mode 100644
hudi-agent-gateway/tests/fixtures/hudi_architect/flink_pr8/cross_path_scenarios.toml
create mode 100644
hudi-agent-gateway/tests/fixtures/hudi_architect/flink_pr8/existing_multi_writer_out_of_scope.json
create mode 100644
hudi-agent-gateway/tests/fixtures/hudi_architect/flink_pr8/malformed_cross_path.json
create mode 100644 hudi-agent-gateway/tests/test_hudi_architect_cross_path.py
create mode 100644
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/async/TestAsyncTableServiceFailure.java
create mode 100644
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/SharedCoordinatorMetricGroup.java
create mode 100644 hudi-integ-test/spark-pipelines/ACCEPTANCE.md
create mode 100644 hudi-integ-test/spark-pipelines/KEYED_FLOWS.md
create mode 100644 hudi-integ-test/spark-pipelines/MATERIALIZED_VIEWS.md
create mode 100644 hudi-integ-test/spark-pipelines/keyed_flows.py
create mode 100644 hudi-integ-test/spark-pipelines/materialized_views.py
create mode 100644 hudi-integ-test/spark-pipelines/phase1_acceptance.py
copy hudi-agent-gateway/src/hudi_agent_gateway/__init__.py =>
hudi-lakehouse/superset-spark-connect/Dockerfile (64%)
create mode 100644 hudi-lakehouse/superset-spark-connect/QUICKSTART.md
create mode 100644 hudi-lakehouse/superset-spark-connect/README.md
copy {hudi-agent-gateway/src/hudi_agent_gateway/api =>
hudi-lakehouse/superset-spark-connect/hudi_spark_connect}/__init__.py (100%)
create mode 100644
hudi-lakehouse/superset-spark-connect/hudi_spark_connect/dialect.py
create mode 100644
hudi-lakehouse/superset-spark-connect/hudi_spark_connect/engine_spec.py
copy hudi-agent-gateway/src/hudi_agent_gateway/__init__.py =>
hudi-lakehouse/superset-spark-connect/pyproject.toml (58%)
create mode 100644 hudi-lakehouse/superset-spark-connect/test_dialect.py
create mode 100644 hudi-lakehouse/superset-spark-connect/test_engine_spec.py
create mode 100644 hudi-lakehouse/superset-spark-connect/test_live.py
create mode 100644 hudi-lakehouse/superset-spark-connect/test_superset_live.py
create mode 100644 hudi-lakehouse/superset-spark-connect/test_zeppelin_live.py
delete mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/HoodieSparkTypeUtils.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/Leaf.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/RangeRule.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/StatsFilterCompiler.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TermRange.scala
copy
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkRowSerDe.scala
=>
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TimeZones.scala
(55%)
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/Verdict.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/CastRule.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/DateTimeFormatRule.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/DateTimeOrdering.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/MonotonicFunctionRule.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/PrefixRule.scala
create mode 100644
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/TimeZoneShiftRule.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/resources/data-skipping/golden-index-filters.txt
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/DataSkippingGoldenInputs.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSkippingGoldenFilters.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSkippingStages.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestPartitionStatsIndexSupport.scala
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStreamingSinkRecovery.scala
copy
hudi-spark-datasource/{hudi-spark-common/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/TimeTravelRelation.scala
=>
hudi-spark/src/test/scala/org/apache/hudi/testutils/HoodieDummyExpressionHolder.scala}
(55%)
create mode 100644
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/others/TestTimeTravelInCte.scala
create mode 100644
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieStreamerAsyncTableServiceFailure.java
