Re: [PR] feat: init iceberg writer [iceberg-rust]

2024-04-06 Thread via GitHub
marvinlanhenke commented on PR #275: URL: https://github.com/apache/iceberg-rust/pull/275#issuecomment-2041304545 @liurenjie1024 @ZENOTME What's the current status on this PR - as it looks very promising as well as the outlined framework in #34 ? Since we have already completed

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.35.0 [iceberg]

2024-04-06 Thread via GitHub
dependabot[bot] commented on PR #10070: URL: https://github.com/apache/iceberg/pull/10070#issuecomment-2041309006 Superseded by #10094. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[PR] Build: Bump net.snowflake:snowflake-jdbc from 3.14.5 to 3.15.1 [iceberg]

2024-04-06 Thread via GitHub
dependabot[bot] opened a new pull request, #10095: URL: https://github.com/apache/iceberg/pull/10095 Bumps [net.snowflake:snowflake-jdbc](https://github.com/snowflakedb/snowflake-jdbc) from 3.14.5 to 3.15.1. Release notes Sourced from

[PR] Build: Bump software.amazon.awssdk:bom from 2.25.21 to 2.25.26 [iceberg]

2024-04-06 Thread via GitHub
dependabot[bot] opened a new pull request, #10093: URL: https://github.com/apache/iceberg/pull/10093 Bumps software.amazon.awssdk:bom from 2.25.21 to 2.25.26. [![Dependabot compatibility

[PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.37.0 [iceberg]

2024-04-06 Thread via GitHub
dependabot[bot] opened a new pull request, #10094: URL: https://github.com/apache/iceberg/pull/10094 Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.28.0 to 26.37.0. Release notes Sourced from

Re: [PR] [Bug Fix] Allow HiveCatalog to create table with TimestamptzType [iceberg-python]

2024-04-06 Thread via GitHub
Fokko commented on code in PR #585: URL: https://github.com/apache/iceberg-python/pull/585#discussion_r1554715441 ## pyiceberg/catalog/hive.py: ## @@ -199,6 +184,7 @@ def _annotate_namespace(database: HiveDatabase, properties: Properties) -> HiveD DateType: "date",

Re: [I] master branch - flink sql create hive catalog error [iceberg]

2024-04-06 Thread via GitHub
github-actions[bot] commented on issue #2468: URL: https://github.com/apache/iceberg/issues/2468#issuecomment-2041247963 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [PR] Implement __getstate__ and __setstate__ on PyArrowFileIO and FsSpecFileIO so that they can be pickled [iceberg-python]

2024-04-06 Thread via GitHub
amogh-jahagirdar commented on code in PR #543: URL: https://github.com/apache/iceberg-python/pull/543#discussion_r1554762228 ## tests/io/test_pyarrow.py: ## @@ -256,6 +257,14 @@ def test_raise_on_opening_a_local_file_not_found() -> None: assert "[Errno 2] Failed to

Re: [I] How do I fetch the latest ROADMAP [iceberg]

2024-04-06 Thread via GitHub
github-actions[bot] commented on issue #2469: URL: https://github.com/apache/iceberg/issues/2469#issuecomment-2041247971 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

[PR] Build: Bump mkdocs-material from 9.5.15 to 9.5.17 [iceberg]

2024-04-06 Thread via GitHub
dependabot[bot] opened a new pull request, #10092: URL: https://github.com/apache/iceberg/pull/10092 Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 9.5.15 to 9.5.17. Release notes Sourced from

[PR] Parquet: Make row-group filters cooperate to filter [iceberg]

2024-04-06 Thread via GitHub
zhongyujiang opened a new pull request, #10090: URL: https://github.com/apache/iceberg/pull/10090 This PR refactors three Parquet row-group filters into a form that computes residual expressions, allowing it to return a residual expression for the given row-groups. The residual computed by

Re: [PR] Parquet: Make row-group filters cooperate to filter [iceberg]

2024-04-06 Thread via GitHub
zhongyujiang closed pull request #6893: Parquet: Make row-group filters cooperate to filter URL: https://github.com/apache/iceberg/pull/6893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] Parquet: Make row-group filters cooperate to filter [iceberg]

2024-04-06 Thread via GitHub
zhongyujiang commented on PR #6893: URL: https://github.com/apache/iceberg/pull/6893#issuecomment-2041294392 Replaced by #10090. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[PR] Docs: Fix On-screen display issues and minor expressions on Branching and Tagging DDL [iceberg]

2024-04-06 Thread via GitHub
lawofcycles opened a new pull request, #10091: URL: https://github.com/apache/iceberg/pull/10091 I propose following three modifications. - Fix a broken bullet point display. https://github.com/apache/iceberg/assets/70102274/1ce9bfe1-4424-4b59-baec-ccf7e6a3fec7;>

Re: [PR] Docs: Fix inconsistency in branching and tagging scenario [iceberg]

2024-04-06 Thread via GitHub
lawofcycles commented on PR #9968: URL: https://github.com/apache/iceberg/pull/9968#issuecomment-2041291407 @bitsondatadev While referring to your comments, I pushed new version to improve entire explanation for Historical Tags. I was aware of the following points. -

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.35.0 [iceberg]

2024-04-06 Thread via GitHub
dependabot[bot] closed pull request #10070: Build: Bump com.google.cloud:libraries-bom from 26.28.0 to 26.35.0 URL: https://github.com/apache/iceberg/pull/10070 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[I] Suppress duplicate OAuth token fetching in rest catalog client [iceberg-python]

2024-04-06 Thread via GitHub
TennyZhuang opened a new issue, #587: URL: https://github.com/apache/iceberg-python/issues/587 ### Feature Request / Improvement In the rest catalog client, we implemented the OAuth token refresh based on retry mechanism.

Re: [PR] Add Struct Accessors to BoundReferences [iceberg-rust]

2024-04-06 Thread via GitHub
sdd commented on code in PR #317: URL: https://github.com/apache/iceberg-rust/pull/317#discussion_r1554540831 ## crates/iceberg/src/expr/accessor.rs: ## @@ -0,0 +1,119 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Add Struct Accessors to BoundReferences [iceberg-rust]

2024-04-06 Thread via GitHub
sdd commented on code in PR #317: URL: https://github.com/apache/iceberg-rust/pull/317#discussion_r1554540767 ## crates/iceberg/src/expr/accessor.rs: ## @@ -0,0 +1,119 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements.

Re: [PR] Add Struct Accessors to BoundReferences [iceberg-rust]

2024-04-06 Thread via GitHub
sdd commented on code in PR #317: URL: https://github.com/apache/iceberg-rust/pull/317#discussion_r1554542369 ## crates/iceberg/src/spec/schema.rs: ## @@ -137,9 +142,55 @@ impl SchemaBuilder { name_to_id, lowercase_name_to_id, id_to_name,

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-06 Thread via GitHub
sdd commented on code in PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#discussion_r1554546747 ## crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs: ## @@ -0,0 +1,366 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more

Re: [PR] Add Struct Accessors to BoundReferences [iceberg-rust]

2024-04-06 Thread via GitHub
sdd commented on PR #317: URL: https://github.com/apache/iceberg-rust/pull/317#issuecomment-2041050879 @liurenjie1024 I've added a test as well now for build_accessors :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[I] Concern about possible consistency issue in HiveCatalog's _commit_table [iceberg-python]

2024-04-06 Thread via GitHub
HonahX opened a new issue, #588: URL: https://github.com/apache/iceberg-python/issues/588 ### Question Currently, the HiveCatalog's `_commit_table` workflow looks like: 1. load current table metadata via `load_table` 2. construct updated metadata 3. lock the hive table

Re: [PR] Add Struct Accessors to BoundReferences [iceberg-rust]

2024-04-06 Thread via GitHub
liurenjie1024 commented on code in PR #317: URL: https://github.com/apache/iceberg-rust/pull/317#discussion_r1554560769 ## crates/iceberg/src/spec/schema.rs: ## @@ -137,9 +142,55 @@ impl SchemaBuilder { name_to_id, lowercase_name_to_id,

Re: [PR] Add Struct Accessors to BoundReferences [iceberg-rust]

2024-04-06 Thread via GitHub
liurenjie1024 merged PR #317: URL: https://github.com/apache/iceberg-rust/pull/317 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [I] Spark rewrite Files Action OOM [iceberg]

2024-04-06 Thread via GitHub
Zhanxiao-Ma commented on issue #10054: URL: https://github.com/apache/iceberg/issues/10054#issuecomment-2041085307 > @nk1506 Echoing Russell's comments, how many small files are there in your OOM case? How much memory do you set up? @RussellSpitzer I believe increasing memory is not

Re: [I] What is the meaning of `delete_rows_count` and `delete_data_count_file ` at manifest [iceberg]

2024-04-06 Thread via GitHub
wg1026688210 closed issue #2445: What is the meaning of `delete_rows_count` and `delete_data_count_file ` at manifest URL: https://github.com/apache/iceberg/issues/2445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-06 Thread via GitHub
marvinlanhenke commented on code in PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#discussion_r1554653083 ## crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs: ## @@ -0,0 +1,260 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [PR] Add `BoundPredicateVisitor` trait [iceberg-rust]

2024-04-06 Thread via GitHub
marvinlanhenke commented on code in PR #320: URL: https://github.com/apache/iceberg-rust/pull/320#discussion_r1554654358 ## crates/iceberg/src/expr/visitors/bound_predicate_visitor.rs: ## @@ -0,0 +1,260 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or

Re: [I] [BUG] Valid column characters fail on to_arrow() or to_pandas() ArrowInvalid: No match for FieldRef.Name [iceberg-python]

2024-04-06 Thread via GitHub
kevinjqliu commented on issue #584: URL: https://github.com/apache/iceberg-python/issues/584#issuecomment-2041172613 Thanks for reporting this! And providing a reproducible example. Here's what I've found. In the `_task_to_table` function, the provided schema is modified