Re: [PR] GH-47244: [CI][Dev] Fix shellcheck errors in the ci/scripts/msys2_setup.sh [arrow]

2025-07-30 Thread via GitHub
github-actions[bot] commented on PR #47245: URL: https://github.com/apache/arrow/pull/47245#issuecomment-3138759303 :warning: GitHub issue #47244 **has been automatically assigned in GitHub** to PR creator. -- This is an automated message from the Apache Git Service. To respond to the mes

[PR] GH-47244: [CI][Dev] Fix shellcheck errors in the ci/scripts/msys2_setup.sh [arrow]

2025-07-30 Thread via GitHub
hiroyuki-sato opened a new pull request, #47245: URL: https://github.com/apache/arrow/pull/47245 ### Rationale for this change This is the sub issue #44748. * SC2206: Quote to prevent word splitting/globbing * SC2086: Double quote to prevent globbing ``` shellcheck

Re: [PR] GH-47075: [Release][Dev] Use GH_TOKEN as GitHub token environment variable [arrow]

2025-07-30 Thread via GitHub
conbench-apache-arrow[bot] commented on PR #47181: URL: https://github.com/apache/arrow/pull/47181#issuecomment-3138755992 After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit ea5dad5fb0ead086878addaaf8473cf623c41005. There weren't

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244485742 ## arrow-avro/src/schema.rs: ## @@ -260,13 +274,369 @@ pub struct Fixed<'a> { pub attributes: Attributes<'a>, } +/// Supported fingerprint algorithms for Av

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244483679 ## arrow-avro/src/schema.rs: ## @@ -260,13 +274,369 @@ pub struct Fixed<'a> { pub attributes: Attributes<'a>, } +/// Supported fingerprint algorithms for Av

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244482440 ## arrow-avro/src/schema.rs: ## @@ -260,13 +274,369 @@ pub struct Fixed<'a> { pub attributes: Attributes<'a>, } +/// Supported fingerprint algorithms for Av

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244481107 ## arrow-avro/src/reader/mod.rs: ## @@ -520,20 +498,13 @@ impl ReaderBuilder { self.writer_schema_store.as_ref(), self.reader_schema.as_re

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244476447 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,134 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244475539 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,134 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r226409 ## arrow-avro/src/schema.rs: ## @@ -260,13 +274,369 @@ pub struct Fixed<'a> { pub attributes: Attributes<'a>, } +/// Supported fingerprint algorithms for Av

[PR] fix(array): add nil checks in Data.Release() for childData [arrow-go]

2025-07-30 Thread via GitHub
secfree opened a new pull request, #456: URL: https://github.com/apache/arrow-go/pull/456 ### Rationale for this change My program crashed with below log ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation cod

Re: [PR] feat: add method for sync Parquet reader read bloom filter [arrow-rs]

2025-07-30 Thread via GitHub
mapleFU commented on PR #8024: URL: https://github.com/apache/arrow-rs/pull/8024#issuecomment-3138666565 @alamb would you mind take a look? (or just close this). I use this in a poc, might switch to async_reader when I goto production version -- This is an automated message from the Apach

Re: [I] Support for "Schema evolution" / Schema Adapters [arrow-rs]

2025-07-30 Thread via GitHub
mapleFU commented on issue #6735: URL: https://github.com/apache/arrow-rs/issues/6735#issuecomment-3138655493 I noticed that in parquet internal there're multiple kinds of evolution: 1. Without any real cast. This happens on type changes (1) Like `Utf8 -> Utf8View`, `List -> ListView` the

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244423031 ## arrow-avro/src/schema.rs: ## @@ -260,13 +274,369 @@ pub struct Fixed<'a> { pub attributes: Attributes<'a>, } +/// Supported fingerprint algorithms for Av

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244404191 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,130 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data: &

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244379215 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,130 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data: &

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244379215 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,130 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data: &

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244379215 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,130 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data: &

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244379215 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,130 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data: &

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244379215 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,130 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data: &

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244379215 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,130 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data: &

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244284471 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,130 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data: &

Re: [PR] fix(csharp/src/Drivers/Databricks): Tighten Statement, Reader, Poller Disposal [arrow-adbc]

2025-07-30 Thread via GitHub
toddmeng-db commented on code in PR #3217: URL: https://github.com/apache/arrow-adbc/pull/3217#discussion_r2244010258 ## csharp/src/Drivers/Databricks/DatabricksOperationStatusPoller.cs: ## @@ -69,13 +72,18 @@ private async Task PollOperationStatus(CancellationToken cancellatio

Re: [PR] fix(csharp/src/Drivers/Databricks): Tighten Statement, Reader, Poller Disposal [arrow-adbc]

2025-07-30 Thread via GitHub
toddmeng-db commented on code in PR #3217: URL: https://github.com/apache/arrow-adbc/pull/3217#discussion_r2244010258 ## csharp/src/Drivers/Databricks/DatabricksOperationStatusPoller.cs: ## @@ -69,13 +72,18 @@ private async Task PollOperationStatus(CancellationToken cancellatio

Re: [PR] fix(csharp/src/Drivers/Databricks): Tighten Statement, Reader, Poller Disposal [arrow-adbc]

2025-07-30 Thread via GitHub
toddmeng-db commented on code in PR #3217: URL: https://github.com/apache/arrow-adbc/pull/3217#discussion_r2244010258 ## csharp/src/Drivers/Databricks/DatabricksOperationStatusPoller.cs: ## @@ -69,13 +72,18 @@ private async Task PollOperationStatus(CancellationToken cancellatio

Re: [I] [Python] pyarrow 21.0.0 binary mismatch with raspberry 4 hardware [arrow]

2025-07-30 Thread via GitHub
amoeba commented on issue #47229: URL: https://github.com/apache/arrow/issues/47229#issuecomment-3138537418 Hi @epigramx reporting here is fine. I can reproduce on an aarch64 pi, ```sh pi@pi:~$ lsb_release -a No LSB modules are available. Distributor ID: Debian Descrip

Re: [PR] GH-47211: [CI][R] Disable non-system memory allocators when on linux-devel [arrow]

2025-07-30 Thread via GitHub
conbench-apache-arrow[bot] commented on PR #47212: URL: https://github.com/apache/arrow/pull/47212#issuecomment-3138511628 After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit c3be5fc93d3409301a92b61268129180ebb067ff. There weren't

Re: [I] [Release] Use "upstream" as apache/arrow remote name [arrow]

2025-07-30 Thread via GitHub
amoeba commented on issue #47223: URL: https://github.com/apache/arrow/issues/47223#issuecomment-3138506738 +1 for the convenience point you make. Do we have any idea where/when the project started using the apache remote name? -- This is an automated message from the Apache Git Service.

Re: [PR] GH-18923: [C++] Switch to S3-CRT library for S3 FileSystem implementation [arrow]

2025-07-30 Thread via GitHub
adamreeve commented on PR #47219: URL: https://github.com/apache/arrow/pull/47219#issuecomment-3138502032 Most of the other tests have the same problem as above. The `TestS3FSHTTPS` tests seem to have a different but similar issue. These require using TLS and [set the listen address to `lo

Re: [PR] GH-47234: [C++][Python] DRAFT Debugging fill_null error [arrow]

2025-07-30 Thread via GitHub
github-actions[bot] commented on PR #47240: URL: https://github.com/apache/arrow/pull/47240#issuecomment-3138500606 Revision: 9aa816405b4885113405a87484971e06681dd93f Submitted crossbow builds: [ursacomputing/crossbow @ actions-f0222fb0d0](https://github.com/ursacomputing/crossbow/bra

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#issuecomment-3138500246 I'll let Felipe take another look before I merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] GH-47234: [C++][Python] DRAFT Debugging fill_null error [arrow]

2025-07-30 Thread via GitHub
amoeba commented on PR #47240: URL: https://github.com/apache/arrow/pull/47240#issuecomment-3138497836 @github-actions crossbow submit wheel-windows-* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] GH-47234: [C++][Python] DRAFT Debugging fill_null error [arrow]

2025-07-30 Thread via GitHub
amoeba commented on PR #47240: URL: https://github.com/apache/arrow/pull/47240#issuecomment-3138496150 Good. Looks like the unit test reproduced the failure above. From [wheel-windows-cp13-cp13-amd64](https://github.com/ursacomputing/crossbow/actions/runs/16637800781/job/47082104737#s

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244292448 ## arrow-avro/src/schema.rs: ## @@ -260,13 +274,369 @@ pub struct Fixed<'a> { pub attributes: Attributes<'a>, } +/// Supported fingerprint algorithms for Av

Re: [PR] GH-47243: [C++] initialize arrow::compute in execution_plan_documentation_examples [arrow]

2025-07-30 Thread via GitHub
github-actions[bot] commented on PR #47227: URL: https://github.com/apache/arrow/pull/47227#issuecomment-3138481755 :warning: GitHub issue #47243 **has been automatically assigned in GitHub** to PR creator. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [I] go/adbc/driver/snowflake: handling of wildcards in GetObjects [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on issue #3220: URL: https://github.com/apache/arrow-adbc/issues/3220#issuecomment-3138453261 I think I've stalled on those enough; @zeroshade @amoeba and Curt what do you think of adding a way to disable wildcards here (and elsewhere)? I think escaping kind of doesn't ma

Re: [I] go/adbc/driver/snowflake: handling of wildcards in GetObjects [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on issue #3220: URL: https://github.com/apache/arrow-adbc/issues/3220#issuecomment-3138450733 https://github.com/apache/arrow-adbc/issues/1398 is also in principle related -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] go/adbc/driver/snowflake: handling of wildcards in GetObjects [arrow-adbc]

2025-07-30 Thread via GitHub
CurtHagenlocher commented on issue #3220: URL: https://github.com/apache/arrow-adbc/issues/3220#issuecomment-3138430422 (This was also mentioned in #1508) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
eitsupi commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244242683 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))?

Re: [I] go/adbc/driver/snowflake: handling of wildcards in GetObjects [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on issue #3220: URL: https://github.com/apache/arrow-adbc/issues/3220#issuecomment-3138403766 Hmm, we don't specify a way to escape, but I think there should be a way to escape (if perhaps database-specific), or to disable the wildcards when desired. -- This is an autom

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244238673 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))

Re: [PR] feat: add support for utf8view type [arrow-js]

2025-07-30 Thread via GitHub
NguyenHoangSon96 commented on PR #225: URL: https://github.com/apache/arrow-js/pull/225#issuecomment-3138399675 Hi @trxcllnt Can you help me review this PR? I can't add you to the Reviewers for some reason, so I commented here. This is the first time I have created a PR for arrow-js,

Re: [PR] ci(parquet/pqarrow): integration tests for reading shredded variants DO NOT MERGE YET [arrow-go]

2025-07-30 Thread via GitHub
wgtmac commented on PR #455: URL: https://github.com/apache/arrow-go/pull/455#issuecomment-3138388899 > My preference would be to relax the spec for this issue. It doesn't seem like there's much benefit to enforcing it on the read side, and it's easy to imagine a writer failing to enforce i

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244224068 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
eitsupi commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244215273 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))?

Re: [I] [C++][Python] pa.compute.fill_null produces unexpected result in arrow 21 on windows [arrow]

2025-07-30 Thread via GitHub
amoeba commented on issue #47234: URL: https://github.com/apache/arrow/issues/47234#issuecomment-3138366675 The xsimd PR looks worth investigating. I'm running some crossbow jobs which I'm hope reproduce the error in https://github.com/apache/arrow/pull/47240. There are some troubling

Re: [PR] GH-47241: [C++][Parquet] Fix VariantExtentionType conversion [arrow]

2025-07-30 Thread via GitHub
wgtmac commented on PR #47242: URL: https://github.com/apache/arrow/pull/47242#issuecomment-3138362282 @pitrou @mapleFU Please take a look when you have time. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] GH-47241: [C++][Parquet] Fix VariantExtentionType conversion [arrow]

2025-07-30 Thread via GitHub
wgtmac commented on code in PR #47242: URL: https://github.com/apache/arrow/pull/47242#discussion_r2244203680 ## cpp/src/parquet/types.cc: ## @@ -750,8 +750,9 @@ bool LogicalType::is_valid() const { } bool LogicalType::is_invalid() const { return !is_valid(); } bool LogicalTy

Re: [PR] ci(parquet/pqarrow): integration tests for reading shredded variants DO NOT MERGE YET [arrow-go]

2025-07-30 Thread via GitHub
cashmand commented on PR #455: URL: https://github.com/apache/arrow-go/pull/455#issuecomment-3138348439 > Correct, the spec states that if the typed_value field is omitted, then the value field must be required, so Go errors if it is optional when the typed_value field is omitted causing th

Re: [PR] GH-47241: [C++][Parquet] Fix VariantExtentionType conversion [arrow]

2025-07-30 Thread via GitHub
github-actions[bot] commented on PR #47242: URL: https://github.com/apache/arrow/pull/47242#issuecomment-3138347952 :warning: GitHub issue #47241 **has been automatically assigned in GitHub** to PR creator. -- This is an automated message from the Apache Git Service. To respond to the mes

[PR] GH-47241: [C++][Parquet] Fix VariantExtentionType conversion [arrow]

2025-07-30 Thread via GitHub
wgtmac opened a new pull request, #47242: URL: https://github.com/apache/arrow/pull/47242 ### Rationale for this change The test case of VariantExtentionType is incomplete. We need to make sure its conversion is well covered. ### What changes are included in this PR? Fix

Re: [PR] GH-47191: [R] Turn GCS back on by default on MacOS source builds [arrow]

2025-07-30 Thread via GitHub
conbench-apache-arrow[bot] commented on PR #47192: URL: https://github.com/apache/arrow/pull/47192#issuecomment-3138328811 After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit f9301c0ba8a7ed1b0b63275cfdd4c44c26b04675. There weren't

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
eitsupi commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244160285 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))?

Re: [PR] GH-47234: [C++][Python] DRAFT Debugging fill_null error [arrow]

2025-07-30 Thread via GitHub
github-actions[bot] commented on PR #47240: URL: https://github.com/apache/arrow/pull/47240#issuecomment-3138310287 Revision: 7f5788e07628b2da8dd5803a48a93cb4ae500cd9 Submitted crossbow builds: [ursacomputing/crossbow @ actions-1b6ca3b835](https://github.com/ursacomputing/crossbow/bra

Re: [PR] GH-47234: [C++][Python] DRAFT Debugging fill_null error [arrow]

2025-07-30 Thread via GitHub
amoeba commented on PR #47240: URL: https://github.com/apache/arrow/pull/47240#issuecomment-3138307664 @github-actions crossbow submit wheel-windows-* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] GH-47234: [C++][Python] DRAFT Debugging fill_null error [arrow]

2025-07-30 Thread via GitHub
github-actions[bot] commented on PR #47240: URL: https://github.com/apache/arrow/pull/47240#issuecomment-3138306879 :warning: GitHub issue #47234 **has been automatically assigned in GitHub** to PR creator. -- This is an automated message from the Apache Git Service. To respond to the mes

[PR] GH-47234: [C++][Python] DRAFT Debugging fill_null error [arrow]

2025-07-30 Thread via GitHub
amoeba opened a new pull request, #47240: URL: https://github.com/apache/arrow/pull/47240 This PR is a draft for testing. Do not merge. Testing that we can reproduce https://github.com/apache/arrow/issues/47234 in CI. -- This is an automated message from the Apache Git Service. To

Re: [PR] fix(csharp/src/Drivers/Databricks): Tighten Statement, Reader, Poller Disposal [arrow-adbc]

2025-07-30 Thread via GitHub
toddmeng-db commented on code in PR #3217: URL: https://github.com/apache/arrow-adbc/pull/3217#discussion_r2244166363 ## csharp/src/Drivers/Databricks/DatabricksCompositeReader.cs: ## @@ -108,5 +111,41 @@ private BaseDatabricksReader DetermineReader(TFetchResultsResp initialRes

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
eitsupi commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244160285 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))?

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244142378 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))

Re: [PR] ci(parquet/pqarrow): integration tests for reading shredded variants DO NOT MERGE YET [arrow-go]

2025-07-30 Thread via GitHub
aihuaxu commented on PR #455: URL: https://github.com/apache/arrow-go/pull/455#issuecomment-3138264152 `Test case 84, testShreddedObjectWithOptionalFieldStructs tests the schenario where the shredded fields of an object are listed as optional in the schema, but the spec states that they mus

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
eitsupi commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244136404 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))?

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244100996 ## arrow-avro/src/reader/mod.rs: ## @@ -272,17 +450,70 @@ impl ReaderBuilder { self } -/// Sets the Avro schema. +/// Sets the Avro reader s

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244126396 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))

Re: [PR] refactor(rust/core)!: move the driver_manager feature to the new adbc_driver_manager package [arrow-adbc]

2025-07-30 Thread via GitHub
eitsupi commented on code in PR #3197: URL: https://github.com/apache/arrow-adbc/pull/3197#discussion_r2244120953 ## rust/driver_manager/src/lib.rs: ## @@ -720,6 +712,7 @@ fn set_option_database( ERR_ONLY_STRING_OPT, Status::NotImplemented, ))?

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244110570 ## arrow-avro/src/reader/mod.rs: ## @@ -272,17 +450,70 @@ impl ReaderBuilder { self } -/// Sets the Avro schema. +/// Sets the Avro reader s

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244100996 ## arrow-avro/src/reader/mod.rs: ## @@ -272,17 +450,70 @@ impl ReaderBuilder { self } -/// Sets the Avro schema. +/// Sets the Avro reader s

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244100996 ## arrow-avro/src/reader/mod.rs: ## @@ -272,17 +450,70 @@ impl ReaderBuilder { self } -/// Sets the Avro schema. +/// Sets the Avro reader s

Re: [PR] docs: Fix pip install command for arrow-adbc-nightlies [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm merged PR #3222: URL: https://github.com/apache/arrow-adbc/pull/3222 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.a

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244086911 ## arrow-avro/src/reader/mod.rs: ## @@ -272,17 +450,70 @@ impl ReaderBuilder { self } -/// Sets the Avro schema. +/// Sets the Avro reader s

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244084404 ## arrow-avro/src/reader/mod.rs: ## @@ -154,39 +167,134 @@ impl Decoder { /// /// Returns the number of bytes consumed. pub fn decode(&mut self, data

[PR] docs: Fix pip install command for arrow-adbc-nightlies [arrow-adbc]

2025-07-30 Thread via GitHub
amoeba opened a new pull request, #3222: URL: https://github.com/apache/arrow-adbc/pull/3222 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] GH-47193: [R] Update R Makefile to exclude flight odbc from cpp sync [arrow]

2025-07-30 Thread via GitHub
conbench-apache-arrow[bot] commented on PR #47194: URL: https://github.com/apache/arrow/pull/47194#issuecomment-3138151846 After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 93915f511a7bac565e58651953b8ef62b701f02b. There weren't

Re: [PR] chore(rust): bump the arrow-datafusion group in /rust with 2 updates [arrow-adbc]

2025-07-30 Thread via GitHub
lidavidm commented on PR #3210: URL: https://github.com/apache/arrow-adbc/pull/3210#issuecomment-3138133368 Looks like @eitsupi's test caught that we need to update the MSRV for the datafusion crate -- This is an automated message from the Apache Git Service. To respond to the message, pl

Re: [PR] GH-47123: [Python] Add Enums to PyArrow Types [arrow]

2025-07-30 Thread via GitHub
conbench-apache-arrow[bot] commented on PR #47139: URL: https://github.com/apache/arrow/pull/47139#issuecomment-3138086607 After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 2949fe82cb674aa72bbdf4b6886a9fc370c0a35c. There weren't

Re: [PR] GH-45055: [C++][Flight] Update Flight Server RecordBatchStreamImpl to reuse ipc::RecordBatchWriter with custom IpcPayloadWriter instead of manually generating FlightPayload [arrow]

2025-07-30 Thread via GitHub
conbench-apache-arrow[bot] commented on PR #47115: URL: https://github.com/apache/arrow/pull/47115#issuecomment-3138086343 After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 25f8f008061b978137a2c1e5bc934d07fae56e3e. There weren't

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
jecsand838 commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2244016597 ## arrow-avro/src/reader/mod.rs: ## @@ -246,8 +252,8 @@ impl Decoder { self.prepare_schema_switch(new_fp)?; // If there are already decode

Re: [PR] feat(go/adbc/driver/bigquery): support service account impersonation [arrow-adbc]

2025-07-30 Thread via GitHub
yu-iskw commented on PR #3174: URL: https://github.com/apache/arrow-adbc/pull/3174#issuecomment-3138050235 I see. Thank you for sharing the information. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] fix(csharp/src/Drivers/Databricks): Tighten Statement, Reader, Poller Disposal [arrow-adbc]

2025-07-30 Thread via GitHub
toddmeng-db commented on code in PR #3217: URL: https://github.com/apache/arrow-adbc/pull/3217#discussion_r2244010258 ## csharp/src/Drivers/Databricks/DatabricksOperationStatusPoller.cs: ## @@ -69,13 +72,18 @@ private async Task PollOperationStatus(CancellationToken cancellatio

Re: [I] [C++] c++23 supported? [arrow]

2025-07-30 Thread via GitHub
BwL1289 commented on issue #47207: URL: https://github.com/apache/arrow/issues/47207#issuecomment-3138032052 @kou it doesn't look like we can create sub-issues (not sure if we have insufficient permissions) but while trying to follow [these directions](https://docs.github.com/en/issues/trac

Re: [PR] feat(ruby): don't use adbc-arrow-glib [arrow-adbc]

2025-07-30 Thread via GitHub
kou commented on PR #3221: URL: https://github.com/apache/arrow-adbc/pull/3221#issuecomment-3137932813 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

Re: [PR] feat(ruby): don't use adbc-arrow-glib [arrow-adbc]

2025-07-30 Thread via GitHub
kou merged PR #3221: URL: https://github.com/apache/arrow-adbc/pull/3221 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache

Re: [PR] feat(csharp/src/Apache.Arrow.Adbc/Tracing): allow ActivitySource tags to be set from TracingConnection [arrow-adbc]

2025-07-30 Thread via GitHub
birschick-bq commented on PR #3218: URL: https://github.com/apache/arrow-adbc/pull/3218#issuecomment-3137916645 > @birschick-bq So if I understand correctly, for the first approach: > > 1. Each connection generates a GUID > 2. Add the GUID to the ActivitySource tags > 3. Each con

Re: [I] [Parquet] reader appears to have bug when supporting unknown sort orders [arrow-rs]

2025-07-30 Thread via GitHub
etseidl commented on issue #7909: URL: https://github.com/apache/arrow-rs/issues/7909#issuecomment-3137897360 > look etseidl did it! Yes, by copying the work of others 🤣. And I'll need a lot of help before this is done 😅. But yes, if we can get a significant query performance gain for

Re: [PR] feat(csharp/src/Apache.Arrow.Adbc/Tracing): allow ActivitySource tags to be set from TracingConnection [arrow-adbc]

2025-07-30 Thread via GitHub
jeremytang-db commented on PR #3218: URL: https://github.com/apache/arrow-adbc/pull/3218#issuecomment-3137895585 @birschick-bq So if I understand correctly, for the first approach: 1. Each connection generates a GUID 2. Add the GUID to the ActivitySource tags 3. Each connecti

Re: [I] [Parquet] reader appears to have bug when supporting unknown sort orders [arrow-rs]

2025-07-30 Thread via GitHub
alamb commented on issue #7909: URL: https://github.com/apache/arrow-rs/issues/7909#issuecomment-3137886581 @etseidl you are going to finally let me have some data for the claim "look all you FlatBuffers lovers out there, if you really care about footer performance you can just write a bet

Re: [I] [Parquet] reader appears to have bug when supporting unknown sort orders [arrow-rs]

2025-07-30 Thread via GitHub
alamb commented on issue #7909: URL: https://github.com/apache/arrow-rs/issues/7909#issuecomment-3137884570 > As far as a road map, I'm finding while doing this exercise that the mixing of structures in the format and basic modules is not ideal. I think first removing any use of format with

[PR] feat(ruby): don't use adbc-arrow-glib [arrow-adbc]

2025-07-30 Thread via GitHub
kou opened a new pull request, #3221: URL: https://github.com/apache/arrow-adbc/pull/3221 adbc-arrow-glib depends on arrow-glib. In general, arrow-glib is released as major version release frequently. Our adbc-arrow-glib deb/rpm packages may refer old arrow-glib binaries soon. If the

Re: [PR] ci(parquet/pqarrow): integration tests for reading shredded variants DO NOT MERGE YET [arrow-go]

2025-07-30 Thread via GitHub
zeroshade commented on PR #455: URL: https://github.com/apache/arrow-go/pull/455#issuecomment-3137832489 > Both value and typed_value are optional per spec and value can be missing as I understand. While the spec states that `typed_value` may be omitted, it does not say the same abou

Re: [PR] feat(csharp/src/Apache.Arrow.Adbc/Tracing): allow ActivitySource tags to be set from TracingConnection [arrow-adbc]

2025-07-30 Thread via GitHub
jadewang-db commented on PR #3218: URL: https://github.com/apache/arrow-adbc/pull/3218#issuecomment-3137801323 > > So the idea is to be able to do something like this? `ShouldListenTo = (activitySource) => activitySource.Tags["Hosturl"] = "abc.databricks.com"` If there are mulitple instance

Re: [PR] ci(parquet/pqarrow): integration tests for reading shredded variants DO NOT MERGE YET [arrow-go]

2025-07-30 Thread via GitHub
aihuaxu commented on PR #455: URL: https://github.com/apache/arrow-go/pull/455#issuecomment-3137794519 > Several test cases test variations on situations where the value column is missing. Based on my reading of the [spec](https://github.com/apache/parquet-format/blob/master/VariantShreddin

Re: [PR] ci(parquet/pqarrow): integration tests for reading shredded variants DO NOT MERGE YET [arrow-go]

2025-07-30 Thread via GitHub
zeroshade commented on PR #455: URL: https://github.com/apache/arrow-go/pull/455#issuecomment-3137763209 I'm away for the rest of this week but I can retest on Tuesday. The regeneration of the tests wouldn't fix the rest of the issues I listed right? -- This is an automated message from t

Re: [I] Push-Based Parquet Reader [arrow-rs]

2025-07-30 Thread via GitHub
alamb commented on issue #1605: URL: https://github.com/apache/arrow-rs/issues/1605#issuecomment-3137702633 BTW we are discussing (and I am prototyping) a proper push decoder to fully decouple things. It is getting close. See this for more details - https://github.com/apache/arrow

Re: [PR] Implement arrow-avro SchemaStore and Fingerprinting To Enable Schema Resolution [arrow-rs]

2025-07-30 Thread via GitHub
scovich commented on code in PR #8006: URL: https://github.com/apache/arrow-rs/pull/8006#discussion_r2243631735 ## arrow-avro/src/reader/mod.rs: ## @@ -216,34 +330,98 @@ impl ReaderBuilder { /// - `batch_size` = 1024 /// - `strict_mode` = false /// - `utf8_view` =

Re: [I] [Parquet] reader appears to have bug when supporting unknown sort orders [arrow-rs]

2025-07-30 Thread via GitHub
etseidl commented on issue #7909: URL: https://github.com/apache/arrow-rs/issues/7909#issuecomment-3137639506 Quick follow up. I've gone down the rabbit hole of a custom implementation. So far I've found that the thrift code in `TCompactSliceInputProtocol` is pretty good 😄, but by repeating

Re: [PR] fix(csharp/src/Drivers/Databricks): Tighten Statement, Reader, Poller Disposal [arrow-adbc]

2025-07-30 Thread via GitHub
toddmeng-db commented on code in PR #3217: URL: https://github.com/apache/arrow-adbc/pull/3217#discussion_r2243585568 ## csharp/src/Drivers/Databricks/DatabricksCompositeReader.cs: ## @@ -16,17 +16,12 @@ */ Review Comment: Could consider putting OperationStatusPoller log

Re: [PR] fix(csharp/src/Drivers/Databricks): Tighten Statement, Reader, Poller Disposal [arrow-adbc]

2025-07-30 Thread via GitHub
toddmeng-db commented on code in PR #3217: URL: https://github.com/apache/arrow-adbc/pull/3217#discussion_r2241479129 ## csharp/src/Drivers/Databricks/CloudFetch/CloudFetchResultFetcher.cs: ## @@ -247,6 +247,10 @@ private async Task FetchResultsAsync(CancellationToken cancellat

Re: [PR] feat(csharp/src/Apache.Arrow.Adbc/Tracing): allow ActivitySource tags to be set from TracingConnection [arrow-adbc]

2025-07-30 Thread via GitHub
birschick-bq commented on PR #3218: URL: https://github.com/apache/arrow-adbc/pull/3218#issuecomment-3137449634 > So the idea is to be able to do something like this? `ShouldListenTo = (activitySource) => activitySource.Tags["Hosturl"] = "abc.databricks.com"` If there are mulitple instances

Re: [PR] ci(parquet/pqarrow): integration tests for reading shredded variants DO NOT MERGE YET [arrow-go]

2025-07-30 Thread via GitHub
aihuaxu commented on PR #455: URL: https://github.com/apache/arrow-go/pull/455#issuecomment-3137427446 @zeroshade I regenerated the files with Variant logical type (https://github.com/apache/parquet-testing/pull/91). Can you retest it? -- This is an automated message from the Apache Git

Re: [PR] fix(csharp/src/Drivers/Databricks): Tighten Statement, Reader, Poller Disposal [arrow-adbc]

2025-07-30 Thread via GitHub
toddmeng-db commented on code in PR #3217: URL: https://github.com/apache/arrow-adbc/pull/3217#discussion_r2243551863 ## csharp/src/Drivers/Apache/Hive2/HiveServer2Statement.cs: ## @@ -726,20 +738,15 @@ protected internal QueryResult EnhanceGetColumnsResult(Schema originalSchem

  1   2   >