(arrow) branch main updated (ca0910a533 -> 5b03b707db)
This is an automated email from the ASF dual-hosted git repository. curth pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from ca0910a533 GH-39897: [C++][FS][S3] Ensure `AwsInstance::EnsureInitialized` to do initialization exactly once under concurrency (#40110) add 5b03b707db MINOR: [C#] Bump coverlet.collector from 6.0.0 to 6.0.1 in /csharp (#40245) No new revisions were added by this update. Summary of changes: .../Apache.Arrow.Flight.Sql.Tests/Apache.Arrow.Flight.Sql.Tests.csproj | 2 +- csharp/test/Apache.Arrow.Flight.Tests/Apache.Arrow.Flight.Tests.csproj | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
(arrow) branch dependabot/nuget/csharp/coverlet.collector-6.0.1 deleted (was 63e0d92a0e)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/coverlet.collector-6.0.1 in repository https://gitbox.apache.org/repos/asf/arrow.git was 63e0d92a0e MINOR: [C#] Bump coverlet.collector from 6.0.0 to 6.0.1 in /csharp The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow-datafusion) branch main updated: docs: update parquet_sql_multiple_files.rs with a relative path ex (#9310)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new b8c6e0bb3f docs: update parquet_sql_multiple_files.rs with a relative
path ex (#9310)
b8c6e0bb3f is described below
commit b8c6e0bb3f3f8ab60cf394b46d6aacc616b67d41
Author: Trent Hauck
AuthorDate: Mon Feb 26 10:05:09 2024 -0800
docs: update parquet_sql_multiple_files.rs with a relative path ex (#9310)
* docs: update parquet_sql_multiple_files.rs with a relative path ex
* style: run cargo fmt
* docs: update comment
* docs: better
---
.../examples/parquet_sql_multiple_files.rs | 66 ++
1 file changed, 56 insertions(+), 10 deletions(-)
diff --git a/datafusion-examples/examples/parquet_sql_multiple_files.rs
b/datafusion-examples/examples/parquet_sql_multiple_files.rs
index 451de96f2e..0e2968f203 100644
--- a/datafusion-examples/examples/parquet_sql_multiple_files.rs
+++ b/datafusion-examples/examples/parquet_sql_multiple_files.rs
@@ -17,31 +17,35 @@
use datafusion::datasource::file_format::parquet::ParquetFormat;
use datafusion::datasource::listing::ListingOptions;
-use datafusion::error::Result;
use datafusion::prelude::*;
-use datafusion_common::{FileType, GetExt};
+use object_store::local::LocalFileSystem;
+use std::path::Path;
use std::sync::Arc;
/// This example demonstrates executing a simple query against an Arrow data
source (a directory
-/// with multiple Parquet files) and fetching results
+/// with multiple Parquet files) and fetching results. The query is run twice,
once showing
+/// how to used `register_listing_table` with an absolute path, and once
registering an
+/// ObjectStore to use a relative path.
#[tokio::main]
-async fn main() -> Result<()> {
+async fn main() -> Result<(), Box> {
// create local execution context
let ctx = SessionContext::new();
-let testdata = datafusion::test_util::parquet_test_data();
+let test_data = datafusion::test_util::parquet_test_data();
// Configure listing options
let file_format = ParquetFormat::default().with_enable_pruning(Some(true));
let listing_options = ListingOptions::new(Arc::new(file_format))
-.with_file_extension(FileType::PARQUET.get_ext());
+// This is a workaround for this example since `test_data` contains
+// many different parquet different files,
+// in practice use FileType::PARQUET.get_ext().
+.with_file_extension("alltypes_plain.parquet");
-// Register a listing table - this will use all files in the directory as
data sources
-// for the query
+// First example were we use an absolute path, which requires no
additional setup.
ctx.register_listing_table(
"my_table",
-!("file://{testdata}/alltypes_plain.parquet"),
-listing_options,
+!("file://{test_data}/"),
+listing_options.clone(),
None,
None,
)
@@ -60,5 +64,47 @@ async fn main() -> Result<()> {
// print the results
df.show().await?;
+// Second example were we temporarily move into the test data's parent
directory and
+// simulate a relative path, this requires registering an ObjectStore.
+let cur_dir = std::env::current_dir()?;
+
+let test_data_path = Path::new(_data);
+let test_data_path_parent = test_data_path
+.parent()
+.ok_or("test_data path needs a parent")?;
+
+std::env::set_current_dir(test_data_path_parent)?;
+
+let local_fs = Arc::new(LocalFileSystem::default());
+
+let u = url::Url::parse("file://./")?;
+ctx.runtime_env().register_object_store(, local_fs);
+
+// Register a listing table - this will use all files in the directory as
data sources
+// for the query
+ctx.register_listing_table(
+"relative_table",
+"./data",
+listing_options.clone(),
+None,
+None,
+)
+.await?;
+
+// execute the query
+let df = ctx
+.sql(
+"SELECT * \
+FROM relative_table \
+LIMIT 1",
+)
+.await?;
+
+// print the results
+df.show().await?;
+
+// Reset the current directory
+std::env::set_current_dir(cur_dir)?;
+
Ok(())
}
(arrow-datafusion) branch main updated: tests: add tests for writing hive-partitioned parquet (#9316)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new a26f583d27 tests: add tests for writing hive-partitioned parquet
(#9316)
a26f583d27 is described below
commit a26f583d2766da746ff30199cc7341227526737f
Author: Trent Hauck
AuthorDate: Mon Feb 26 10:06:11 2024 -0800
tests: add tests for writing hive-partitioned parquet (#9316)
* tests: adds tests associated with #9237
* style: clippy
---
.../src/datasource/physical_plan/parquet/mod.rs| 74 --
datafusion/core/tests/dataframe/mod.rs | 160 -
2 files changed, 158 insertions(+), 76 deletions(-)
diff --git a/datafusion/core/src/datasource/physical_plan/parquet/mod.rs
b/datafusion/core/src/datasource/physical_plan/parquet/mod.rs
index badd870848..3aa1998bde 100644
--- a/datafusion/core/src/datasource/physical_plan/parquet/mod.rs
+++ b/datafusion/core/src/datasource/physical_plan/parquet/mod.rs
@@ -2066,80 +2066,6 @@ mod tests {
Ok(())
}
-#[tokio::test]
-async fn write_parquet_results() -> Result<()> {
-// create partitioned input file and context
-let tmp_dir = TempDir::new()?;
-// let mut ctx = create_ctx(_dir, 4).await?;
-let ctx = SessionContext::new_with_config(
-SessionConfig::new().with_target_partitions(8),
-);
-let schema = populate_csv_partitions(_dir, 4, ".csv")?;
-// register csv file with the execution context
-ctx.register_csv(
-"test",
-tmp_dir.path().to_str().unwrap(),
-CsvReadOptions::new().schema(),
-)
-.await?;
-
-// register a local file system object store for /tmp directory
-let local = Arc::new(LocalFileSystem::new_with_prefix(_dir)?);
-let local_url = Url::parse("file://local").unwrap();
-ctx.runtime_env().register_object_store(_url, local);
-
-// execute a simple query and write the results to parquet
-let out_dir = tmp_dir.as_ref().to_str().unwrap().to_string() + "/out/";
-let out_dir_url = "file://local/out/";
-let df = ctx.sql("SELECT c1, c2 FROM test").await?;
-df.write_parquet(out_dir_url, DataFrameWriteOptions::new(), None)
-.await?;
-// write_parquet( ctx, "SELECT c1, c2 FROM test", _dir,
None).await?;
-
-// create a new context and verify that the results were saved to a
partitioned parquet file
-let ctx = SessionContext::new();
-
-// get write_id
-let mut paths = fs::read_dir(_dir).unwrap();
-let path = paths.next();
-let name = path
-.unwrap()?
-.path()
-.file_name()
-.expect("Should be a file name")
-.to_str()
-.expect("Should be a str")
-.to_owned();
-let (parsed_id, _) = name.split_once('_').expect("File should contain
_ !");
-let write_id = parsed_id.to_owned();
-
-// register each partition as well as the top level dir
-ctx.register_parquet(
-"part0",
-!("{out_dir}/{write_id}_0.parquet"),
-ParquetReadOptions::default(),
-)
-.await?;
-
-ctx.register_parquet("allparts", _dir,
ParquetReadOptions::default())
-.await?;
-
-let part0 = ctx.sql("SELECT c1, c2 FROM
part0").await?.collect().await?;
-let allparts = ctx
-.sql("SELECT c1, c2 FROM allparts")
-.await?
-.collect()
-.await?;
-
-let allparts_count: usize = allparts.iter().map(|batch|
batch.num_rows()).sum();
-
-assert_eq!(part0[0].schema(), allparts[0].schema());
-
-assert_eq!(allparts_count, 40);
-
-Ok(())
-}
-
fn logical2physical(expr: , schema: ) -> Arc
{
let df_schema = schema.clone().to_dfschema().unwrap();
let execution_props = ExecutionProps::new();
diff --git a/datafusion/core/tests/dataframe/mod.rs
b/datafusion/core/tests/dataframe/mod.rs
index b08b2b8fc7..ee84200417 100644
--- a/datafusion/core/tests/dataframe/mod.rs
+++ b/datafusion/core/tests/dataframe/mod.rs
@@ -30,15 +30,19 @@ use arrow::{
};
use arrow_array::Float32Array;
use arrow_schema::ArrowError;
+use object_store::local::LocalFileSystem;
+use std::fs;
use std::sync::Arc;
+use tempfile::TempDir;
+use url::Url;
-use datafusion::dataframe::DataFrame;
+use datafusion::dataframe::{DataFrame, DataFrameWriteOptions};
use datafusion::datasource::MemTable;
use datafusion::error::Result;
use datafusion::execution::context::{SessionContext, SessionState};
use datafusion::prelude::JoinType;
use datafusion::prelude::{CsvReadOptions, ParquetReadOptions};
-use datafusion::test_util::parquet_test_data;
+use
(arrow) branch dependabot/nuget/csharp/Grpc.Tools-2.62.0 deleted (was cb61d5ec57)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/Grpc.Tools-2.62.0 in repository https://gitbox.apache.org/repos/asf/arrow.git was cb61d5ec57 MINOR: [C#] Bump Grpc.Tools from 2.60.0 to 2.62.0 in /csharp The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow) branch main updated (5b03b707db -> 43daa32d74)
This is an automated email from the ASF dual-hosted git repository. curth pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from 5b03b707db MINOR: [C#] Bump coverlet.collector from 6.0.0 to 6.0.1 in /csharp (#40245) add 43daa32d74 MINOR: [C#] Bump Grpc.Tools from 2.60.0 to 2.62.0 in /csharp (#40246) No new revisions were added by this update. Summary of changes: csharp/src/Apache.Arrow.Flight.Sql/Apache.Arrow.Flight.Sql.csproj | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(arrow) branch main updated: GH-39582: [C++][Acero] Increase size of Acero TempStack (#40007)
This is an automated email from the ASF dual-hosted git repository.
apitrou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 9a7662b41b GH-39582: [C++][Acero] Increase size of Acero TempStack
(#40007)
9a7662b41b is described below
commit 9a7662b41b77a40a76b07435b3eff4fec7454596
Author: Sten Larsson
AuthorDate: Mon Feb 26 18:02:44 2024 +0100
GH-39582: [C++][Acero] Increase size of Acero TempStack (#40007)
We have had problems for a long time with a specific batch job that
combines data from different sources. There is something in the data causing an
Acero execution plan to hang or crash at random. The problem has been
reproduced since Arrow 11.0.0, originally in Ruby, but it has also in Python.
There is unfortunately no test case that reliably reproduces the issue in a
release build.
However, in a debug build we can see that the batch job causes an overflow
on the temp stack in arrow/cpp/src/arrow/compute/util.cc:38. Increasing the
size of the stack created in the Acero QueryContext works around the issue, but
a real fix should be investigated separately.
**This PR contains a "Critical Fix".**
* Closes: #39582
Lead-authored-by: Sten Larsson
Co-authored-by: Antoine Pitrou
Signed-off-by: Antoine Pitrou
---
cpp/src/arrow/acero/query_context.cc | 2 +-
cpp/src/arrow/compute/util.cc| 15 ---
2 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/cpp/src/arrow/acero/query_context.cc
b/cpp/src/arrow/acero/query_context.cc
index 9f838508fc..a27397d120 100644
--- a/cpp/src/arrow/acero/query_context.cc
+++ b/cpp/src/arrow/acero/query_context.cc
@@ -53,7 +53,7 @@ size_t QueryContext::max_concurrency() const { return
thread_indexer_.Capacity()
Result QueryContext::GetTempStack(size_t thread_index)
{
if (!tld_[thread_index].is_init) {
RETURN_NOT_OK(tld_[thread_index].stack.Init(
-memory_pool(), 8 * util::MiniBatch::kMiniBatchLength *
sizeof(uint64_t)));
+memory_pool(), 32 * util::MiniBatch::kMiniBatchLength *
sizeof(uint64_t)));
tld_[thread_index].is_init = true;
}
return _[thread_index].stack;
diff --git a/cpp/src/arrow/compute/util.cc b/cpp/src/arrow/compute/util.cc
index c55143af0c..2058ba9f30 100644
--- a/cpp/src/arrow/compute/util.cc
+++ b/cpp/src/arrow/compute/util.cc
@@ -32,17 +32,18 @@ using internal::CpuInfo;
namespace util {
void TempVectorStack::alloc(uint32_t num_bytes, uint8_t** data, int* id) {
- int64_t old_top = top_;
- top_ += PaddedAllocationSize(num_bytes) + 2 * sizeof(uint64_t);
- // Stack overflow check
- ARROW_DCHECK(top_ <= buffer_size_);
- *data = buffer_->mutable_data() + old_top + sizeof(uint64_t);
+ int64_t new_top = top_ + PaddedAllocationSize(num_bytes) + 2 *
sizeof(uint64_t);
+ // Stack overflow check (see GH-39582).
+ // XXX cannot return a regular Status because most consumers do not either.
+ ARROW_CHECK_LE(new_top, buffer_size_) << "TempVectorStack::alloc overflow";
+ *data = buffer_->mutable_data() + top_ + sizeof(uint64_t);
// We set 8 bytes before the beginning of the allocated range and
// 8 bytes after the end to check for stack overflow (which would
// result in those known bytes being corrupted).
- reinterpret_cast(buffer_->mutable_data() + old_top)[0] = kGuard1;
- reinterpret_cast(buffer_->mutable_data() + top_)[-1] = kGuard2;
+ reinterpret_cast(buffer_->mutable_data() + top_)[0] = kGuard1;
+ reinterpret_cast(buffer_->mutable_data() + new_top)[-1] = kGuard2;
*id = num_vectors_++;
+ top_ = new_top;
}
void TempVectorStack::release(int id, uint32_t num_bytes) {
(arrow) branch dependabot/nuget/csharp/coverlet.collector-6.0.1 created (now 63e0d92a0e)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/coverlet.collector-6.0.1 in repository https://gitbox.apache.org/repos/asf/arrow.git at 63e0d92a0e MINOR: [C#] Bump coverlet.collector from 6.0.0 to 6.0.1 in /csharp No new revisions were added by this update.
(arrow) branch main updated (8ec7044824 -> a7ac7e0e10)
This is an automated email from the ASF dual-hosted git repository. apitrou pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from 8ec7044824 GH-40236: [Python][CI] Disable generating C lines in Cython tracebacks (#40225) add a7ac7e0e10 GH-40068: [C++] Possible data race when reading metadata of a parquet file (#40111) No new revisions were added by this update. Summary of changes: cpp/src/arrow/dataset/file_parquet.cc | 5 + cpp/src/arrow/dataset/file_parquet.h | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-)
(arrow) branch dependabot/nuget/csharp/Grpc.Tools-2.62.0 created (now cb61d5ec57)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/Grpc.Tools-2.62.0 in repository https://gitbox.apache.org/repos/asf/arrow.git at cb61d5ec57 MINOR: [C#] Bump Grpc.Tools from 2.60.0 to 2.62.0 in /csharp No new revisions were added by this update.
(arrow-datafusion-comet) branch main updated: fix: Another attempt to fix libcrypto.dylib loading issue (#112)
This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 7be5a18 fix: Another attempt to fix libcrypto.dylib loading issue
(#112)
7be5a18 is described below
commit 7be5a1806af6bf213c8cb7739c0263f1fb6ab8fa
Author: advancedxy
AuthorDate: Tue Feb 27 01:04:52 2024 +0800
fix: Another attempt to fix libcrypto.dylib loading issue (#112)
---
.github/actions/setup-macos-builder/action.yaml | 9 +
.github/workflows/pr_build.yml | 2 +-
2 files changed, 6 insertions(+), 5 deletions(-)
diff --git a/.github/actions/setup-macos-builder/action.yaml
b/.github/actions/setup-macos-builder/action.yaml
index 63010ea..cc1b631 100644
--- a/.github/actions/setup-macos-builder/action.yaml
+++ b/.github/actions/setup-macos-builder/action.yaml
@@ -49,12 +49,13 @@ runs:
unzip $PROTO_ZIP
echo "$HOME/d/protoc/bin" >> $GITHUB_PATH
export PATH=$PATH:$HOME/d/protoc/bin
-# install openssl and setup DYLD_LIBRARY_PATH to work with
libcrypto.dylib loading issues with x86_64 mac runners
-# see PR https://github.com/apache/arrow-datafusion-comet/pull/55 for
more details
+# install openssl and setup DYLD_LIBRARY_PATH
brew install openssl
-OPENSSL_LIB_PATH=$(dirname `brew list openssl | grep
'lib/libcrypto.dylib'`)
+OPENSSL_LIB_PATH=`brew --prefix openssl`/lib
echo "openssl lib path is: ${OPENSSL_LIB_PATH}"
-export DYLD_LIBRARY_PATH=$OPENSSL_LIB_PATH:$DYLD_LIBRARY_PATH
+echo "DYLD_LIBRARY_PATH=$OPENSSL_LIB_PATH:$DYLD_LIBRARY_PATH" >>
$GITHUB_ENV
+# output the current status of SIP for later debugging
+csrutil status || true
- name: Install JDK ${{inputs.jdk-version}}
uses: actions/setup-java@v4
diff --git a/.github/workflows/pr_build.yml b/.github/workflows/pr_build.yml
index 669eddd..fe83032 100644
--- a/.github/workflows/pr_build.yml
+++ b/.github/workflows/pr_build.yml
@@ -70,7 +70,7 @@ jobs:
macos-test:
strategy:
matrix:
-os: [macos-latest]
+os: [macos-13]
java_version: [8, 11, 17]
test-target: [rust, java]
is_push_event:
(arrow) branch dependabot/maven/java/org.hamcrest-hamcrest-core-2.2 created (now 66da56e524)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/org.hamcrest-hamcrest-core-2.2 in repository https://gitbox.apache.org/repos/asf/arrow.git at 66da56e524 MINOR: [Java] Bump org.hamcrest:hamcrest-core from 1.3 to 2.2 in /java No new revisions were added by this update.
(arrow) branch main updated (9a7662b41b -> ca0910a533)
This is an automated email from the ASF dual-hosted git repository. apitrou pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from 9a7662b41b GH-39582: [C++][Acero] Increase size of Acero TempStack (#40007) add ca0910a533 GH-39897: [C++][FS][S3] Ensure `AwsInstance::EnsureInitialized` to do initialization exactly once under concurrency (#40110) No new revisions were added by this update. Summary of changes: cpp/src/arrow/filesystem/s3fs.cc | 15 ++- python/pyarrow/tests/test_fs.py | 23 +++ 2 files changed, 33 insertions(+), 5 deletions(-)
(arrow) branch main updated (b8fff043c6 -> 8ec7044824)
This is an automated email from the ASF dual-hosted git repository. apitrou pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from b8fff043c6 GH-40221: [C++][CMake] Use arrow/util/config.h.cmake instead of add_definitions() (#40222) add 8ec7044824 GH-40236: [Python][CI] Disable generating C lines in Cython tracebacks (#40225) No new revisions were added by this update. Summary of changes: python/CMakeLists.txt | 3 +++ 1 file changed, 3 insertions(+)
(arrow-datafusion) branch main updated (ec86acbc1f -> c56840734c)
This is an automated email from the ASF dual-hosted git repository. comphead pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git from ec86acbc1f feat: expand `unnest` to accept arbitrary single array expression (#9342) add c56840734c fix: flight examples (#9335) No new revisions were added by this update. Summary of changes: ci/scripts/rust_example.sh | 1 + datafusion-examples/Cargo.toml | 27 +- .../examples/external_dependency/catalog.rs| 4 ++-- .../examples/flight/flight_sql_server.rs | 4 ++-- 4 files changed, 31 insertions(+), 5 deletions(-)
(arrow-datafusion-comet) branch main updated: build: Show time duration for scala test (#116)
This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new a2b1f9b build: Show time duration for scala test (#116)
a2b1f9b is described below
commit a2b1f9bea381eb61355a16cd509e9383a0bf7abc
Author: advancedxy
AuthorDate: Tue Feb 27 01:46:32 2024 +0800
build: Show time duration for scala test (#116)
---
pom.xml | 1 +
1 file changed, 1 insertion(+)
diff --git a/pom.xml b/pom.xml
index e46a49a..5a96eae 100644
--- a/pom.xml
+++ b/pom.xml
@@ -641,6 +641,7 @@ under the License.
${project.build.directory}/surefire-reports
.
SparkTestSuite.txt
+D
org.apache.comet.IntegrationTestSuite
-ea -Xmx4g -Xss4m ${extraJavaTestArgs}
(arrow-datafusion-comet) branch main updated: feat: Add license header by spotless:apply automatically (#110)
This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new f359ed7 feat: Add license header by spotless:apply automatically
(#110)
f359ed7 is described below
commit f359ed79a3dc710fadbc0484944f280c55c181e1
Author: advancedxy
AuthorDate: Tue Feb 27 01:47:02 2024 +0800
feat: Add license header by spotless:apply automatically (#110)
---
dev/copyright/java-header.txt | 19
dev/copyright/scala-header.txt | 1 +
pom.xml| 6 +
.../org/apache/spark/sql/comet/CometPlan.scala | 26 --
.../src/test/scala/org/apache/spark/sql/TPCH.scala | 10 -
.../test/scala/org/apache/spark/sql/Tables.scala | 13 +--
.../spark/sql/comet/CometPlanStabilitySuite.scala | 1 +
7 files changed, 52 insertions(+), 24 deletions(-)
diff --git a/dev/copyright/java-header.txt b/dev/copyright/java-header.txt
new file mode 100644
index 000..bd244d0
--- /dev/null
+++ b/dev/copyright/java-header.txt
@@ -0,0 +1,19 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
diff --git a/dev/copyright/scala-header.txt b/dev/copyright/scala-header.txt
new file mode 12
index 000..372bb40
--- /dev/null
+++ b/dev/copyright/scala-header.txt
@@ -0,0 +1 @@
+java-header.txt
\ No newline at end of file
diff --git a/pom.xml b/pom.xml
index 5a96eae..d7cd076 100644
--- a/pom.xml
+++ b/pom.xml
@@ -723,6 +723,9 @@ under the License.
java|javax,scala,org,org.apache,com,org.apache.comet,\#,\#org.apache.comet
+
+
${maven.multiModuleProjectDirectory}/dev/copyright/java-header.txt
+
@@ -730,6 +733,9 @@ under the License.
3.6.1
${maven.multiModuleProjectDirectory}/scalafmt.conf
+
+
${maven.multiModuleProjectDirectory}/dev/copyright/scala-header.txt
+
diff --git a/spark/src/main/scala/org/apache/spark/sql/comet/CometPlan.scala
b/spark/src/main/scala/org/apache/spark/sql/comet/CometPlan.scala
index fe2ce7e..e5d268c 100644
--- a/spark/src/main/scala/org/apache/spark/sql/comet/CometPlan.scala
+++ b/spark/src/main/scala/org/apache/spark/sql/comet/CometPlan.scala
@@ -1,18 +1,20 @@
/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
*
- *http://www.apache.org/licenses/LICENSE-2.0
+ * http://www.apache.org/licenses/LICENSE-2.0
*
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
*/
(arrow) branch dependabot/maven/java/de.huxhorn.lilith-de.huxhorn.lilith.logback.appender.multiplex-classic-8.3.0 created (now 0bf1a999e7)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/de.huxhorn.lilith-de.huxhorn.lilith.logback.appender.multiplex-classic-8.3.0 in repository https://gitbox.apache.org/repos/asf/arrow.git at 0bf1a999e7 MINOR: [Java] Bump de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic No new revisions were added by this update.
(arrow) branch dependabot/maven/java/commons-codec-commons-codec-1.16.1 created (now 8fe87e428c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/commons-codec-commons-codec-1.16.1 in repository https://gitbox.apache.org/repos/asf/arrow.git at 8fe87e428c MINOR: [Java] Bump commons-codec:commons-codec in /java No new revisions were added by this update.
(arrow) branch dependabot/maven/java/org.apache.maven.plugins-maven-compiler-plugin-3.12.1 created (now 8cb7dcf422)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/org.apache.maven.plugins-maven-compiler-plugin-3.12.1 in repository https://gitbox.apache.org/repos/asf/arrow.git at 8cb7dcf422 MINOR: [Java] Bump org.apache.maven.plugins:maven-compiler-plugin No new revisions were added by this update.
(arrow) branch dependabot/maven/java/org.apache.maven.plugins-maven-site-plugin-3.12.1 created (now 7faa001981)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/org.apache.maven.plugins-maven-site-plugin-3.12.1 in repository https://gitbox.apache.org/repos/asf/arrow.git at 7faa001981 MINOR: [Java] Bump org.apache.maven.plugins:maven-site-plugin in /java No new revisions were added by this update.
(arrow-adbc) branch main updated: chore(csharp): bump Google.Cloud.BigQuery.V2 from 3.5.0 to 3.6.0 in /csharp (#1567)
This is an automated email from the ASF dual-hosted git repository. curth pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git The following commit(s) were added to refs/heads/main by this push: new 1c9912af chore(csharp): bump Google.Cloud.BigQuery.V2 from 3.5.0 to 3.6.0 in /csharp (#1567) 1c9912af is described below commit 1c9912afea7458916125f597cb983fc9ef611b3c Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Feb 26 12:21:43 2024 -0800 chore(csharp): bump Google.Cloud.BigQuery.V2 from 3.5.0 to 3.6.0 in /csharp (#1567) Bumps [Google.Cloud.BigQuery.V2](https://github.com/googleapis/google-cloud-dotnet) from 3.5.0 to 3.6.0. Commits https://github.com/googleapis/google-cloud-dotnet/commit/2f95acef888d1e6b1dbe775236bed8c42c577e83;>2f95ace Release Google.Cloud.BigQuery.V2 version 3.6.0 (https://redirect.github.com/googleapis/google-cloud-dotnet/issues/11767;>#11767) https://github.com/googleapis/google-cloud-dotnet/commit/efa6b92e6491ee6947c2e20b6cbd64d6def384f8;>efa6b92 Release Google.Cloud.BigQuery.Storage.V1 version 3.12.0 (https://redirect.github.com/googleapis/google-cloud-dotnet/issues/11768;>#11768) https://github.com/googleapis/google-cloud-dotnet/commit/7c24c793439c0ec85a16102d9a2a1a44d69f87b7;>7c24c79 Release Google.Cloud.BigQuery.DataTransfer.V1 version 4.5.0 (https://redirect.github.com/googleapis/google-cloud-dotnet/issues/11766;>#11766) https://github.com/googleapis/google-cloud-dotnet/commit/f85ba561e54aa036f1c64216a5bd3a44b91c5772;>f85ba56 Release Google.Apps.Meet.V2 version 1.0.0-beta01 (https://redirect.github.com/googleapis/google-cloud-dotnet/issues/11765;>#11765) https://github.com/googleapis/google-cloud-dotnet/commit/9677fcb585130adead6bbb1963477dc77be13f57;>9677fcb feat: Implement int64-based timestamps https://github.com/googleapis/google-cloud-dotnet/commit/c71087b85be943b71af130ec440ccc73d6a2f95b;>c71087b chore: Update BigQuery to use latest Discovery-generated version https://github.com/googleapis/google-cloud-dotnet/commit/55db110ba7ed23644b93130960b980cd12c4aebf;>55db110 chore: Remove extraneous whitespace https://github.com/googleapis/google-cloud-dotnet/commit/9f01c3100b3ca37f461b8f76d36ec57c9549b266;>9f01c31 chore: Remove ListRowsOptions.ToGetQueryResultsOptions https://github.com/googleapis/google-cloud-dotnet/commit/cb3035dd532f97eb1a791ae8f90bd17681ff3ff9;>cb3035d chore: Add example issue of uploading object https://github.com/googleapis/google-cloud-dotnet/commit/289fa6e1f785e4da53676b952eefe2f7182ef4c2;>289fa6e feat: add session token support for Autocomplete (New) sessions that end with... Additional commits viewable in https://github.com/googleapis/google-cloud-dotnet/compare/Google.Cloud.BigQuery.V2-3.5.0...Google.Cloud.BigQuery.V2-3.6.0;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
(arrow-adbc) branch dependabot/nuget/csharp/Google.Cloud.BigQuery.V2-3.6.0 deleted (was 765a305d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/Google.Cloud.BigQuery.V2-3.6.0 in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git was 765a305d chore(csharp): bump Google.Cloud.BigQuery.V2 in /csharp The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow-datafusion) branch main updated: feature: support nvl(ifnull) function (#9284)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new b55d0edb0e feature: support nvl(ifnull) function (#9284)
b55d0edb0e is described below
commit b55d0edb0e8b9e8ef8cfaaca164dd5ae93a46854
Author: junxiangMu <63799833+guoji...@users.noreply.github.com>
AuthorDate: Tue Feb 27 04:31:51 2024 +0800
feature: support nvl(ifnull) function (#9284)
* feature: support nvl(ifnull) function
* add sqllogictest
* add docs entry
* Update docs/source/user-guide/sql/scalar_functions.md
Co-authored-by: Jonah Gao
* fix some code
* fix docs
-
Co-authored-by: Jonah Gao
---
datafusion/functions/src/core/mod.rs | 5 +-
datafusion/functions/src/core/nvl.rs | 277 +
datafusion/sqllogictest/test_files/nvl.slt | 120 +++
docs/source/user-guide/sql/scalar_functions.md | 21 ++
4 files changed, 422 insertions(+), 1 deletion(-)
diff --git a/datafusion/functions/src/core/mod.rs
b/datafusion/functions/src/core/mod.rs
index 9aab4bd450..db47c62218 100644
--- a/datafusion/functions/src/core/mod.rs
+++ b/datafusion/functions/src/core/mod.rs
@@ -18,12 +18,15 @@
//! "core" DataFusion functions
mod nullif;
+mod nvl;
// create UDFs
make_udf_function!(nullif::NullIfFunc, NULLIF, nullif);
+make_udf_function!(nvl::NVLFunc, NVL, nvl);
// Export the functions out of this package, both as expr_fn as well as a list
of functions
export_functions!(
-(nullif, arg_1 arg_2, "returns NULL if value1 equals value2; otherwise it
returns value1. This can be used to perform the inverse operation of the
COALESCE expression.")
+(nullif, arg_1 arg_2, "returns NULL if value1 equals value2; otherwise it
returns value1. This can be used to perform the inverse operation of the
COALESCE expression."),
+(nvl, arg_1 arg_2, "returns value2 if value1 is NULL; otherwise it returns
value1")
);
diff --git a/datafusion/functions/src/core/nvl.rs
b/datafusion/functions/src/core/nvl.rs
new file mode 100644
index 00..6d6ad1cdeb
--- /dev/null
+++ b/datafusion/functions/src/core/nvl.rs
@@ -0,0 +1,277 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::datatypes::DataType;
+use datafusion_common::{internal_err, Result, DataFusionError};
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl, Signature, Volatility};
+use arrow::compute::kernels::zip::zip;
+use arrow::compute::is_not_null;
+use arrow::array::Array;
+
+#[derive(Debug)]
+pub(super) struct NVLFunc {
+signature: Signature,
+aliases: Vec,
+}
+
+/// Currently supported types by the nvl/ifnull function.
+/// The order of these types correspond to the order on which coercion applies
+/// This should thus be from least informative to most informative
+static SUPPORTED_NVL_TYPES: &[DataType] = &[
+DataType::Boolean,
+DataType::UInt8,
+DataType::UInt16,
+DataType::UInt32,
+DataType::UInt64,
+DataType::Int8,
+DataType::Int16,
+DataType::Int32,
+DataType::Int64,
+DataType::Float32,
+DataType::Float64,
+DataType::Utf8,
+DataType::LargeUtf8,
+];
+
+impl NVLFunc {
+pub fn new() -> Self {
+Self {
+signature:
+Signature::uniform(2, SUPPORTED_NVL_TYPES.to_vec(),
+Volatility::Immutable,
+),
+aliases: vec![String::from("ifnull")],
+}
+}
+}
+
+impl ScalarUDFImpl for NVLFunc {
+fn as_any() -> std::any::Any {
+self
+}
+
+fn name() -> {
+"nvl"
+}
+
+fn signature() -> {
+
+}
+
+fn return_type(, arg_types: &[DataType]) -> Result {
+// NVL has two args and they might get coerced, get a preview of this
+let coerced_types =
datafusion_expr::type_coercion::functions::data_types(arg_types,
);
+coerced_types.map(|typs| typs[0].clone())
+.map_err(|e| e.context("Failed to coerce arguments for NVL")
+)
+}
+
+fn invoke(, args: &[ColumnarValue])
(arrow-adbc) branch dependabot/nuget/csharp/Google.Cloud.BigQuery.Storage.V1-3.12.0 created (now 8a884703)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/Google.Cloud.BigQuery.Storage.V1-3.12.0 in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git at 8a884703 chore(csharp): bump Google.Cloud.BigQuery.Storage.V1 in /csharp No new revisions were added by this update.
(arrow-adbc) branch dependabot/nuget/csharp/Google.Cloud.BigQuery.V2-3.6.0 created (now 765a305d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/Google.Cloud.BigQuery.V2-3.6.0 in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git at 765a305d chore(csharp): bump Google.Cloud.BigQuery.V2 in /csharp No new revisions were added by this update.
(arrow-adbc) branch asf-site updated: publish documentation
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git The following commit(s) were added to refs/heads/asf-site by this push: new d998159c publish documentation d998159c is described below commit d998159cbd8f63fe4657d85e901d17340e7b215e Author: github-actions[bot] AuthorDate: Mon Feb 26 23:32:59 2024 + publish documentation --- main/java/api/allclasses-index.html | 2 +- main/java/api/allpackages-index.html| 2 +- main/java/api/constant-values.html | 2 +- main/java/api/deprecated-list.html | 2 +- main/java/api/help-doc.html | 2 +- main/java/api/index-all.html| 2 +- main/java/api/index.html| 2 +- .../org/apache/arrow/adbc/core/AdbcConnection.GetObjectsDepth.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcConnection.html| 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcDatabase.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcDriver.html| 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcException.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcInfoCode.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcOptions.html | 2 +- .../org/apache/arrow/adbc/core/AdbcStatement.PartitionResult.html | 2 +- .../api/org/apache/arrow/adbc/core/AdbcStatement.QueryResult.html | 2 +- .../api/org/apache/arrow/adbc/core/AdbcStatement.UpdateResult.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcStatement.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcStatusCode.html| 2 +- main/java/api/org/apache/arrow/adbc/core/BulkIngestMode.html| 2 +- main/java/api/org/apache/arrow/adbc/core/ErrorDetail.html | 2 +- main/java/api/org/apache/arrow/adbc/core/IsolationLevel.html| 2 +- main/java/api/org/apache/arrow/adbc/core/PartitionDescriptor.html | 2 +- main/java/api/org/apache/arrow/adbc/core/StandardSchemas.html | 2 +- main/java/api/org/apache/arrow/adbc/core/StandardStatistics.html| 2 +- main/java/api/org/apache/arrow/adbc/core/TypedKey.html | 2 +- .../arrow/adbc/core/class-use/AdbcConnection.GetObjectsDepth.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/AdbcConnection.html| 2 +- .../java/api/org/apache/arrow/adbc/core/class-use/AdbcDatabase.html | 2 +- main/java/api/org/apache/arrow/adbc/core/class-use/AdbcDriver.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/AdbcException.html | 2 +- .../java/api/org/apache/arrow/adbc/core/class-use/AdbcInfoCode.html | 2 +- main/java/api/org/apache/arrow/adbc/core/class-use/AdbcOptions.html | 2 +- .../arrow/adbc/core/class-use/AdbcStatement.PartitionResult.html| 2 +- .../apache/arrow/adbc/core/class-use/AdbcStatement.QueryResult.html | 2 +- .../arrow/adbc/core/class-use/AdbcStatement.UpdateResult.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/AdbcStatement.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/AdbcStatusCode.html| 2 +- .../api/org/apache/arrow/adbc/core/class-use/BulkIngestMode.html| 2 +- main/java/api/org/apache/arrow/adbc/core/class-use/ErrorDetail.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/IsolationLevel.html| 2 +- .../org/apache/arrow/adbc/core/class-use/PartitionDescriptor.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/StandardSchemas.html | 2 +- .../org/apache/arrow/adbc/core/class-use/StandardStatistics.html| 2 +- main/java/api/org/apache/arrow/adbc/core/class-use/TypedKey.html| 2 +- main/java/api/org/apache/arrow/adbc/core/package-summary.html | 2 +- main/java/api/org/apache/arrow/adbc/core/package-tree.html | 2 +- main/java/api/org/apache/arrow/adbc/core/package-use.html | 2 +- .../org/apache/arrow/adbc/driver/flightsql/BaseFlightReader.html| 2 +- .../org/apache/arrow/adbc/driver/flightsql/FlightInfoReader.html| 2 +- .../arrow/adbc/driver/flightsql/FlightSqlClientWithCallOptions.html | 2 +- .../org/apache/arrow/adbc/driver/flightsql/FlightSqlConnection.html | 2 +- .../arrow/adbc/driver/flightsql/FlightSqlConnectionProperties.html | 2 +- .../org/apache/arrow/adbc/driver/flightsql/FlightSqlDatabase.html | 2 +- .../api/org/apache/arrow/adbc/driver/flightsql/FlightSqlDriver.html | 2 +- .../apache/arrow/adbc/driver/flightsql/FlightSqlDriverFactory.html | 2 +- .../org/apache/arrow/adbc/driver/flightsql/FlightSqlStatement.html | 2 +- .../arrow/adbc/driver/flightsql/class-use/BaseFlightReader.html | 2 +- .../arrow/adbc/driver/flightsql/class-use/FlightInfoReader.html | 2 +-
(arrow) branch dependabot/maven/java/org.hamcrest-hamcrest-core-2.2 updated (66da56e524 -> 411c2f28c6)
This is an automated email from the ASF dual-hosted git repository. kou pushed a change to branch dependabot/maven/java/org.hamcrest-hamcrest-core-2.2 in repository https://gitbox.apache.org/repos/asf/arrow.git from 66da56e524 MINOR: [Java] Bump org.hamcrest:hamcrest-core from 1.3 to 2.2 in /java add 411c2f28c6 hamcrest-core -> hamcrest No new revisions were added by this update. Summary of changes: java/flight/flight-sql-jdbc-core/pom.xml | 2 +- java/flight/flight-sql-jdbc-driver/pom.xml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-)
(arrow-adbc) branch main updated: fix(csharp/src/Client/SchemaConverter): add check for keys on precision and scale (#1566)
This is an automated email from the ASF dual-hosted git repository.
curth pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git
The following commit(s) were added to refs/heads/main by this push:
new b29e4a96 fix(csharp/src/Client/SchemaConverter): add check for keys on
precision and scale (#1566)
b29e4a96 is described below
commit b29e4a96c02ccd8e40c63fcbd32b39e12906ea2a
Author: davidhcoe <13318837+davidh...@users.noreply.github.com>
AuthorDate: Mon Feb 26 18:17:27 2024 -0500
fix(csharp/src/Client/SchemaConverter): add check for keys on precision and
scale (#1566)
fixes https://github.com/apache/arrow-adbc/issues/1565
Co-authored-by: David Coe
---
csharp/src/Client/SchemaConverter.cs | 13 +++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/csharp/src/Client/SchemaConverter.cs
b/csharp/src/Client/SchemaConverter.cs
index 38863417..cc71f499 100644
--- a/csharp/src/Client/SchemaConverter.cs
+++ b/csharp/src/Client/SchemaConverter.cs
@@ -70,8 +70,17 @@ namespace Apache.Arrow.Adbc.Client
f.HasMetadata
)
{
-row[SchemaTableColumn.NumericPrecision] =
Convert.ToInt32(f.Metadata["precision"]);
-row[SchemaTableColumn.NumericScale] =
Convert.ToInt32(f.Metadata["scale"]);
+if (f.Metadata.TryGetValue("precision", out string
precisionKey))
+{
+if(!string.IsNullOrEmpty(precisionKey))
+row[SchemaTableColumn.NumericPrecision] =
Convert.ToInt32(f.Metadata[precisionKey]);
+}
+
+if(f.Metadata.TryGetValue("scale", out string scaleKey))
+{
+if(!string.IsNullOrEmpty(scaleKey))
+row[SchemaTableColumn.NumericScale] =
Convert.ToInt32(f.Metadata[scaleKey]);
+}
}
else if (f.DataType is Decimal128Type decimal128Type)
{
(arrow-adbc) branch main updated: chore(csharp): bump Google.Cloud.BigQuery.Storage.V1 from 3.11.0 to 3.12.0 in /csharp (#1568)
This is an automated email from the ASF dual-hosted git repository. curth pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git The following commit(s) were added to refs/heads/main by this push: new f26ca923 chore(csharp): bump Google.Cloud.BigQuery.Storage.V1 from 3.11.0 to 3.12.0 in /csharp (#1568) f26ca923 is described below commit f26ca92376945b13dbf49363b8ddf7283d1b3dbf Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Mon Feb 26 13:00:50 2024 -0800 chore(csharp): bump Google.Cloud.BigQuery.Storage.V1 from 3.11.0 to 3.12.0 in /csharp (#1568) Bumps [Google.Cloud.BigQuery.Storage.V1](https://github.com/googleapis/google-cloud-dotnet) from 3.11.0 to 3.12.0. Commits https://github.com/googleapis/google-cloud-dotnet/commit/efa6b92e6491ee6947c2e20b6cbd64d6def384f8;>efa6b92 Release Google.Cloud.BigQuery.Storage.V1 version 3.12.0 (https://redirect.github.com/googleapis/google-cloud-dotnet/issues/11768;>#11768) https://github.com/googleapis/google-cloud-dotnet/commit/7c24c793439c0ec85a16102d9a2a1a44d69f87b7;>7c24c79 Release Google.Cloud.BigQuery.DataTransfer.V1 version 4.5.0 (https://redirect.github.com/googleapis/google-cloud-dotnet/issues/11766;>#11766) https://github.com/googleapis/google-cloud-dotnet/commit/f85ba561e54aa036f1c64216a5bd3a44b91c5772;>f85ba56 Release Google.Apps.Meet.V2 version 1.0.0-beta01 (https://redirect.github.com/googleapis/google-cloud-dotnet/issues/11765;>#11765) https://github.com/googleapis/google-cloud-dotnet/commit/9677fcb585130adead6bbb1963477dc77be13f57;>9677fcb feat: Implement int64-based timestamps https://github.com/googleapis/google-cloud-dotnet/commit/c71087b85be943b71af130ec440ccc73d6a2f95b;>c71087b chore: Update BigQuery to use latest Discovery-generated version https://github.com/googleapis/google-cloud-dotnet/commit/55db110ba7ed23644b93130960b980cd12c4aebf;>55db110 chore: Remove extraneous whitespace https://github.com/googleapis/google-cloud-dotnet/commit/9f01c3100b3ca37f461b8f76d36ec57c9549b266;>9f01c31 chore: Remove ListRowsOptions.ToGetQueryResultsOptions https://github.com/googleapis/google-cloud-dotnet/commit/cb3035dd532f97eb1a791ae8f90bd17681ff3ff9;>cb3035d chore: Add example issue of uploading object https://github.com/googleapis/google-cloud-dotnet/commit/289fa6e1f785e4da53676b952eefe2f7182ef4c2;>289fa6e feat: add session token support for Autocomplete (New) sessions that end with... https://github.com/googleapis/google-cloud-dotnet/commit/354560a797a3038d45cab928a5d4e8b0bd0cac91;>354560a docs: refine proto comment for run_as_non_root Additional commits viewable in https://github.com/googleapis/google-cloud-dotnet/compare/Google.Cloud.BigQuery.Storage.V1-3.11.0...Google.Cloud.BigQuery.Storage.V1-3.12.0;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
(arrow-adbc) branch dependabot/nuget/csharp/Google.Cloud.BigQuery.Storage.V1-3.12.0 deleted (was cea7aba1)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/Google.Cloud.BigQuery.Storage.V1-3.12.0 in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git was cea7aba1 chore(csharp): bump Google.Cloud.BigQuery.Storage.V1 in /csharp The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow-adbc) branch asf-site updated: publish documentation
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git The following commit(s) were added to refs/heads/asf-site by this push: new e5f84f31 publish documentation e5f84f31 is described below commit e5f84f31b2d8a91ee677ccd69cd0145c1b61598b Author: github-actions[bot] AuthorDate: Mon Feb 26 21:15:24 2024 + publish documentation --- main/java/api/allclasses-index.html | 2 +- main/java/api/allpackages-index.html| 2 +- main/java/api/constant-values.html | 2 +- main/java/api/deprecated-list.html | 2 +- main/java/api/help-doc.html | 2 +- main/java/api/index-all.html| 2 +- main/java/api/index.html| 2 +- .../org/apache/arrow/adbc/core/AdbcConnection.GetObjectsDepth.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcConnection.html| 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcDatabase.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcDriver.html| 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcException.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcInfoCode.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcOptions.html | 2 +- .../org/apache/arrow/adbc/core/AdbcStatement.PartitionResult.html | 2 +- .../api/org/apache/arrow/adbc/core/AdbcStatement.QueryResult.html | 2 +- .../api/org/apache/arrow/adbc/core/AdbcStatement.UpdateResult.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcStatement.html | 2 +- main/java/api/org/apache/arrow/adbc/core/AdbcStatusCode.html| 2 +- main/java/api/org/apache/arrow/adbc/core/BulkIngestMode.html| 2 +- main/java/api/org/apache/arrow/adbc/core/ErrorDetail.html | 2 +- main/java/api/org/apache/arrow/adbc/core/IsolationLevel.html| 2 +- main/java/api/org/apache/arrow/adbc/core/PartitionDescriptor.html | 2 +- main/java/api/org/apache/arrow/adbc/core/StandardSchemas.html | 2 +- main/java/api/org/apache/arrow/adbc/core/StandardStatistics.html| 2 +- main/java/api/org/apache/arrow/adbc/core/TypedKey.html | 2 +- .../arrow/adbc/core/class-use/AdbcConnection.GetObjectsDepth.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/AdbcConnection.html| 2 +- .../java/api/org/apache/arrow/adbc/core/class-use/AdbcDatabase.html | 2 +- main/java/api/org/apache/arrow/adbc/core/class-use/AdbcDriver.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/AdbcException.html | 2 +- .../java/api/org/apache/arrow/adbc/core/class-use/AdbcInfoCode.html | 2 +- main/java/api/org/apache/arrow/adbc/core/class-use/AdbcOptions.html | 2 +- .../arrow/adbc/core/class-use/AdbcStatement.PartitionResult.html| 2 +- .../apache/arrow/adbc/core/class-use/AdbcStatement.QueryResult.html | 2 +- .../arrow/adbc/core/class-use/AdbcStatement.UpdateResult.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/AdbcStatement.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/AdbcStatusCode.html| 2 +- .../api/org/apache/arrow/adbc/core/class-use/BulkIngestMode.html| 2 +- main/java/api/org/apache/arrow/adbc/core/class-use/ErrorDetail.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/IsolationLevel.html| 2 +- .../org/apache/arrow/adbc/core/class-use/PartitionDescriptor.html | 2 +- .../api/org/apache/arrow/adbc/core/class-use/StandardSchemas.html | 2 +- .../org/apache/arrow/adbc/core/class-use/StandardStatistics.html| 2 +- main/java/api/org/apache/arrow/adbc/core/class-use/TypedKey.html| 2 +- main/java/api/org/apache/arrow/adbc/core/package-summary.html | 2 +- main/java/api/org/apache/arrow/adbc/core/package-tree.html | 2 +- main/java/api/org/apache/arrow/adbc/core/package-use.html | 2 +- .../org/apache/arrow/adbc/driver/flightsql/BaseFlightReader.html| 2 +- .../org/apache/arrow/adbc/driver/flightsql/FlightInfoReader.html| 2 +- .../arrow/adbc/driver/flightsql/FlightSqlClientWithCallOptions.html | 2 +- .../org/apache/arrow/adbc/driver/flightsql/FlightSqlConnection.html | 2 +- .../arrow/adbc/driver/flightsql/FlightSqlConnectionProperties.html | 2 +- .../org/apache/arrow/adbc/driver/flightsql/FlightSqlDatabase.html | 2 +- .../api/org/apache/arrow/adbc/driver/flightsql/FlightSqlDriver.html | 2 +- .../apache/arrow/adbc/driver/flightsql/FlightSqlDriverFactory.html | 2 +- .../org/apache/arrow/adbc/driver/flightsql/FlightSqlStatement.html | 2 +- .../arrow/adbc/driver/flightsql/class-use/BaseFlightReader.html | 2 +- .../arrow/adbc/driver/flightsql/class-use/FlightInfoReader.html | 2 +-
(arrow-rs) branch master updated: Improve docs for logical and physical nulls even more (#5434)
This is an automated email from the ASF dual-hosted git repository.
tustvold pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/master by this push:
new ab590801698 Improve docs for logical and physical nulls even more
(#5434)
ab590801698 is described below
commit ab590801698f9b5f577a26e22dcd91e349e3
Author: Andrew Lamb
AuthorDate: Mon Feb 26 15:07:51 2024 -0500
Improve docs for logical and physical nulls even more (#5434)
* Improve docs for logical and physical nulls even more
* Apply suggestions from code review
---
arrow-array/src/array/mod.rs | 30 +-
1 file changed, 21 insertions(+), 9 deletions(-)
diff --git a/arrow-array/src/array/mod.rs b/arrow-array/src/array/mod.rs
index 1a58598543f..7aa3f92bfbd 100644
--- a/arrow-array/src/array/mod.rs
+++ b/arrow-array/src/array/mod.rs
@@ -175,23 +175,35 @@ pub trait Array: std::fmt::Debug + Send + Sync {
/// Returns the null buffer of this array if any.
///
-/// The null buffer encodes the "physical" nulls of an array.
-/// However, some arrays can also encode nullability in their children,
for example,
-/// [`DictionaryArray::values`] values or [`RunArray::values`], or without
a null buffer,
-/// such as [`NullArray`]. To determine if each element of such an array
is logically null,
-/// you can use the slower [`Array::logical_nulls`] to obtain a computed
mask .
+/// The null buffer contains the "physical" nulls of an array, that is how
+/// the nulls are represented in the underlying arrow format.
+///
+/// The physical representation is efficient, but is sometimes non
intuitive
+/// for certain array types such as those with nullable child arrays like
+/// [`DictionaryArray::values`] or [`RunArray::values`], or without a
+/// null buffer, such as [`NullArray`].
+///
+/// To determine if each element of such an array is "logically" null,
+/// use the slower [`Array::logical_nulls`] to obtain a computed mask.
fn nulls() -> Option<>;
-/// Returns a potentially computed [`NullBuffer`] that represent the
logical null values of this array, if any.
+/// Returns a potentially computed [`NullBuffer`] that represents the
logical
+/// null values of this array, if any.
+///
+/// Logical nulls represent the values that are null in the array,
+/// regardless of the underlying physical arrow representation.
///
-/// In most cases this will be the same as [`Array::nulls`], except for:
+/// For most array types, this is equivalent to the "physical" nulls
+/// returned by [`Array::nulls`]. It is different for the following cases,
because which
+/// elements are null is not encoded in a single null buffer:
///
/// * [`DictionaryArray`] where [`DictionaryArray::values`] contains nulls
/// * [`RunArray`] where [`RunArray::values`] contains nulls
/// * [`NullArray`] where all indices are nulls
///
-/// In these cases a logical [`NullBuffer`] will be computed, encoding the
logical nullability
-/// of these arrays, beyond what is encoded in [`Array::nulls`]
+/// In these cases a logical [`NullBuffer`] will be computed, encoding the
+/// logical nullability of these arrays, beyond what is encoded in
+/// [`Array::nulls`]
fn logical_nulls() -> Option {
self.nulls().cloned()
}
(arrow-adbc) branch dependabot/nuget/csharp/Google.Cloud.BigQuery.Storage.V1-3.12.0 updated (8a884703 -> cea7aba1)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/nuget/csharp/Google.Cloud.BigQuery.Storage.V1-3.12.0 in repository https://gitbox.apache.org/repos/asf/arrow-adbc.git discard 8a884703 chore(csharp): bump Google.Cloud.BigQuery.Storage.V1 in /csharp add 1c9912af chore(csharp): bump Google.Cloud.BigQuery.V2 from 3.5.0 to 3.6.0 in /csharp (#1567) add cea7aba1 chore(csharp): bump Google.Cloud.BigQuery.Storage.V1 in /csharp This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (8a884703) \ N -- N -- N refs/heads/dependabot/nuget/csharp/Google.Cloud.BigQuery.Storage.V1-3.12.0 (cea7aba1) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: csharp/src/Drivers/BigQuery/Apache.Arrow.Adbc.Drivers.BigQuery.csproj | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(arrow) branch main updated: MINOR: [Java] Bump de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic from 0.9.44 to 8.3.0 in /java (#40239)
This is an automated email from the ASF dual-hosted git repository. kou pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/main by this push: new 8940a427a5 MINOR: [Java] Bump de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic from 0.9.44 to 8.3.0 in /java (#40239) 8940a427a5 is described below commit 8940a427a5432469823783c52dc07d939b122acc Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Feb 27 05:46:51 2024 +0900 MINOR: [Java] Bump de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic from 0.9.44 to 8.3.0 in /java (#40239) Bumps [de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic](https://github.com/huxi/lilith) from 0.9.44 to 8.3.0. Changelog Sourced from https://github.com/huxi/lilith/blob/master/CHANGELOG.md;>de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic's changelog. [8.3.0] - 2021-12-11 Changed Demand Java 8 1.8.0_312. Fixed SLF4J 1.7.32, logback 1.2.7, log4j2 2.15.0, Groovy 2.5.15, Spring 5.3.13, jackson 2.13.0, commons-io 2.11.0, commons-text 1.5, commons-lang3 3.12.0, flying-saucer 9.1.18, aspectj 1.9.7, woodstox 6.2.7, httpcore 4.4.14, httpclient 4.5.13, protobuf 3.19.1 [8.2.0] - 2018-08-09 Added Added Find previous active and Find next active buttons to toolbar. Added Find previous and Find next buttons to toolbar. Added lots of missing mnemonics. If the connection is lost then Message/RequestURI in table will now show Connection closed. instead of nothing. Added smooth horizontal table scrolling option that is enabled by default. Added support for Log4j 2 JsonLayout, YamlLayout and XmlLayout. SerializedLayout has been deprecated in log4j2 2.9.0 so you should use one of the other options instead. Added Automatic-Module-Names for artifacts where appropriate. See http://branchandbound.net/blog/java/2017/12/automatic-module-name/;>Automatic-Module-Name: Calling all Java Library Maintainers. Changed Clean all inactive logs is now less noisy in the Lilith log. Changed icons for Find previous active and Find next active. They now differ from Find previous and Find next as they should. Refactored actions and icon handling. Don't add null events to global logs. Unchecking Enable global logs. in Preferences is now deleting existing global log files automatically. Keyboard help will now always be up-to-date. Demand Java 8 1.8.0_181. Deprecated Nothing. Removed Previous and Next buttons in find panel. Pause action. Pausing only paused updating of the table, not receiving of events. This was confusing (even me) and served no actual purpose. This action was a left-over from the early days of Lilith when it was used for debugging during development. Fixed All LF support mac screen menu bar with Java 9 or higher. Zero-delimited event receivers did not add a null event when end of stream was reached. Fixed initial enabled state of Go to source. Fixed enabled state of Edit menu. Mustn't be disabled anymore because Paste StackTraceElement is always available. Fixed enabled state of Copy selection. Menu entries related to global logs are now disabled if Enable global logs. is unchecked in Preferences. Added more dependencies and entries to the deserialization whitelist. This is essentially necessary because logback-access does not have an AccessEventVO. See also http://jira.qos.ch/browse/LOGBACK-1182;>LOGBACK-1182 - Problem deserializing AccessEvent.. Not all event producers expect a heartbeat. Made sure that You have changed the look feel. and You have changed the application path. dialogs aren't hidden by the preferences dialog. Fixed java executable detection in Windows bat file. Thanks, https://github.com/tha2015;>tha2015! Logback 1.2.3, log4j2 2.11.1, Groovy 2.5.1, jackson 2.9.6, spring 5.0.8, protobuf 3.6.1, junique 1.0.4, jcommander 1.72, commons-lang 3.7, commons-text 1.4, commons-io 2.6, flying-saucer 9.1.14, glazedlists 1.10.0, aspectj 1.9.1, httpcore 4.4.10, httpclient 4.5.6, woodstox 5.1.0 Fixed several split package issues. Because of this, some classes have changed package names: the two most commonly used classes de.huxhorn.lilith.logback.appender.ClassicMultiplexSocketAppender and de.huxhorn.lilith.logback.encoder.ClassicLilithEncoder have not been moved. de.huxhorn.lilith.logback.encoder.AccessLilithEncoder changed to de.huxhorn.lilith.logback.encoder.access.AccessLilithEncoder. ... (truncated) Commits https://github.com/huxi/lilith/commit/79c8ec553a7a3fffe64516342435ca1ac0ede990;>79c8ec5 console is not working
(arrow) branch dependabot/maven/java/de.huxhorn.lilith-de.huxhorn.lilith.logback.appender.multiplex-classic-8.3.0 deleted (was 0bf1a999e7)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/de.huxhorn.lilith-de.huxhorn.lilith.logback.appender.multiplex-classic-8.3.0 in repository https://gitbox.apache.org/repos/asf/arrow.git was 0bf1a999e7 MINOR: [Java] Bump de.huxhorn.lilith:de.huxhorn.lilith.logback.appender.multiplex-classic The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow) branch dependabot/maven/java/org.apache.maven.plugins-maven-site-plugin-3.12.1 deleted (was 7faa001981)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/org.apache.maven.plugins-maven-site-plugin-3.12.1 in repository https://gitbox.apache.org/repos/asf/arrow.git was 7faa001981 MINOR: [Java] Bump org.apache.maven.plugins:maven-site-plugin in /java The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow) branch main updated: MINOR: [Java] Bump org.apache.maven.plugins:maven-site-plugin from 3.7.1 to 3.12.1 in /java (#40240)
This is an automated email from the ASF dual-hosted git repository. kou pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git The following commit(s) were added to refs/heads/main by this push: new 229f09730d MINOR: [Java] Bump org.apache.maven.plugins:maven-site-plugin from 3.7.1 to 3.12.1 in /java (#40240) 229f09730d is described below commit 229f09730d83099a22c74de99c0bcf1ecafc219e Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> AuthorDate: Tue Feb 27 05:47:20 2024 +0900 MINOR: [Java] Bump org.apache.maven.plugins:maven-site-plugin from 3.7.1 to 3.12.1 in /java (#40240) Bumps [org.apache.maven.plugins:maven-site-plugin](https://github.com/apache/maven-site-plugin) from 3.7.1 to 3.12.1. Commits https://github.com/apache/maven-site-plugin/commit/ecae28fb0990eb5a7fc8f2d4ffe07f348d927f4b;>ecae28f [maven-release-plugin] prepare release maven-site-plugin-3.12.1 https://github.com/apache/maven-site-plugin/commit/d98569b083ded7a5182bf6cb5814ddcbd3150267;>d98569b [MSITE-908] Upgrade Maven Reporting API to 3.1.1 https://github.com/apache/maven-site-plugin/commit/bd3376f52d0053e05c78327aad39353710702a7a;>bd3376f [MSITE-901] If precending standalone report has been run, site:jar does not r... https://github.com/apache/maven-site-plugin/commit/b99c0ef371774a414ade764f2921bdfe8918ed60;>b99c0ef [MSITE-902] Upgrade Plexus Utils to 3.4.2 https://github.com/apache/maven-site-plugin/commit/3c6ff2e285063231b7042bfe7875871c9d339830;>3c6ff2e Update CI URL https://github.com/apache/maven-site-plugin/commit/f314e9da6ba0b5611fd5dd7dcff2d9ecc36dcd61;>f314e9d [MSITE-898] Upgrade Parent to 36 https://github.com/apache/maven-site-plugin/commit/bce7458375464e58f5d2d6cff92f8dde5f45de67;>bce7458 [MSITE-897] Upgrade Plexus Archiver to 4.2.7 https://github.com/apache/maven-site-plugin/commit/3c8d426aae79c793a2a3acfddbd47a6826346382;>3c8d426 keep only release month, drop day https://github.com/apache/maven-site-plugin/commit/6604ab3b53d3f4045cc755340aeb0c4feaeaf8df;>6604ab3 also keep only Doxia versions changes https://github.com/apache/maven-site-plugin/commit/789a7a1054babde3c5b01e48bbf8abca49f5af8f;>789a7a1 lighten content: keep only meaningful values Additional commits viewable in https://github.com/apache/maven-site-plugin/compare/maven-site-plugin-3.7.1...maven-site-plugin-3.12.1;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@ dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@ dependabot rebase` will rebase this PR - `@ dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@ dependabot merge` will merge this PR after your CI passes on it - `@ dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@ dependabot cancel merge` will cancel a previously requested merge and block automerging - `@ dependabot reopen` will reopen this PR if it is closed - `@ dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@ dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@ dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@ dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) Authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Sutou Kouhei --- java/bom/pom.xml | 4 ++-- java/maven/pom.xml | 4 ++-- java/pom.xml | 6 +++--- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/java/bom/pom.xml b/java/bom/pom.xml index 025632c45a..2406886222 100644 --- a/java/bom/pom.xml +++ b/java/bom/pom.xml @@ -151,7 +151,7 @@ org.apache.maven.plugins maven-site-plugin -3.7.1 +
(arrow) branch dependabot/npm_and_yarn/js/es5-ext-0.10.63 created (now 6c6c17bcf4)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/js/es5-ext-0.10.63 in repository https://gitbox.apache.org/repos/asf/arrow.git at 6c6c17bcf4 MINOR: [JS] Bump es5-ext from 0.10.62 to 0.10.63 in /js No new revisions were added by this update.
(arrow) branch main updated (229f09730d -> 9e3f0f1c1d)
This is an automated email from the ASF dual-hosted git repository. kou pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from 229f09730d MINOR: [Java] Bump org.apache.maven.plugins:maven-site-plugin from 3.7.1 to 3.12.1 in /java (#40240) add 9e3f0f1c1d MINOR: [Java] Bump org.apache.maven.plugins:maven-compiler-plugin from 3.11.0 to 3.12.1 in /java (#40241) No new revisions were added by this update. Summary of changes: java/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(arrow) branch dependabot/maven/java/org.apache.maven.plugins-maven-compiler-plugin-3.12.1 deleted (was 8cb7dcf422)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/org.apache.maven.plugins-maven-compiler-plugin-3.12.1 in repository https://gitbox.apache.org/repos/asf/arrow.git was 8cb7dcf422 MINOR: [Java] Bump org.apache.maven.plugins:maven-compiler-plugin The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow) branch main updated (9e3f0f1c1d -> 8805de7bf2)
This is an automated email from the ASF dual-hosted git repository. kou pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from 9e3f0f1c1d MINOR: [Java] Bump org.apache.maven.plugins:maven-compiler-plugin from 3.11.0 to 3.12.1 in /java (#40241) add 8805de7bf2 MINOR: [Java] Bump commons-codec:commons-codec from 1.16.0 to 1.16.1 in /java (#40242) No new revisions were added by this update. Summary of changes: java/vector/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(arrow) branch dependabot/maven/java/commons-codec-commons-codec-1.16.1 deleted (was 8fe87e428c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/commons-codec-commons-codec-1.16.1 in repository https://gitbox.apache.org/repos/asf/arrow.git was 8fe87e428c MINOR: [Java] Bump commons-codec:commons-codec in /java The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow-datafusion) branch main updated: Move abs to datafusion_functions (#9313)
This is an automated email from the ASF dual-hosted git repository.
jayzhan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new 85f7a8e88e Move abs to datafusion_functions (#9313)
85f7a8e88e is described below
commit 85f7a8e88e3596b3ec900c43fabb8f7f42bbea5c
Author: Junhao Liu
AuthorDate: Mon Feb 26 18:24:16 2024 -0600
Move abs to datafusion_functions (#9313)
* feat: move abs to datafusion_functions
* fix proto
* fix proto
* fix CI vendored code
* Fix proto
* add support type
* fix signature
* fix typo
* fix test cases
* disable a test case
* remove old code from math_expressions
* feat: add test
* fix clippy
* use unknown for proto
* fix unknown proto
---
datafusion/expr/src/built_in_function.rs | 7 -
datafusion/expr/src/expr.rs | 5 -
datafusion/expr/src/expr_fn.rs | 2 -
datafusion/functions/src/math/abs.rs | 177 +++
datafusion/functions/src/math/mod.rs | 8 +-
datafusion/physical-expr/src/functions.rs| 4 -
datafusion/physical-expr/src/math_expressions.rs | 93 +---
datafusion/proto/proto/datafusion.proto | 4 +-
datafusion/proto/src/generated/pbjson.rs | 6 +-
datafusion/proto/src/generated/prost.rs | 8 +-
datafusion/proto/src/logical_plan/from_proto.rs | 6 +-
datafusion/proto/src/logical_plan/to_proto.rs| 1 -
12 files changed, 198 insertions(+), 123 deletions(-)
diff --git a/datafusion/expr/src/built_in_function.rs
b/datafusion/expr/src/built_in_function.rs
index 8b4e65121c..cf1e73f780 100644
--- a/datafusion/expr/src/built_in_function.rs
+++ b/datafusion/expr/src/built_in_function.rs
@@ -42,8 +42,6 @@ use strum_macros::EnumIter;
#[derive(Debug, Clone, PartialEq, Eq, Hash, EnumIter, Copy)]
pub enum BuiltinScalarFunction {
// math functions
-/// abs
-Abs,
/// acos
Acos,
/// asin
@@ -364,7 +362,6 @@ impl BuiltinScalarFunction {
pub fn volatility() -> Volatility {
match self {
// Immutable scalar builtins
-BuiltinScalarFunction::Abs => Volatility::Immutable,
BuiltinScalarFunction::Acos => Volatility::Immutable,
BuiltinScalarFunction::Asin => Volatility::Immutable,
BuiltinScalarFunction::Atan => Volatility::Immutable,
@@ -868,8 +865,6 @@ impl BuiltinScalarFunction {
BuiltinScalarFunction::ArrowTypeof => Ok(Utf8),
-BuiltinScalarFunction::Abs => Ok(input_expr_types[0].clone()),
-
BuiltinScalarFunction::OverLay => {
utf8_to_str_type(_expr_types[0], "overlay")
}
@@ -1338,7 +1333,6 @@ impl BuiltinScalarFunction {
Signature::uniform(2, vec![Int64], self.volatility())
}
BuiltinScalarFunction::ArrowTypeof => Signature::any(1,
self.volatility()),
-BuiltinScalarFunction::Abs => Signature::any(1, self.volatility()),
BuiltinScalarFunction::OverLay => Signature::one_of(
vec![
Exact(vec![Utf8, Utf8, Int64, Int64]),
@@ -1444,7 +1438,6 @@ impl BuiltinScalarFunction {
/// Returns all names that can be used to call this function
pub fn aliases() -> &'static [&'static str] {
match self {
-BuiltinScalarFunction::Abs => &["abs"],
BuiltinScalarFunction::Acos => &["acos"],
BuiltinScalarFunction::Acosh => &["acosh"],
BuiltinScalarFunction::Asin => &["asin"],
diff --git a/datafusion/expr/src/expr.rs b/datafusion/expr/src/expr.rs
index f40ccb6cdb..c3d9269d15 100644
--- a/datafusion/expr/src/expr.rs
+++ b/datafusion/expr/src/expr.rs
@@ -2033,11 +2033,6 @@ mod test {
.is_volatile()
.unwrap()
);
-assert!(
-!ScalarFunctionDefinition::BuiltIn(BuiltinScalarFunction::Abs)
-.is_volatile()
-.unwrap()
-);
// UDF
#[derive(Debug)]
diff --git a/datafusion/expr/src/expr_fn.rs b/datafusion/expr/src/expr_fn.rs
index 4aa270e6dd..55bd40a189 100644
--- a/datafusion/expr/src/expr_fn.rs
+++ b/datafusion/expr/src/expr_fn.rs
@@ -557,7 +557,6 @@ nary_scalar_expr!(
trunc,
"truncate toward zero, with optional precision"
);
-scalar_expr!(Abs, abs, num, "absolute value");
scalar_expr!(Signum, signum, num, "sign of the argument (-1, 0, +1) ");
scalar_expr!(Exp, exp, num, "exponential");
scalar_expr!(Gcd, gcd, arg_1 arg_2, "greatest common divisor");
@@ -1354,7 +1353,6 @@ mod test {
test_nary_scalar_expr!(Round, round, input, decimal_places);
test_nary_scalar_expr!(Trunc, trunc, num);
test_nary_scalar_expr!(Trunc,
(arrow-datafusion-comet) branch main updated: fix: Fix compilation error for Spark 3.2 & 3.3 (#117)
This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 6ec8cb9 fix: Fix compilation error for Spark 3.2 & 3.3 (#117)
6ec8cb9 is described below
commit 6ec8cb912b938ef0ad7291f4f3de9ce7a883ae34
Author: Chao Sun
AuthorDate: Mon Feb 26 21:55:14 2024 -0800
fix: Fix compilation error for Spark 3.2 & 3.3 (#117)
---
spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
index 0414671..29b6e12 100644
--- a/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
+++ b/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala
@@ -43,6 +43,7 @@ import
org.apache.spark.sql.internal.SQLConf.SESSION_LOCAL_TIMEZONE
import org.apache.spark.unsafe.types.UTF8String
import org.apache.comet.CometConf
+import org.apache.comet.CometSparkSessionExtensions.isSpark34Plus
class CometExecSuite extends CometTestBase {
import testImplicits._
@@ -1055,6 +1056,7 @@ class CometExecSuite extends CometTestBase {
}
test("Fallback to Spark for TakeOrderedAndProjectExec with offset") {
+assume(isSpark34Plus)
Seq("true", "false").foreach(aqeEnabled =>
withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqeEnabled) {
withTable("t1") {
@@ -1066,7 +1068,7 @@ class CometExecSuite extends CometTestBase {
.write
.saveAsTable("t1")
- val df = sql("SELECT * FROM t1 ORDER BY a, b LIMIT
3").offset(1).groupBy($"a").sum("b")
+ val df = sql("SELECT * FROM t1 ORDER BY a, b LIMIT 3 OFFSET
1").groupBy($"a").sum("b")
checkSparkAnswer(df)
}
})
(arrow-datafusion-comet) branch main updated: feat: Add dictionary binary to shuffle writer (#111)
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 9cfe96f feat: Add dictionary binary to shuffle writer (#111)
9cfe96f is described below
commit 9cfe96f27647d7822e09a14115a02d677ab72f41
Author: Liang-Chi Hsieh
AuthorDate: Mon Feb 26 16:25:02 2024 -0800
feat: Add dictionary binary to shuffle writer (#111)
Native shuffle writer can write dictionary of string but dictionary of
binary is not supported. We should add it.
---
core/src/execution/datafusion/shuffle_writer.rs| 157 +
core/src/execution/datafusion/spark_hash.rs| 6 +
.../org/apache/comet/exec/CometShuffleSuite.scala | 19 +++
3 files changed, 153 insertions(+), 29 deletions(-)
diff --git a/core/src/execution/datafusion/shuffle_writer.rs
b/core/src/execution/datafusion/shuffle_writer.rs
index fc15fac..f836e3a 100644
--- a/core/src/execution/datafusion/shuffle_writer.rs
+++ b/core/src/execution/datafusion/shuffle_writer.rs
@@ -460,32 +460,32 @@ fn append_columns(
};
}
-macro_rules! append_string_dict {
-($kt:ident) => {{
+macro_rules! append_byte_dict {
+($kt:ident, $byte_type:ty, $array_type:ty) => {{
match $kt.as_ref() {
DataType::Int8 => {
-append_dict!(Int8Type, StringDictionaryBuilder,
StringArray)
+append_dict!(Int8Type,
GenericByteDictionaryBuilder, $array_type)
}
DataType::Int16 => {
-append_dict!(Int16Type,
StringDictionaryBuilder, StringArray)
+append_dict!(Int16Type,
GenericByteDictionaryBuilder, $array_type)
}
DataType::Int32 => {
-append_dict!(Int32Type,
StringDictionaryBuilder, StringArray)
+append_dict!(Int32Type,
GenericByteDictionaryBuilder, $array_type)
}
DataType::Int64 => {
-append_dict!(Int64Type,
StringDictionaryBuilder, StringArray)
+append_dict!(Int64Type,
GenericByteDictionaryBuilder, $array_type)
}
DataType::UInt8 => {
-append_dict!(UInt8Type,
StringDictionaryBuilder, StringArray)
+append_dict!(UInt8Type,
GenericByteDictionaryBuilder, $array_type)
}
DataType::UInt16 => {
-append_dict!(UInt16Type,
StringDictionaryBuilder, StringArray)
+append_dict!(UInt16Type,
GenericByteDictionaryBuilder, $array_type)
}
DataType::UInt32 => {
-append_dict!(UInt32Type,
StringDictionaryBuilder, StringArray)
+append_dict!(UInt32Type,
GenericByteDictionaryBuilder, $array_type)
}
DataType::UInt64 => {
-append_dict!(UInt64Type,
StringDictionaryBuilder, StringArray)
+append_dict!(UInt64Type,
GenericByteDictionaryBuilder, $array_type)
}
_ => unreachable!("Unknown key type for dictionary"),
}
@@ -522,7 +522,22 @@ fn append_columns(
DataType::Dictionary(key_type, value_type)
if matches!(value_type.as_ref(), DataType::Utf8) =>
{
-append_string_dict!(key_type)
+append_byte_dict!(key_type, GenericStringType, StringArray)
+}
+DataType::Dictionary(key_type, value_type)
+if matches!(value_type.as_ref(), DataType::LargeUtf8) =>
+{
+append_byte_dict!(key_type, GenericStringType,
LargeStringArray)
+}
+DataType::Dictionary(key_type, value_type)
+if matches!(value_type.as_ref(), DataType::Binary) =>
+{
+append_byte_dict!(key_type, GenericBinaryType, BinaryArray)
+}
+DataType::Dictionary(key_type, value_type)
+if matches!(value_type.as_ref(), DataType::LargeBinary) =>
+{
+append_byte_dict!(key_type, GenericBinaryType,
LargeBinaryArray)
}
DataType::Binary => append!(Binary),
DataType::LargeBinary => append!(LargeBinary),
@@ -1028,7 +1043,7 @@ macro_rules! primitive_dict_builder_helper {
};
}
-macro_rules! string_dict_builder_inner_helper {
+macro_rules! byte_dict_builder_inner_helper {
($kt:ty, $capacity:ident, $builder:ident) => {
Box::new($builder::<$kt>::with_capacity(
$capacity,
@@ -1068,28 +1083,28 @@ fn make_dict_builder(datatype: , capacity:
usize) -> Box {
-string_dict_builder_inner_helper!(Int16Type, capacity,
StringDictionaryBuilder)
+byte_dict_builder_inner_helper!(Int8Type, capacity,
(arrow) branch main updated (8805de7bf2 -> 06935a3690)
This is an automated email from the ASF dual-hosted git repository. kou pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from 8805de7bf2 MINOR: [Java] Bump commons-codec:commons-codec from 1.16.0 to 1.16.1 in /java (#40242) add 06935a3690 MINOR: [Java] Bump org.hamcrest:hamcrest-core from 1.3 to 2.2 in /java (#40238) No new revisions were added by this update. Summary of changes: java/flight/flight-sql-jdbc-core/pom.xml | 4 ++-- java/flight/flight-sql-jdbc-driver/pom.xml | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-)
(arrow) branch dependabot/maven/java/org.hamcrest-hamcrest-core-2.2 deleted (was 411c2f28c6)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/java/org.hamcrest-hamcrest-core-2.2 in repository https://gitbox.apache.org/repos/asf/arrow.git was 411c2f28c6 hamcrest-core -> hamcrest The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(arrow-rs) branch master updated: Add BufWriter for Adapative Put / Multipart Upload (#5431)
This is an automated email from the ASF dual-hosted git repository.
tustvold pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/master by this push:
new ef5c45cf418 Add BufWriter for Adapative Put / Multipart Upload (#5431)
ef5c45cf418 is described below
commit ef5c45cf4186a8124da5a1603ebdbc09ef9928fc
Author: Raphael Taylor-Davies <1781103+tustv...@users.noreply.github.com>
AuthorDate: Tue Feb 27 15:39:36 2024 +1300
Add BufWriter for Adapative Put / Multipart Upload (#5431)
* Add BufWriter
* Review feedback
---
object_store/src/buffered.rs | 163 ++-
1 file changed, 161 insertions(+), 2 deletions(-)
diff --git a/object_store/src/buffered.rs b/object_store/src/buffered.rs
index 3a1354f4f20..fdefe599f79 100644
--- a/object_store/src/buffered.rs
+++ b/object_store/src/buffered.rs
@@ -18,7 +18,7 @@
//! Utilities for performing tokio-style buffered IO
use crate::path::Path;
-use crate::{ObjectMeta, ObjectStore};
+use crate::{MultipartId, ObjectMeta, ObjectStore};
use bytes::Bytes;
use futures::future::{BoxFuture, FutureExt};
use futures::ready;
@@ -27,7 +27,7 @@ use std::io::{Error, ErrorKind, SeekFrom};
use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
-use tokio::io::{AsyncBufRead, AsyncRead, AsyncSeek, ReadBuf};
+use tokio::io::{AsyncBufRead, AsyncRead, AsyncSeek, AsyncWrite, AsyncWriteExt,
ReadBuf};
/// The default buffer size used by [`BufReader`]
pub const DEFAULT_BUFFER_SIZE: usize = 1024 * 1024;
@@ -205,6 +205,138 @@ impl AsyncBufRead for BufReader {
}
}
+/// An async buffered writer compatible with the tokio IO traits
+///
+/// Up to `capacity` bytes will be buffered in memory, and flushed on shutdown
+/// using [`ObjectStore::put`]. If `capacity` is exceeded, data will instead be
+/// streamed using [`ObjectStore::put_multipart`]
+pub struct BufWriter {
+capacity: usize,
+state: BufWriterState,
+multipart_id: Option,
+store: Arc,
+}
+
+impl std::fmt::Debug for BufWriter {
+fn fmt(, f: std::fmt::Formatter<'_>) -> std::fmt::Result {
+f.debug_struct("BufWriter")
+.field("capacity", )
+.field("multipart_id", _id)
+.finish()
+}
+}
+
+type MultipartResult = (MultipartId, Box);
+
+enum BufWriterState {
+/// Buffer up to capacity bytes
+Buffer(Path, Vec),
+/// [`ObjectStore::put_multipart`]
+Prepare(BoxFuture<'static, std::io::Result>),
+/// Write to a multipart upload
+Write(Box),
+/// [`ObjectStore::put`]
+Put(BoxFuture<'static, std::io::Result<()>>),
+}
+
+impl BufWriter {
+/// Create a new [`BufWriter`] from the provided [`ObjectStore`] and
[`Path`]
+pub fn new(store: Arc, path: Path) -> Self {
+Self::with_capacity(store, path, 10 * 1024 * 1024)
+}
+
+/// Create a new [`BufWriter`] from the provided [`ObjectStore`], [`Path`]
and `capacity`
+pub fn with_capacity(store: Arc, path: Path, capacity:
usize) -> Self {
+Self {
+capacity,
+store,
+state: BufWriterState::Buffer(path, Vec::new()),
+multipart_id: None,
+}
+}
+
+/// Returns the [`MultipartId`] if multipart upload
+pub fn multipart_id() -> Option<> {
+self.multipart_id.as_ref()
+}
+}
+
+impl AsyncWrite for BufWriter {
+fn poll_write(
+mut self: Pin< Self>,
+cx: Context<'_>,
+buf: &[u8],
+) -> Poll> {
+let cap = self.capacity;
+loop {
+return match self.state {
+BufWriterState::Write(write) => Pin::new(write).poll_write(cx,
buf),
+BufWriterState::Put(_) => panic!("Already shut down"),
+BufWriterState::Prepare(f) => {
+let (id, w) = ready!(f.poll_unpin(cx)?);
+self.state = BufWriterState::Write(w);
+self.multipart_id = Some(id);
+continue;
+}
+BufWriterState::Buffer(path, b) => {
+if b.len().saturating_add(buf.len()) >= cap {
+let buffer = std::mem::take(b);
+let path = std::mem::take(path);
+let store = Arc::clone();
+self.state = BufWriterState::Prepare(Box::pin(async
move {
+let (id, mut writer) =
store.put_multipart().await?;
+writer.write_all().await?;
+Ok((id, writer))
+}));
+continue;
+}
+b.extend_from_slice(buf);
+Poll::Ready(Ok(buf.len()))
+}
+};
+}
+}
+
+fn poll_flush(mut self: Pin< Self>, cx: Context<'_>) ->
Poll> {
+loop {
+return
(arrow) branch main updated (06935a3690 -> 46407df093)
This is an automated email from the ASF dual-hosted git repository. ianmcook pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/arrow.git from 06935a3690 MINOR: [Java] Bump org.hamcrest:hamcrest-core from 1.3 to 2.2 in /java (#40238) add 46407df093 GH-40215: [Format][Docs] Document Arrow Columnar Format version history (#40219) No new revisions were added by this update. Summary of changes: docs/source/format/CanonicalExtensions.rst | 2 +- docs/source/format/Columnar.rst| 17 ++- docs/source/format/Versioning.rst | 46 ++ docs/source/status.rst | 6 ++-- 4 files changed, 61 insertions(+), 10 deletions(-)
(arrow) branch main updated: GH-40228: [C++][CMake] Improve description why we need to initialize AWS C++ SDK in arrow-s3fs-test (#40229)
This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 2235a7ed40 GH-40228: [C++][CMake] Improve description why we need to
initialize AWS C++ SDK in arrow-s3fs-test (#40229)
2235a7ed40 is described below
commit 2235a7ed40b999d919d7d17cbb34097e819a5acf
Author: Sutou Kouhei
AuthorDate: Tue Feb 27 13:54:26 2024 +0900
GH-40228: [C++][CMake] Improve description why we need to initialize AWS
C++ SDK in arrow-s3fs-test (#40229)
### Rationale for this change
Only static linking isn't important. Static linking + private symbols are
important.
### What changes are included in this PR?
Improve comment and macro name.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
No.
* GitHub Issue: #40228
Authored-by: Sutou Kouhei
Signed-off-by: Sutou Kouhei
---
cpp/src/arrow/filesystem/CMakeLists.txt | 18 +++---
cpp/src/arrow/filesystem/s3fs_test.cc | 6 +++---
2 files changed, 10 insertions(+), 14 deletions(-)
diff --git a/cpp/src/arrow/filesystem/CMakeLists.txt
b/cpp/src/arrow/filesystem/CMakeLists.txt
index a42a8d0f8c..77e93223cd 100644
--- a/cpp/src/arrow/filesystem/CMakeLists.txt
+++ b/cpp/src/arrow/filesystem/CMakeLists.txt
@@ -71,17 +71,13 @@ if(ARROW_S3)
get_target_property(AWS_CPP_SDK_S3_TYPE aws-cpp-sdk-s3 TYPE)
# We need to initialize AWS C++ SDK for direct use (not via
# arrow::fs::S3FileSystem) in arrow-s3fs-test if we use static AWS
-# C++ SDK. Because AWS C++ SDK has internal static variables that
-# aren't shared in libarrow and arrow-s3fs-test. It means that
-# arrow::fs::InitializeS3() doesn't initialize AWS C++ SDK that is
-# directly used in arrow-s3fs-test.
-#
-# But it seems that internal static variables in AWS C++ SDK are
-# shared on macOS even if we link static AWS C++ SDK to both
-# libarrow and arrow-s3fs-test. So we don't need to initialize AWS
-# C++ SDK in arrow-s3fs-test on macOS.
-if(AWS_CPP_SDK_S3_TYPE STREQUAL "STATIC_LIBRARY" AND NOT APPLE)
- list(APPEND ARROW_S3FS_TEST_COMPILE_DEFINITIONS
"AWS_CPP_SDK_S3_NOT_SHARED")
+# C++ SDK and hide symbols of them. Because AWS C++ SDK has
+# internal static variables that aren't shared in libarrow and
+# arrow-s3fs-test. It means that arrow::fs::InitializeS3() doesn't
+# initialize AWS C++ SDK that is directly used in arrow-s3fs-test.
+if(AWS_CPP_SDK_S3_TYPE STREQUAL "STATIC_LIBRARY"
+ AND CXX_LINKER_SUPPORTS_VERSION_SCRIPT)
+ list(APPEND ARROW_S3FS_TEST_COMPILE_DEFINITIONS
"AWS_CPP_SDK_S3_PRIVATE_STATIC")
endif()
target_compile_definitions(arrow-s3fs-test
PRIVATE ${ARROW_S3FS_TEST_COMPILE_DEFINITIONS})
diff --git a/cpp/src/arrow/filesystem/s3fs_test.cc
b/cpp/src/arrow/filesystem/s3fs_test.cc
index 33e9712a66..394f59e91a 100644
--- a/cpp/src/arrow/filesystem/s3fs_test.cc
+++ b/cpp/src/arrow/filesystem/s3fs_test.cc
@@ -150,7 +150,7 @@ class ShortRetryStrategy : public S3RetryStrategy {
class AwsTestMixin : public ::testing::Test {
public:
void SetUp() override {
-#ifdef AWS_CPP_SDK_S3_NOT_SHARED
+#ifdef AWS_CPP_SDK_S3_PRIVATE_STATIC
auto aws_log_level = Aws::Utils::Logging::LogLevel::Fatal;
aws_options_.loggingOptions.logLevel = aws_log_level;
aws_options_.loggingOptions.logger_create_fn = [_log_level] {
@@ -161,13 +161,13 @@ class AwsTestMixin : public ::testing::Test {
}
void TearDown() override {
-#ifdef AWS_CPP_SDK_S3_NOT_SHARED
+#ifdef AWS_CPP_SDK_S3_PRIVATE_STATIC
Aws::ShutdownAPI(aws_options_);
#endif
}
private:
-#ifdef AWS_CPP_SDK_S3_NOT_SHARED
+#ifdef AWS_CPP_SDK_S3_PRIVATE_STATIC
Aws::SDKOptions aws_options_;
#endif
};
(arrow-rs) branch master updated: Improve debug output of Time32/Time64 arrays (#5428)
This is an automated email from the ASF dual-hosted git repository.
tustvold pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/master by this push:
new 37cf8a6c19e Improve debug output of Time32/Time64 arrays (#5428)
37cf8a6c19e is described below
commit 37cf8a6c19e02a4dc822fadf331c2fae9e4b5656
Author: Clide S <109172241+monkwi...@users.noreply.github.com>
AuthorDate: Mon Feb 26 19:14:57 2024 -0500
Improve debug output of Time32/Time64 arrays (#5428)
* Improve debug output of Time32/Time64 arrays
* Update arrow-array/src/array/primitive_array.rs
Co-authored-by: Raphael Taylor-Davies
<1781103+tustv...@users.noreply.github.com>
* Update arrow-array/src/array/primitive_array.rs
Co-authored-by: Raphael Taylor-Davies
<1781103+tustv...@users.noreply.github.com>
* Fix
-
Co-authored-by: Raphael Taylor-Davies
<1781103+tustv...@users.noreply.github.com>
Co-authored-by: Raphael Taylor-Davies
---
arrow-array/src/array/primitive_array.rs | 85 ++--
1 file changed, 82 insertions(+), 3 deletions(-)
diff --git a/arrow-array/src/array/primitive_array.rs
b/arrow-array/src/array/primitive_array.rs
index ca437a569a1..a800aa6bf92 100644
--- a/arrow-array/src/array/primitive_array.rs
+++ b/arrow-array/src/array/primitive_array.rs
@@ -1093,20 +1093,31 @@ where
impl std::fmt::Debug for PrimitiveArray {
fn fmt(, f: std::fmt::Formatter) -> std::fmt::Result {
let data_type = self.data_type();
+
write!(f, "PrimitiveArray<{data_type:?}>\n[\n")?;
print_long_array(self, f, |array, index, f| match data_type {
DataType::Date32 | DataType::Date64 => {
let v = self.value(index).to_isize().unwrap() as i64;
match as_date::(v) {
Some(date) => write!(f, "{date:?}"),
-None => write!(f, "null"),
+None => {
+write!(
+f,
+"Cast error: Failed to convert {v} to temporal for
{data_type:?}"
+)
+}
}
}
DataType::Time32(_) | DataType::Time64(_) => {
let v = self.value(index).to_isize().unwrap() as i64;
match as_time::(v) {
Some(time) => write!(f, "{time:?}"),
-None => write!(f, "null"),
+None => {
+write!(
+f,
+"Cast error: Failed to convert {v} to temporal for
{data_type:?}"
+)
+}
}
}
DataType::Timestamp(_, tz_string_opt) => {
@@ -1948,7 +1959,8 @@ mod tests {
// chrono::NaiveDatetime::from_timestamp_opt returns None while input
is invalid
let arr: PrimitiveArray = vec![-7201, -60054].into();
assert_eq!(
-"PrimitiveArray\n[\n null,\n null,\n]",
+"PrimitiveArray\n[\n Cast error: Failed to convert
-7201 to temporal for Time32(Second),\n Cast error: Failed to convert -60054
to temporal for Time32(Second),\n]",
+// "PrimitiveArray\n[\n null,\n null,\n]",
format!("{arr:?}")
)
}
@@ -2482,4 +2494,71 @@ mod tests {
fn test_with_data_type() {
Int32Array::new(vec![1, 2, 3, 4].into(),
None).with_data_type(DataType::Date32);
}
+
+#[test]
+fn test_time_32second_output() {
+let array: Time32SecondArray = vec![
+Some(-1),
+Some(0),
+Some(86_399),
+Some(86_400),
+Some(86_401),
+None,
+]
+.into();
+let debug_str = format!("{:?}", array);
+assert_eq!("PrimitiveArray\n[\n Cast error: Failed to
convert -1 to temporal for Time32(Second),\n 00:00:00,\n 23:59:59,\n Cast
error: Failed to convert 86400 to temporal for Time32(Second),\n Cast error:
Failed to convert 86401 to temporal for Time32(Second),\n null,\n]",
+debug_str
+);
+}
+
+#[test]
+fn test_time_32millisecond_debug_output() {
+let array: Time32MillisecondArray = vec![
+Some(-1),
+Some(0),
+Some(86_399_000),
+Some(86_400_000),
+Some(86_401_000),
+None,
+]
+.into();
+let debug_str = format!("{:?}", array);
+assert_eq!("PrimitiveArray\n[\n Cast error:
Failed to convert -1 to temporal for Time32(Millisecond),\n 00:00:00,\n
23:59:59,\n Cast error: Failed to convert 8640 to temporal for
Time32(Millisecond),\n Cast error: Failed to convert 86401000 to temporal for
Time32(Millisecond),\n null,\n]",
+debug_str
+);
+}
+
+#[test]
+
(arrow) 01/01: GH-40181: [C++] Support glog 0.7 build
This is an automated email from the ASF dual-hosted git repository.
uwe pushed a commit to branch ARROW-40181-glog07
in repository https://gitbox.apache.org/repos/asf/arrow.git
commit 147504673e409fc5267f0422a58f170f442b5455
Author: Uwe L. Korn
AuthorDate: Mon Feb 26 09:21:31 2024 +0100
GH-40181: [C++] Support glog 0.7 build
---
cpp/cmake_modules/FindGLOG.cmake | 8 +++-
cpp/src/arrow/util/logging.cc| 6 +++---
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/cpp/cmake_modules/FindGLOG.cmake b/cpp/cmake_modules/FindGLOG.cmake
index 61b7d0694e..62b235ee91 100644
--- a/cpp/cmake_modules/FindGLOG.cmake
+++ b/cpp/cmake_modules/FindGLOG.cmake
@@ -17,6 +17,11 @@
#
# find_package(GLOG)
+find_package(glog CONFIG)
+if(glog_FOUND)
+ return()
+endif()
+
if(GLOG_FOUND)
return()
endif()
@@ -56,5 +61,6 @@ if(GLOG_FOUND)
add_library(glog::glog UNKNOWN IMPORTED)
set_target_properties(glog::glog
PROPERTIES IMPORTED_LOCATION "${GLOG_LIB}"
- INTERFACE_INCLUDE_DIRECTORIES
"${GLOG_INCLUDE_DIR}")
+ INTERFACE_INCLUDE_DIRECTORIES
"${GLOG_INCLUDE_DIR}"
+ INTERFACE_COMPILE_DEFINITIONS
"GLOG_USE_GLOG_EXPORT")
endif()
diff --git a/cpp/src/arrow/util/logging.cc b/cpp/src/arrow/util/logging.cc
index d293113237..25c336a6d2 100644
--- a/cpp/src/arrow/util/logging.cc
+++ b/cpp/src/arrow/util/logging.cc
@@ -116,7 +116,7 @@ static std::unique_ptr log_dir_;
#ifdef ARROW_USE_GLOG
// Glog's severity map.
-static int GetMappedSeverity(ArrowLogLevel severity) {
+static google::LogSeverity GetMappedSeverity(ArrowLogLevel severity) {
switch (severity) {
case ArrowLogLevel::ARROW_DEBUG:
return google::GLOG_INFO;
@@ -148,7 +148,7 @@ void ArrowLog::StartArrowLog(const std::string& app_name,
app_name_.reset(new std::string(app_name));
log_dir_.reset(new std::string(log_dir));
#ifdef ARROW_USE_GLOG
- int mapped_severity_threshold = GetMappedSeverity(severity_threshold_);
+ google::LogSeverity mapped_severity_threshold =
GetMappedSeverity(severity_threshold_);
google::SetStderrLogging(mapped_severity_threshold);
// Enable log file if log_dir is not empty.
if (!log_dir.empty()) {
@@ -173,7 +173,7 @@ void ArrowLog::StartArrowLog(const std::string& app_name,
google::SetLogFilenameExtension(app_name_without_path.c_str());
for (int i = static_cast(severity_threshold_);
i <= static_cast(ArrowLogLevel::ARROW_FATAL); ++i) {
- int level = GetMappedSeverity(static_cast(i));
+ google::LogSeverity level =
GetMappedSeverity(static_cast(i));
google::SetLogDestination(level, dir_ends_with_slash.c_str());
}
}
(arrow) branch ARROW-40181-glog07 created (now 147504673e)
This is an automated email from the ASF dual-hosted git repository. uwe pushed a change to branch ARROW-40181-glog07 in repository https://gitbox.apache.org/repos/asf/arrow.git at 147504673e GH-40181: [C++] Support glog 0.7 build This branch includes the following new commits: new 147504673e GH-40181: [C++] Support glog 0.7 build The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(arrow-datafusion) branch dependabot/cargo/main/nix-0.28.0 created (now eaa5557457)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/cargo/main/nix-0.28.0 in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git at eaa5557457 Update nix requirement from 0.27.1 to 0.28.0 No new revisions were added by this update.
(arrow-datafusion) branch main updated: feat: support `FixedSizeList` Type Coercion (#9108)
This is an automated email from the ASF dual-hosted git repository.
jayzhan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new b728232b91 feat: support `FixedSizeList` Type Coercion (#9108)
b728232b91 is described below
commit b728232b91862826061dfa878cefe90d25576f78
Author: Alex Huang
AuthorDate: Mon Feb 26 21:10:13 2024 +0800
feat: support `FixedSizeList` Type Coercion (#9108)
* support FixedSizeList Type Coercion
* add allow null type coercion parameter
* support null column in FixedSizeList
* Add test
* Add tests for cardinality with fixed size lists
* chore
* fix ci
* add comment
* Fix array_element function signature
* Remove unused imports and simplify code
* Fix array function signatures and behavior
* fix conflict
* fix conflict
* add tests for FixedSizeList
* remove unreacheable null check
* simplify the code
* remove null checking
* reformat output
* simplify code
* add tests for array_dims
* Refactor type coercion functions in datafusion/expr module
---
datafusion/expr/src/built_in_function.rs | 23 +-
datafusion/expr/src/signature.rs | 17 +-
datafusion/expr/src/type_coercion/functions.rs | 109 ++---
datafusion/sqllogictest/test_files/array.slt | 562 -
4 files changed, 628 insertions(+), 83 deletions(-)
diff --git a/datafusion/expr/src/built_in_function.rs
b/datafusion/expr/src/built_in_function.rs
index f92ae87d6e..8b4e65121c 100644
--- a/datafusion/expr/src/built_in_function.rs
+++ b/datafusion/expr/src/built_in_function.rs
@@ -31,7 +31,7 @@ use crate::{
};
use arrow::datatypes::{DataType, Field, Fields, IntervalUnit, TimeUnit};
-use datafusion_common::{internal_err, plan_err, DataFusionError, Result};
+use datafusion_common::{exec_err, plan_err, DataFusionError, Result};
use strum::IntoEnumIterator;
use strum_macros::EnumIter;
@@ -543,10 +543,11 @@ impl BuiltinScalarFunction {
BuiltinScalarFunction::Flatten => {
fn get_base_type(data_type: ) -> Result {
match data_type {
-DataType::List(field) if matches!(field.data_type(),
DataType::List(_)) => get_base_type(field.data_type()),
+DataType::List(field) | DataType::FixedSizeList(field,
_) if matches!(field.data_type(), DataType::List(_)|DataType::FixedSizeList(_,_
)) => get_base_type(field.data_type()),
DataType::LargeList(field) if
matches!(field.data_type(), DataType::LargeList(_)) =>
get_base_type(field.data_type()),
DataType::Null | DataType::List(_) |
DataType::LargeList(_) => Ok(data_type.to_owned()),
-_ => internal_err!("Not reachable, data_type should be
List or LargeList"),
+DataType::FixedSizeList(field,_ ) =>
Ok(DataType::List(field.clone())),
+_ => exec_err!("Not reachable, data_type should be
List, LargeList or FixedSizeList"),
}
}
@@ -929,18 +930,18 @@ impl BuiltinScalarFunction {
// 0 or more arguments of arbitrary type
Signature::one_of(vec![VariadicEqual, Any(0)],
self.volatility())
}
-BuiltinScalarFunction::ArrayPopFront => Signature::any(1,
self.volatility()),
-BuiltinScalarFunction::ArrayPopBack => Signature::any(1,
self.volatility()),
+BuiltinScalarFunction::ArrayPopFront =>
Signature::array(self.volatility()),
+BuiltinScalarFunction::ArrayPopBack =>
Signature::array(self.volatility()),
BuiltinScalarFunction::ArrayConcat => {
Signature::variadic_any(self.volatility())
}
-BuiltinScalarFunction::ArrayDims => Signature::any(1,
self.volatility()),
-BuiltinScalarFunction::ArrayEmpty => Signature::any(1,
self.volatility()),
+BuiltinScalarFunction::ArrayDims =>
Signature::array(self.volatility()),
+BuiltinScalarFunction::ArrayEmpty =>
Signature::array(self.volatility()),
BuiltinScalarFunction::ArrayElement => {
Signature::array_and_index(self.volatility())
}
BuiltinScalarFunction::ArrayExcept => Signature::any(2,
self.volatility()),
-BuiltinScalarFunction::Flatten => Signature::any(1,
self.volatility()),
+BuiltinScalarFunction::Flatten =>
Signature::array(self.volatility()),
BuiltinScalarFunction::ArrayHasAll |
BuiltinScalarFunction::ArrayHasAny => {
Signature::any(2, self.volatility())
}
@@ -950,8 +951,8 @@ impl BuiltinScalarFunction {
(arrow-datafusion) branch main updated: feat: expand `unnest` to accept arbitrary single array expression (#9342)
This is an automated email from the ASF dual-hosted git repository.
jayzhan pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-datafusion.git
The following commit(s) were added to refs/heads/main by this push:
new ec86acbc1f feat: expand `unnest` to accept arbitrary single array
expression (#9342)
ec86acbc1f is described below
commit ec86acbc1fbc0da1e0bec9ad066a5177ec586c96
Author: Jonah Gao
AuthorDate: Mon Feb 26 21:33:19 2024 +0800
feat: expand `unnest` to accept arbitrary single array expression (#9342)
* feat: expand `unnest` to accept any single array expression
* unnest null
* review feedback
---
datafusion/sql/src/expr/function.rs | 58 ---
datafusion/sqllogictest/test_files/unnest.slt | 48 +-
2 files changed, 64 insertions(+), 42 deletions(-)
diff --git a/datafusion/sql/src/expr/function.rs
b/datafusion/sql/src/expr/function.rs
index f56138066c..db572a23cf 100644
--- a/datafusion/sql/src/expr/function.rs
+++ b/datafusion/sql/src/expr/function.rs
@@ -16,16 +16,17 @@
// under the License.
use crate::planner::{ContextProvider, PlannerContext, SqlToRel};
+use arrow_schema::DataType;
use datafusion_common::{
-exec_err, not_impl_err, plan_datafusion_err, plan_err, DFSchema,
DataFusionError,
-Dependency, Result,
+not_impl_err, plan_datafusion_err, plan_err, DFSchema, DataFusionError,
Dependency,
+Result,
};
use datafusion_expr::expr::{ScalarFunction, Unnest};
use datafusion_expr::function::suggest_valid_function;
use datafusion_expr::window_frame::{check_window_frame,
regularize_window_order_by};
use datafusion_expr::{
-expr, AggregateFunction, BuiltinScalarFunction, Expr,
ScalarFunctionDefinition,
-WindowFrame, WindowFunctionDefinition,
+expr, AggregateFunction, BuiltinScalarFunction, Expr, ExprSchemable,
WindowFrame,
+WindowFunctionDefinition,
};
use sqlparser::ast::{
Expr as SQLExpr, Function as SQLFunction, FunctionArg, FunctionArgExpr,
WindowType,
@@ -80,41 +81,34 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
if name.eq("unnest") {
let exprs =
self.function_args_to_expr(args.clone(), schema,
planner_context)?;
-
-match exprs.len() {
+// Currently only one argument is supported
+let arg = match exprs.len() {
0 => {
-return exec_err!("unnest() requires at least one
argument");
-}
-1 => {
-if let Expr::ScalarFunction(ScalarFunction {
-func_def:
-ScalarFunctionDefinition::BuiltIn(
-BuiltinScalarFunction::MakeArray,
-),
-..
-}) = exprs[0]
-{
-// valid
-} else if let Expr::Column(_) = exprs[0] {
-// valid
-} else if let Expr::ScalarFunction(ScalarFunction {
-func_def:
-
ScalarFunctionDefinition::BuiltIn(BuiltinScalarFunction::Struct),
-..
-}) = exprs[0]
-{
-return not_impl_err!("unnest() does not support struct
yet");
-} else {
-return plan_err!(
-"unnest() can only be applied to array and structs
and null"
-);
-}
+return plan_err!("unnest() requires at least one
argument");
}
+1 => [0],
_ => {
return not_impl_err!(
"unnest() does not support multiple arguments yet"
);
}
+};
+// Check argument type, array types are supported
+match arg.get_type(schema)? {
+DataType::List(_)
+| DataType::LargeList(_)
+| DataType::FixedSizeList(_, _) => {}
+DataType::Struct(_) => {
+return not_impl_err!("unnest() does not support struct
yet");
+}
+DataType::Null => {
+return not_impl_err!("unnest() does not support null yet");
+}
+_ => {
+return plan_err!(
+"unnest() can only be applied to array, struct and
null"
+);
+}
}
return Ok(Expr::Unnest(Unnest { exprs }));
diff --git a/datafusion/sqllogictest/test_files/unnest.slt
b/datafusion/sqllogictest/test_files/unnest.slt
index 7e4ce06be2..9990c00f75 100644
--- a/datafusion/sqllogictest/test_files/unnest.slt
+++
(arrow-rs) branch dependabot/cargo/object_store/master/nix-0.28.0 created (now a2a19cd4d96)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/cargo/object_store/master/nix-0.28.0 in repository https://gitbox.apache.org/repos/asf/arrow-rs.git at a2a19cd4d96 Update nix requirement from 0.27.1 to 0.28.0 in /object_store No new revisions were added by this update.
(arrow-rs) branch master updated: Update nix requirement from 0.27.1 to 0.28.0 in /object_store (#5432)
This is an automated email from the ASF dual-hosted git repository.
tustvold pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/master by this push:
new 282123441c7 Update nix requirement from 0.27.1 to 0.28.0 in
/object_store (#5432)
282123441c7 is described below
commit 282123441c7bfd549c2a2216fba1e253e20501ba
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Tue Feb 27 03:52:13 2024 +1300
Update nix requirement from 0.27.1 to 0.28.0 in /object_store (#5432)
Updates the requirements on [nix](https://github.com/nix-rust/nix) to
permit the latest version.
- [Changelog](https://github.com/nix-rust/nix/blob/master/CHANGELOG.md)
- [Commits](https://github.com/nix-rust/nix/compare/v0.27.1...v0.28.0)
---
updated-dependencies:
- dependency-name: nix
dependency-type: direct:production
...
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot]
<49699333+dependabot[bot]@users.noreply.github.com>
---
object_store/Cargo.toml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/object_store/Cargo.toml b/object_store/Cargo.toml
index f3aaf35fbb0..6b38a8d21ff 100644
--- a/object_store/Cargo.toml
+++ b/object_store/Cargo.toml
@@ -57,7 +57,7 @@ tokio = { version = "1.25.0", features = ["sync", "macros",
"rt", "time", "io-ut
md-5 = { version = "0.10.6", default-features = false, optional = true }
[target.'cfg(target_family="unix")'.dev-dependencies]
-nix = { version = "0.27.1", features = ["fs"] }
+nix = { version = "0.28.0", features = ["fs"] }
[features]
cloud = ["serde", "serde_json", "quick-xml", "hyper", "reqwest",
"reqwest/json", "reqwest/stream", "chrono/serde", "base64", "rand", "ring"]
(arrow-rs) branch dependabot/cargo/object_store/master/nix-0.28.0 deleted (was a2a19cd4d96)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/cargo/object_store/master/nix-0.28.0 in repository https://gitbox.apache.org/repos/asf/arrow-rs.git was a2a19cd4d96 Update nix requirement from 0.27.1 to 0.28.0 in /object_store The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
