kosiew commented on code in PR #26934:
URL: https://github.com/apache/datafusion/pull/26934#discussion_r4627585926
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -2944,7 +2936,7 @@ macro_rules! dataframe {
}};
($($name:expr => $data:expr),+ $(,)?) => {{
- use datafusion::prelude::DataFrame;
+ use datafusion::prelude::SessionContext;
Review Comment:
The block-scoped SessionContext import can shadow caller types used in macro
arguments. Consider using $crate-qualified paths without local imports in both
macro arms.
##########
datafusion/core/tests/dataframe/mod.rs:
##########
@@ -7521,75 +7525,100 @@ async fn test_dataframe_from_columns() -> Result<()> {
}
#[test]
-fn test_dataframe_from_columns_empty() {
- let result = DataFrame::from_columns(vec![]);
- assert!(result.is_err());
-
- let result = DataFrame::from_columns([]);
- assert!(result.is_err());
+fn test_dataframe_read_columns_empty() {
+ let ctx = SessionContext::new();
+ assert!(ctx.read_columns(vec![]).is_err());
+ assert!(ctx.read_columns([]).is_err());
Review Comment:
Consider testing columns with mismatched lengths to verify the documented
error behavior. RecordBatch::try_new already validates lengths, but the current
negative test covers only empty input.
##########
datafusion/core/src/execution/context/mod.rs:
##########
@@ -1894,6 +1896,73 @@ impl SessionContext {
.build()?,
))
}
+
+ /// Create a [`DataFrame`] from named Arrow arrays.
+ ///
+ /// Each `(name, array)` pair becomes one column. The field's nullability
+ /// is derived from the array: a column is non-nullable when its array is
+ /// guaranteed to contain no nulls (see [`Array::is_nullable`]), which lets
+ /// the optimizer simplify expressions such as `col IS NULL` on it.
+ ///
+ /// The arrays are wrapped in a single [`RecordBatch`] and read through
+ /// [`Self::read_batch`], so the returned `DataFrame` runs in this session
+ /// and sees its registered functions, tables, configuration and runtime,
+ /// exactly like a `DataFrame` created with [`Self::read_batch`].
+ ///
+ /// Returns an error if `columns` is empty or if the arrays have different
+ /// lengths.
+ ///
+ /// See also the [`dataframe!`] macro for building a `DataFrame` from
+ /// literal values without an existing `SessionContext`.
+ ///
+ /// [`Array::is_nullable`]: arrow::array::Array::is_nullable
+ /// [`dataframe!`]: macro@crate::dataframe
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use std::sync::Arc;
+ /// use arrow::array::{ArrayRef, Int32Array, StringArray};
+ /// use datafusion::prelude::*;
+ /// # use datafusion::error::Result;
+ /// # use datafusion_common::assert_batches_sorted_eq;
+ /// # #[tokio::main]
+ /// # async fn main() -> Result<()> {
+ /// let id: ArrayRef = Arc::new(Int32Array::from(vec![1, 2, 3]));
+ /// let name: ArrayRef = Arc::new(StringArray::from(vec!["foo", "bar",
"baz"]));
+ /// let ctx = SessionContext::new();
+ /// let df = ctx.read_columns([("id", id), ("name", name)])?;
+ /// let expected = vec![
+ /// "+----+------+",
+ /// "| id | name |",
+ /// "+----+------+",
+ /// "| 1 | foo |",
+ /// "| 2 | bar |",
+ /// "| 3 | baz |",
+ /// "+----+------+",
+ /// ];
+ /// # assert_batches_sorted_eq!(expected, &df.collect().await?);
+ /// # Ok(())
+ /// # }
+ /// ```
+ pub fn read_columns<'a, I>(&self, columns: I) -> Result<DataFrame>
+ where
+ I: IntoIterator<Item = (&'a str, ArrayRef)>,
+ {
+ let (fields, arrays): (Vec<_>, Vec<_>) = columns
+ .into_iter()
+ .map(|(name, array)| {
+ (
+ Field::new(name, array.data_type().clone(),
array.is_nullable()),
Review Comment:
Both existing entry points now infer nullability instead of marking all
fields nullable, potentially breaking exact schema comparisons. Please preserve
the old behavior or document the change for both APIs, and test their
nullability contracts.
##########
datafusion/core/src/execution/context/mod.rs:
##########
@@ -1894,6 +1896,73 @@ impl SessionContext {
.build()?,
))
}
+
+ /// Create a [`DataFrame`] from named Arrow arrays.
+ ///
+ /// Each `(name, array)` pair becomes one column. The field's nullability
+ /// is derived from the array: a column is non-nullable when its array is
+ /// guaranteed to contain no nulls (see [`Array::is_nullable`]), which lets
+ /// the optimizer simplify expressions such as `col IS NULL` on it.
+ ///
+ /// The arrays are wrapped in a single [`RecordBatch`] and read through
+ /// [`Self::read_batch`], so the returned `DataFrame` runs in this session
+ /// and sees its registered functions, tables, configuration and runtime,
Review Comment:
Please clarify that read_batch snapshots SessionState, so later UDF
registrations are not visible through the existing DataFrame, while shared
catalog and runtime state can reflect later changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]