This is an automated email from the ASF dual-hosted git repository.
sarutak pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9b17867be1f3 [SPARK-60735][SQL] Reject a native data source package
that a session adds when Spark cannot use it
9b17867be1f3 is described below
commit 9b17867be1f3cb182b5af8a90d1cfbcf025975a0
Author: Kousuke Saruta <[email protected]>
AuthorDate: Wed Nov 4 17:11:06 2026 +0900
[SPARK-60735][SQL] Reject a native data source package that a session adds
when Spark cannot use it
### What changes were proposed in this pull request?
This PR makes `spark.addArtifact` fail for a native data source package
(SPARK-60101) that Spark cannot use, instead of adding it to the session and
ignoring it when a query looks the data source up.
- When a session adds a `.sparkpkg` file, and
`spark.sql.dataSource.native.enabled` is true, `ArtifactManager.addArtifact`
now checks the package before it becomes an artifact of the session. It fails
with `INVALID_NATIVE_DATA_SOURCE_PACKAGE.INVALID_MANIFEST` if the file is not a
valid package (a file that is not a zip, a missing or invalid manifest, or a
listed library that is missing; a directory entry of the zip is not taken as
the manifest or a library; a file that cannot be read [...]
- The errors of the check name the file that the user added, or the
artifact, such as `files/rs-2.0.sparkpkg`, when Spark staged the file, as for
`spark.addArtifact(bytes, target)` or a Spark Connect client. For a local
package, `addLocalArtifacts` passes the name of the file of the user
explicitly, to a private overload of `addArtifact`; the other local files still
go through the public `addArtifact`, as before, so a subclass of
`ArtifactManager` that overrides it does not see a loca [...]
- Spark first compares the file with the artifact of the same name, as
before this PR, so `ARTIFACT_ALREADY_EXISTS` comes first, and adding the same
file again, also from a Spark Connect client, does nothing, and neither copies
nor checks the file. Spark then copies a local file to a temporary directory,
makes the copy writable and checks it, or checks a staged file, which it does
not copy, compares it with the artifact again right before the transfer, and
moves it to the path of the [...]
- The session then has no file with the name of the rejected package, so it
can add a valid package with the same file name. In Scala and Java, also with
Spark Connect, a call that adds several artifacts still adds the others after a
rejected package, and then fails with the first error, with the others
suppressed in it, as for `ARTIFACT_ALREADY_EXISTS` (a Spark Connect client gets
only the first one): `addLocalArtifacts` and the Spark Connect `AddArtifacts`
handler now also collect ` [...]
- The temporary file of an in-memory artifact (`spark.addArtifact(bytes,
target)`) is deleted also when the artifact is not added, without starting a
process.
- Only the file artifacts (`files/`) with the extension are packages: the
session no longer takes another kind of artifact with the extension, such as
`pyfiles/x.sparkpkg`, which a Spark Connect client could send, as a package
without checking it.
- The packages in `spark.sql.dataSource.native.paths` are still ignored and
logged when Spark cannot read them or does not support their version. An I/O
error while Spark computes the checksum of a package whose manifest it read is
now reported as "The package cannot be read", with the data sources of the
manifest, so a lookup of one of them reports it, as for the other invalid
packages. A configured package whose manifest names a directory entry of the
zip as a library is now invalid [...]
- The message of `UNSUPPORTED_ABI_VERSION`, of both
`INVALID_NATIVE_DATA_SOURCE_PACKAGE` and `INVALID_NATIVE_DATA_SOURCE_LIBRARY`,
now says that the package or the library "implements" the version rather than
"requires" it: the version in a manifest is the oldest that its library
implements, and the version of a loaded library is the one that its table
implements, which can also be older than the version of the interface that
Spark supports.
- The doc of `spark.sql.dataSource.native.paths` now says that the packages
that a session adds take precedence over the configured ones, and the doc of
`spark.sql.dataSource.native.enabled` that, when it is false,
`spark.addArtifact` adds a `.sparkpkg` file as a file of the session, without
the check. The docs of native data sources, the API docs of `addArtifact` and
`addArtifacts`, and the migration guide describe the new behavior. A comment in
`NativeDataSourceRegistry.lookupPackag [...]
### Why are the changes needed?
During the review of SPARK-60101, the package that a session added last was
given precedence over its other packages and the configured ones, so that a
session can upgrade a package by adding a new version under another file name:
a session cannot remove an artifact. But the lookup drops the packages that
Spark cannot use before it picks the last one. So a session that adds
`rs-1.0.sparkpkg`, and then `rs-2.0.sparkpkg` whose manifest is broken or
declares a newer `abiVersion`, keeps u [...]
Failing when the session adds the package reports the problem to the user
who added it, also for a manifest that cannot be read at all, whose data
sources Spark does not know when it looks one up. A session then does not get a
package that is invalid, of a version that Spark does not support, or without a
library for the driver, and it can recover by adding a fixed package. Since the
file is checked before it becomes an artifact, an invalid package is never at
the path of an artifact, [...]
### Does this PR introduce _any_ user-facing change?
No for native data sources, which are not released yet (4.4.0). One effect
on an existing API: unless `spark.sql.dataSource.native.enabled` is false,
`spark.addArtifact` now rejects a file artifact named `*.sparkpkg` that is not
a valid package, or whose version Spark does not support, or that has no
library for the platform of the driver, which earlier releases added as a file,
a Python file or an archive, with `file=True`, `pyfile=True` or `archive=True`
in Python without Spark Conn [...]
### How was this patch tested?
A new test in `NativeDataSourceSuite`, `reject a package that a session
adds when Spark cannot use it`, adds a package with a newer `abiVersion`, a
package whose manifest names a library that it does not contain, a package
whose manifest names a directory entry as a library, a package whose manifest
is a directory entry, a file that is not a zip, a directory, and a package
without a library for the platform, each twice. It checks that each add fails
with its error, which names the fil [...]
A new test in `NativeDataSourceConnectSuite` adds a file that is not a
package and a package with a newer `abiVersion` through a Spark Connect client,
and checks that the client gets the errors, which name the artifacts rather
than the files in which the server staged them. It then adds an invalid package
and a valid one in one call, and checks that the session gets the valid one.
`test_add_native_data_source_package` in `pyspark.sql.tests.test_artifact`
added files that are not packa [...]
Mutation check (each mutant was run on the latest commit that changed the
code it mutates): reverting each of the following makes a test fail: the check,
the version check, the platform check, the restriction to `files/`, inverting
or dropping the condition on `spark.sql.dataSource.native.enabled`, collecting
the errors in `addLocalArtifacts` and in the Spark Connect handler, the copy of
the file of the user, comparing the file with the artifact before the check,
local and staged, `AR [...]
`NativeDataSourceSuite`, `ColumnarDataSourceSuite`, `ArtifactManagerSuite`,
`NativeDataSourceConnectSuite`, `AddArtifactsHandlerSuite`,
`ArtifactStatusesHandlerSuite`, `SparkThrowableSuite` and that Python test pass
locally on Java 22 on macOS, except a test that only runs before Java 22. The
step "Test native data sources" of the Java 25 job of the PR builds runs
`NativeDataSourceSuite`, `NativeDataSourceClusterSuite` and
`NativeDataSourceConnectSuite`, which need Java 22 or later.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Kiro CLI
Closes #59947 from sarutak/native-ds-added-package-check.
Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Kousuke Saruta <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 4 +-
docs/sql-migration-guide.md | 1 +
docs/sql-native-data-sources.md | 41 +++--
python/pyspark/sql/session.py | 9 +-
python/pyspark/sql/tests/test_artifact.py | 33 +++-
.../scala/org/apache/spark/sql/SparkSession.scala | 29 +++-
.../org/apache/spark/sql/internal/SQLConf.scala | 6 +-
.../apache/spark/sql/internal/StaticSQLConf.scala | 3 +-
.../service/SparkConnectAddArtifactsHandler.scala | 2 +-
.../sql/connect/NativeDataSourceConnectSuite.scala | 39 +++++
.../spark/sql/artifact/ArtifactManager.scala | 179 +++++++++++++++++--
.../v2/ffi/NativeDataSourcePackage.scala | 127 ++++++++++++--
.../v2/ffi/NativeDataSourceRegistry.scala | 19 +--
.../spark/sql/artifact/ArtifactManagerSuite.scala | 27 +++
.../datasources/v2/ffi/NativeDataSourceSuite.scala | 190 ++++++++++++++++++++-
15 files changed, 634 insertions(+), 75 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 3b63443bb7a9..b25edfd3a790 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -4891,7 +4891,7 @@
},
"UNSUPPORTED_ABI_VERSION" : {
"message" : [
- "It requires version <version> of the native data source interface,
but Spark supports version <supported>."
+ "It implements version <version> of the native data source
interface, but Spark supports version <supported>."
]
}
},
@@ -4914,7 +4914,7 @@
},
"UNSUPPORTED_ABI_VERSION" : {
"message" : [
- "It requires version <version> of the native data source interface,
but Spark supports version <supported>."
+ "It implements version <version> of the native data source
interface, but Spark supports version <supported>."
]
},
"UNSUPPORTED_PLATFORM" : {
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 2e5a155de3e1..828be69dc31c 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -36,6 +36,7 @@ license: |
- Since Spark 4.4, quoted Variant path keys decode `\\`, `\/`, `\b`, `\f`,
`\n`, `\r`, `\t`, `\uXXXX` (including valid surrogate pairs), and escapes of
the enclosing quote (`\'` or `\"`). Previously, backslashes were literal, so
existing paths can match different keys or become invalid. For example, use
`$['a\\']` instead of `$['a\']`. Unknown escapes and unpaired surrogate escapes
stay literal. Dot-notation keys are unchanged. To keep the old behavior, set
`spark.sql.legacy.variantPathB [...]
- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a column referenced in a lambda function, such as `i` in `transform(a, x
-> x + i)`, or with `IDENTIFIER`, such as `IDENTIFIER('i')`, from the value
stored in that column. Previously, it read the value of the query before the
store assignment cast, so it could silently store a value that does not match
the stored column, and a later write of that row failed with
`CHECK_CONSTRAINT_VIOLATION` (SPARK-60817). [...]
- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a field of another column, such as `s.f`, from the value stored in that
column. Previously, it read the value of the query before the store assignment
cast, so it could silently store a value that does not match the stored column,
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION`
(SPARK-60803). Rows that are already stored keep their values, and data derived
from them may need to be re [...]
+- Since Spark 4.4, `spark.addArtifact` and `spark.addArtifacts` take a file
whose name ends with `.sparkpkg` as a [native data source
package](sql-native-data-sources.html), also when it is added with `file=True`,
`pyfile=True` or `archive=True` in Python without Spark Connect, or with
`file=True` from a Spark Connect client, and fail with
`INVALID_NATIVE_DATA_SOURCE_PACKAGE` for such a file that is not a valid
package, or whose version Spark does not support, or that has no library for
[...]
- Since Spark 4.4, `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled`
is `false` by default: Arrow-based Python UDFs read rows from an Arrow-backed
upstream operator, such as a Data Source V2 scan or the Arrow cache, as before
Spark 4.2. With columnar input, they kept the columns that they pass through
until the results of their batch came back from Python, and failed or returned
wrong results when the operator reused or released the vectors of the batch in
the meantime. To enab [...]
- Since Spark 4.4, the conversion of columnar batches to rows releases each
batch with `ColumnVector.closeIfFreeable` once its rows are consumed also when
it is not whole-stage code-generated, e.g. when the query has more than
`spark.sql.codegen.maxFields` fields or `spark.sql.codegen.wholeStage` is
`false`, as the generated code has done since Spark 3.5.4 and 4.0. A
`ColumnVector` that a Data Source V2 source reuses across batches, such as an
`ArrowColumnVector` over a `VectorSchemaRoot [...]
- Since Spark 4.4, `spark.sql.charVarchar.standardSemantics.enabled` (default
`false`) is a preview feature that may change in future releases. When true, it
enables first-class `CHAR(n)` and `VARCHAR(n)` semantics throughout schemas and
query plans: `CHAR` -> `VARCHAR` -> `STRING` form a type-precedence family,
explicit casts can return constrained types, least-common-type resolution for
`COALESCE`/`CASE`/`UNION` can return `CHAR` or `VARCHAR`, and string
expressions that transform thei [...]
diff --git a/docs/sql-native-data-sources.md b/docs/sql-native-data-sources.md
index aa2df7bf37c3..30dce4be43f4 100644
--- a/docs/sql-native-data-sources.md
+++ b/docs/sql-native-data-sources.md
@@ -305,9 +305,9 @@ is a zip file with the extension `.sparkpkg`, which
contains a manifest named
- `dataSources` lists the names of the data sources that the library
implements. The names are
case-insensitive, and Spark passes the name to `create`.
- `libraries` maps each platform, `<os>-<arch>` with an `os` of `linux`, `osx`
or `windows` and an
- `arch` of `x86_64` or `aarch64`, to the path of the library in the package.
`linux` is Linux with
- glibc: Linux with musl, as Alpine Linux, is `linux-<arch>-musl`, such as
`linux-x86_64-musl`, on
- which native data sources do not run yet.
+ `arch` of `x86_64` or `aarch64`, to the path of the library file in the
package. `linux` is Linux
+ with glibc: Linux with musl, as Alpine Linux, is `linux-<arch>-musl`, such
as `linux-x86_64-musl`,
+ on which native data sources do not run yet.
A session finds the packages in two places, and the executors get them
automatically:
@@ -325,12 +325,31 @@ A session finds the packages in two places, and the
executors get them automatic
The packages of a session take precedence over the installed libraries, and
the package that a
session added last over its other packages, also the configured ones: a
session cannot remove an
-artifact, or add a file with the same name again, so it upgrades a package by
adding the new
-version under another file name, such as `rust_range-1.1.sparkpkg`. Distinct
file names are also
-needed because the sessions that are not isolated share the files they add. A
query fails if more
-than one configured package provides its data source. Spark ignores, and logs,
a package that it
-cannot read or whose version it does not support, unless its manifest lists
the data source that a
-query uses and no other package or installed library provides it.
+artifact, or add another file with the name of one that it added, so it
upgrades a package by adding
+the new version under another file name, such as `rust_range-1.1.sparkpkg`.
Distinct file names are
+also needed because the sessions that are not isolated share the files they
add.
+
+A query fails if more than one configured package provides its data source.
Spark ignores, and logs,
+a configured package that it cannot read or whose version it does not support,
unless its manifest
+lists the data source that a query uses and no other package or installed
library provides it.
+
+Unless `spark.sql.dataSource.native.enabled` is false, `spark.addArtifact`
fails for a file that is
+not a valid package, or for a package whose version Spark does not support, or
that has no library
+for the platform of the driver, which loads the library to plan a query,
instead of adding a package
+that the session could not use: the session would ignore it and silently keep
using the package or
+library that it used before, or fail every query on its data sources. After
the failure, the session
+keeps the packages that it had, and a call that adds several artifacts in
Scala or Java still adds
+the others. In Python, a rejected package stops the rest of the call: without
Spark Connect,
+`spark.addArtifacts` adds the packages first, one at a time, so only the
packages before it are
+added; with Spark Connect, it adds the paths one at a time, so the paths
before the package are
+added. A file that does not exist or cannot be read stops the call, as for the
other artifacts.
+
+Spark does not load the library when it checks a package, so a library that
fails to load fails the
+queries that use it. Nor does it check that the driver can run native data
sources at all: before
+Java 22, a valid package is added, and the queries fail with
+`NATIVE_DATA_SOURCE_UNSUPPORTED_JAVA_VERSION`; on a platform where the C data
interface of Arrow
+Java does not load, such as Linux with musl, only a package with a library for
that platform is
+added, and the queries fail with `NATIVE_DATA_SOURCE_UNSUPPORTED_PLATFORM`.
## Example in Rust
@@ -760,8 +779,8 @@ statically, or install them on the nodes.
| Property Name | Default | Meaning |
|---------------|---------|---------|
-| `spark.sql.dataSource.native.enabled` | true | Whether Spark loads native
data sources. It is a static configuration, so it can only be set when the
Spark application starts. |
-| `spark.sql.dataSource.native.paths` | (none) | Comma-separated list of
native data source packages, and of directories that contain them, on the local
file system of the driver. They take precedence over the installed libraries. |
+| `spark.sql.dataSource.native.enabled` | true | Whether Spark loads native
data sources. It is a static configuration, so it can only be set when the
Spark application starts. When false, `spark.addArtifact` adds a `.sparkpkg`
file as a file of the session, without checking it as a package. |
+| `spark.sql.dataSource.native.paths` | (none) | Comma-separated list of
native data source packages, and of directories that contain them, on the local
file system of the driver. They take precedence over the installed libraries,
and the packages that a session adds with `spark.addArtifact` take precedence
over them. |
## Security
diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py
index 916d4a1640ea..9205612fc500 100644
--- a/python/pyspark/sql/session.py
+++ b/python/pyspark/sql/session.py
@@ -2275,7 +2275,14 @@ class SparkSession(SparkConversionMixin):
----------
*path : tuple of str
Artifact's URIs to add. A native data source package (a
``.sparkpkg`` file) makes
- the native data sources it contains available to the session.
+ the native data sources it contains available to the session. The
call fails for a
+ file that is not a valid package, for a package whose version
Spark does not
+ support, or that has no library for the platform of the driver;
the session then
+ does not get the package. With
``spark.sql.dataSource.native.enabled`` set to false,
+ such a file is added as a file of the session, without the check.
A package that is
+ rejected also stops the rest of the call: without Spark Connect,
the packages are
+ added first, one at a time, so only the packages before it are
added; with Spark
+ Connect, the paths are added one at a time, so the paths before it
are added.
pyfile : bool
Whether to add them as Python dependencies such as .py, .egg, .zip
or .jar files.
The pyfiles are directly inserted into the path when executing
Python functions
diff --git a/python/pyspark/sql/tests/test_artifact.py
b/python/pyspark/sql/tests/test_artifact.py
index 73f18abc9676..9328efebb614 100644
--- a/python/pyspark/sql/tests/test_artifact.py
+++ b/python/pyspark/sql/tests/test_artifact.py
@@ -14,11 +14,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+import json
import os
import tempfile
+import zipfile
from pathlib import Path
-from pyspark.errors import PySparkRuntimeError
+from pyspark.errors import PySparkRuntimeError, SparkRuntimeException
from pyspark.sql.functions import assert_true, lit, udf
from pyspark.sql.tests.connect.client.test_artifact import ArtifactTestsMixin
from pyspark.testing.sqlutils import ReusedSQLTestCase
@@ -110,11 +112,25 @@ class ArtifactTests(ArtifactTestsMixin,
ReusedSQLTestCase):
# A new session, so that the packages do not affect the other tests.
spark = self.spark.newSession()
with
tempfile.TemporaryDirectory(prefix="test_add_native_data_source_package") as d:
+ ffi = spark._jvm.org.apache.spark.sql.execution.datasources.v2.ffi
+ platform = ffi.NativePlatform.current()
def new_file(name):
file_path = os.path.join(d, name)
- with open(file_path, "wb") as f:
- f.write(name.encode("utf-8"))
+ if name.endswith(".sparkpkg"):
+ # Spark checks the manifest of a package when a session
adds it, which has to
+ # have a library for the platform of the driver.
+ manifest = {
+ "abiVersion": 1,
+ "dataSources": ["my_source"],
+ "libraries": {platform: "lib.so"},
+ }
+ with zipfile.ZipFile(file_path, "w") as z:
+ z.writestr("spark-native-datasource.json",
json.dumps(manifest))
+ z.writestr("lib.so", name)
+ else:
+ with open(file_path, "wb") as f:
+ f.write(name.encode("utf-8"))
return file_path
spark.addArtifact(new_file("my_source.sparkpkg"))
@@ -134,6 +150,17 @@ class ArtifactTests(ArtifactTestsMixin, ReusedSQLTestCase):
spark.addArtifact("my_source:1.sparkpkg")
finally:
os.chdir(cwd)
+ # A file that is not a valid package is rejected.
+ invalid = os.path.join(d, "invalid.sparkpkg")
+ with open(invalid, "wb") as f:
+ f.write(b"not a zip")
+ with self.assertRaises(SparkRuntimeException) as e:
+ spark.addArtifact(invalid)
+ self.assertEqual(
+ e.exception.getCondition(),
"INVALID_NATIVE_DATA_SOURCE_PACKAGE.INVALID_MANIFEST"
+ )
+ # It names the file of the user, which the JVM resolves to its
canonical path.
+ self.assertEqual(e.exception.getMessageParameters()["path"],
os.path.realpath(invalid))
packages =
spark._jsparkSession.artifactManager().getNativeDataSourcePackages()._1()
self.assertEqual(
sorted(packages.apply(i).getName() for i in
range(packages.size())),
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/SparkSession.scala
b/sql/api/src/main/scala/org/apache/spark/sql/SparkSession.scala
index cce193fe50db..7f9f403b8983 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/SparkSession.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/SparkSession.scala
@@ -569,7 +569,11 @@ abstract class SparkSession extends Serializable with
Closeable {
* Ivy repositories local to the client.
*
* Since 4.4.0, it also supports native data source packages: local files
with the extension
- * .sparkpkg, which make the native data sources that they contain available
to the session.
+ * .sparkpkg, which make the native data sources that they contain available
to the session. It
+ * fails for a file that is not a valid package, for a package whose version
Spark does not
+ * support, or that has no library for the platform of the driver; the
session then does not get
+ * the package. With `spark.sql.dataSource.native.enabled` set to false,
such a file is added as
+ * a file of the session, without the check.
*
* @since 4.0.0
*/
@@ -587,7 +591,11 @@ abstract class SparkSession extends Serializable with
Closeable {
* Ivy repositories local to the client.
*
* Since 4.4.0, it also supports native data source packages: local files
with the extension
- * .sparkpkg, which make the native data sources that they contain available
to the session.
+ * .sparkpkg, which make the native data sources that they contain available
to the session. It
+ * fails for a file that is not a valid package, for a package whose version
Spark does not
+ * support, or that has no library for the platform of the driver; the
session then does not get
+ * the package. With `spark.sql.dataSource.native.enabled` set to false,
such a file is added as
+ * a file of the session, without the check.
*
* @since 4.0.0
*/
@@ -601,7 +609,10 @@ abstract class SparkSession extends Serializable with
Closeable {
*
* Supported target file extensions are .jar and .class, and since 4.4.0,
.sparkpkg for native
* data source packages, which make the native data sources that they
contain available to the
- * session.
+ * session. It fails for a file that is not a valid package, for a package
whose version Spark
+ * does not support, or that has no library for the platform of the driver;
the session then
+ * does not get the package. With `spark.sql.dataSource.native.enabled` set
to false, such a
+ * file is added as a file of the session, without the check.
*
* ==Example==
* {{{
@@ -623,7 +634,10 @@ abstract class SparkSession extends Serializable with
Closeable {
*
* Supported target file extensions are .jar and .class, and since 4.4.0,
.sparkpkg for native
* data source packages, which make the native data sources that they
contain available to the
- * session.
+ * session. It fails for a file that is not a valid package, for a package
whose version Spark
+ * does not support, or that has no library for the platform of the driver;
the session then
+ * does not get the package. With `spark.sql.dataSource.native.enabled` set
to false, such a
+ * file is added as a file of the session, without the check.
*
* ==Example==
* {{{
@@ -650,7 +664,12 @@ abstract class SparkSession extends Serializable with
Closeable {
* Ivy repositories local to the client.
*
* Since 4.4.0, it also supports native data source packages: local files
with the extension
- * .sparkpkg, which make the native data sources that they contain available
to the session.
+ * .sparkpkg, which make the native data sources that they contain available
to the session. A
+ * file that is not a valid package, or a package whose version Spark does
not support, or that
+ * has no library for the platform of the driver, is rejected: the session
does not get it, and
+ * the call fails after it adds the other artifacts. With
`spark.sql.dataSource.native.enabled`
+ * set to false, such a file is added as a file of the session, without the
check. A file that
+ * does not exist or cannot be read stops the call, as for the other
artifacts.
*
* @since 4.0.0
*/
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 92a75658258a..a5d9a2f322f9 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -7502,9 +7502,9 @@ object SQLConf {
.doc("Comma-separated list of native data source packages (.sparkpkg
files), and of " +
"directories that contain them, on the local file system of the driver.
Spark adds a " +
"package to the artifacts of the session when one of its data sources is
used, so that " +
- "the executors get it too. Packages can also be added with
spark.addArtifact. These " +
- "packages take precedence over the native data source libraries
installed in the native " +
- "library path. See spark.sql.dataSource.native.enabled.")
+ "the executors get it too. Packages added with spark.addArtifact take
precedence over " +
+ "these packages, which take precedence over the native data source
libraries installed " +
+ "in the native library path. See spark.sql.dataSource.native.enabled.")
.version("4.4.0")
.withBindingPolicy(ConfigBindingPolicy.SESSION)
.stringConf
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
index 923c6cec3320..e3624aab4bac 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
@@ -474,7 +474,8 @@ object StaticSQLConf {
"of each installation prefix in PATH. Libraries can also be
distributed in native data " +
"source packages (.sparkpkg files), added with spark.addArtifact or
under " +
"spark.sql.dataSource.native.paths. Loading a native data source runs
its native code " +
- "in the driver and executor processes.")
+ "in the driver and executor processes. When false, spark.addArtifact
adds a .sparkpkg " +
+ "file as a file of the session, without checking it as a package.")
.version("4.4.0")
.withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectAddArtifactsHandler.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectAddArtifactsHandler.scala
index 91180a17950f..643f45ebaa19 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectAddArtifactsHandler.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectAddArtifactsHandler.scala
@@ -225,7 +225,7 @@ class SparkConnectAddArtifactsHandler(val responseObserver:
StreamObserver[AddAr
}
}
} catch {
- case e: SparkRuntimeException if e.getCondition ==
"ARTIFACT_ALREADY_EXISTS" =>
+ case e: SparkRuntimeException if ArtifactManager.isRejectedArtifact(e)
=>
failedArtifactExceptions += e
}
artifact.summary()
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/NativeDataSourceConnectSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/NativeDataSourceConnectSuite.scala
index 97c4b3ae6104..2b7407340537 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/NativeDataSourceConnectSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/NativeDataSourceConnectSuite.scala
@@ -16,6 +16,7 @@
*/
package org.apache.spark.sql.connect
+import java.io.File
import java.nio.file.Files
import java.util.UUID
@@ -92,6 +93,44 @@ class NativeDataSourceConnectSuite extends
SparkConnectServerTest {
}
}
+ nativeTest("reject a package that Spark cannot use when a session adds it") {
+ val invalid = new File(Utils.createTempDir(), "native_range.sparkpkg")
+ Files.writeString(invalid.toPath, "not a zip")
+ withNewSession { session =>
+ // The client gets the error of the server in the message of a
SparkException.
+ val e = intercept[SparkException](session.addArtifact(invalid.getPath))
+
assert(e.getMessage.contains("[INVALID_NATIVE_DATA_SOURCE_PACKAGE.INVALID_MANIFEST]"),
e)
+ // It names the artifact, rather than the file in which the server
staged it.
+ assert(e.getMessage.contains(s"package
files${File.separator}native_range.sparkpkg:"), e)
+ // A package whose version Spark does not support is named after the
artifact too.
+ val newer = createPackage(library, dataSources(""), abiVersion = 2)
+ val unsupported =
intercept[SparkException](session.addArtifact(newer.getPath))
+ assert(
+ unsupported.getMessage.contains(
+ "[INVALID_NATIVE_DATA_SOURCE_PACKAGE.UNSUPPORTED_ABI_VERSION]"),
+ unsupported)
+ assert(
+ unsupported.getMessage.contains(s"package
files${File.separator}${newer.getName}:"),
+ unsupported)
+ // A call that adds several artifacts adds the others: the stream sends
them all.
+ val valid = createPackage(library, dataSources(""), fileName =
invalid.getName)
+ val other = new File(Utils.createTempDir(), "other.sparkpkg")
+ Files.writeString(other.toPath, "not a zip")
+ val rejected = intercept[SparkException] {
+ session.addArtifacts(other.toURI, valid.toURI)
+ }
+ assert(
+
rejected.getMessage.contains("[INVALID_NATIVE_DATA_SOURCE_PACKAGE.INVALID_MANIFEST]"),
+ rejected)
+ assert(
+ rejected.getMessage.contains(s"package
files${File.separator}${other.getName}:"),
+ rejected)
+ // The valid package has the file name of the rejected one, which the
session can add then.
+ val df = session.read.format("native_range").option("end", "3").load()
+ assert(df.select("id").collect().map(_.getLong(0)).sorted.toSeq ==
Seq(0L, 1L, 2L))
+ }
+ }
+
nativeTest("find installed libraries automatically") {
val dir = Utils.createTempDir()
installLibrary(installedLibrary, "installed_native_range", dir)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala
b/sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala
index e0fb397d36cd..8e6f7f920f1e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala
@@ -40,7 +40,7 @@ import
org.apache.spark.sql.execution.datasources.v2.ffi.NativeDataSourcePackage
import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
import org.apache.spark.sql.util.ArtifactUtils
import org.apache.spark.storage.{BlockManager, CacheId, StorageLevel}
-import org.apache.spark.util.{ChildFirstURLClassLoader,
RuntimeDependencyResolver, StubClassLoader, Utils}
+import org.apache.spark.util.{ChildFirstURLClassLoader,
RuntimeDependencyResolver, ShutdownHookManager, StubClassLoader, Utils}
import
org.apache.spark.util.RuntimeDependencyResolver.{AllowRequestedRepositories,
RepositoryPolicy}
/**
@@ -160,13 +160,24 @@ class ArtifactManager(session: SparkSession) extends
AutoCloseable with Logging
*/
private[sql] def getNativeDataSourcePackages: (Seq[File], Option[String]) = {
val packages = sparkContextRelativePaths.asScala.collect {
- case (SparkContextResourceType.FILE, path, _)
- if
path.toString.endsWith(Artifact.NATIVE_DATA_SOURCE_PACKAGE_EXTENSION) =>
+ case (SparkContextResourceType.FILE, path, _) if
isNativeDataSourcePackage(path) =>
artifactPath.resolve(path).toFile
}.toSeq
(packages, Option(state).map(_.uuid))
}
+ /** Whether Spark checks an artifact at the given path as a native data
source package. */
+ private def isCheckedPackage(path: Path): Boolean = {
+ isNativeDataSourcePackage(normalizePath(path)) &&
+ session.conf.get(StaticSQLConf.NATIVE_DATA_SOURCE_ENABLED)
+ }
+
+ /** Whether an artifact at the given normalized path is a native data source
package. */
+ private def isNativeDataSourcePackage(path: Path): Boolean = {
+ path.startsWith(s"files${File.separator}") &&
+ path.toString.endsWith(Artifact.NATIVE_DATA_SOURCE_PACKAGE_EXTENSION)
+ }
+
/** Returns whether a file with the given name was added to this session as
a file artifact. */
private[sql] def hasFileArtifact(fileName: String): Boolean = {
val path = Paths.get("files", fileName)
@@ -204,9 +215,111 @@ class ArtifactManager(session: SparkSession) extends
AutoCloseable with Logging
// Convert the normalized string back to a Path object
Paths.get(normalizedPathString).normalize()
}
+
+ /**
+ * Returns true if an artifact with the same content is at `target` already,
which makes
+ * adding it idempotent, and fails with ARTIFACT_ALREADY_EXISTS if another
file is there: an
+ * artifact is not overwritten.
+ */
+ private def artifactAlreadyAdded(
+ normalizedRemoteRelativePath: Path,
+ serverLocalStagingPath: Path,
+ target: Path): Boolean = {
+ if (Files.exists(target)) {
+ if (Utils.contentEquals(target.toFile, serverLocalStagingPath.toFile)) {
+ return true
+ }
+ throw artifactAlreadyExistsError(normalizedRemoteRelativePath)
+ }
+ false
+ }
+
+ private def artifactAlreadyExistsError(normalizedRemoteRelativePath: Path):
Throwable = {
+ new SparkRuntimeException(
+ "ARTIFACT_ALREADY_EXISTS",
+ Map("normalizedRemoteRelativePath" ->
normalizedRemoteRelativePath.toString)
+ )
+ }
+
+ /**
+ * Transfers a native data source package to `target` if Spark can use it,
see
+ * [[NativeDataSourcePackage.checkAdded]], so that it never is at the path
of an artifact
+ * otherwise. Returns false if the session has the same file already, and
fails with
+ * ARTIFACT_ALREADY_EXISTS if it has another one, as artifactAlreadyAdded
does, or with the error
+ * of the check. The file is compared with the artifact first, so that
adding the same file again
+ * neither copies nor checks it. A file of the user is then copied to a
temporary directory,
+ * outside of the artifacts, which the file server serves and a clone of the
session copies, and
+ * the copy is checked and then moved, so that the session gets the bytes
that are checked. The
+ * errors of the check name the file of the user, or the artifact if Spark
staged the file for
+ * it, such as for a Spark Connect client.
+ */
+ private def addNativeDataSourcePackage(
+ normalizedRemoteRelativePath: Path,
+ serverLocalStagingPath: Path,
+ target: Path,
+ deleteStagedFile: Boolean,
+ sourceName: Option[String]): Boolean = {
+ // Returns true if the same file was added already, and fails with
ARTIFACT_ALREADY_EXISTS if
+ // another one was, as artifactAlreadyAdded does.
+ def alreadyAdded(source: Path): Boolean = {
+ // Utils.contentEquals fails for a directory, which cannot be the same
as a package anyway,
+ // so a directory with the name of an artifact, or an artifact that is a
directory, such
+ // as files/x.sparkpkg after files/x.sparkpkg/y was added, fails as an
existing artifact.
+ if (Files.exists(target) && (Files.isDirectory(source) ||
Files.isDirectory(target))) {
+ throw artifactAlreadyExistsError(normalizedRemoteRelativePath)
+ }
+ artifactAlreadyAdded(normalizedRemoteRelativePath, source, target)
+ }
+ // ARTIFACT_ALREADY_EXISTS comes before the check, which is then not
needed for the same file.
+ if (alreadyAdded(serverLocalStagingPath)) {
+ return false
+ }
+ val copyDir = if (deleteStagedFile) None else Some(Utils.createTempDir())
+ Utils.tryWithSafeFinally {
+ val staged = copyDir.map { dir =>
+ // Fails as for the other artifacts if the file does not exist or
cannot be read.
+ val copy = dir.toPath.resolve(normalizedRemoteRelativePath.getFileName)
+ Files.copy(serverLocalStagingPath, copy)
+ // The copy keeps the permissions of the file, or on Windows its
read-only attribute.
+ // Since Java 25, File.delete does not delete a read-only file on
Windows, so neither the
+ // copy of a rejected package nor the artifact, moved from the copy,
could be deleted.
+ copy.toFile.setWritable(true)
+ copy
+ }.getOrElse(serverLocalStagingPath)
+ NativeDataSourcePackage.checkAdded(
+ staged.toFile,
sourceName.getOrElse(normalizedRemoteRelativePath.toString))
+ // Again right before the transfer, in case another add of the same
artifact ran meanwhile.
+ if (alreadyAdded(staged)) {
+ false
+ } else {
+ transferFile(staged, target, deleteSource = true)
+ true
+ }
+ } {
+ copyDir.foreach(deleteTemporaryDirectory)
+ }
+ }
+
+ /**
+ * Deletes a temporary directory of an add without throwing, so that neither
a failure nor an
+ * interrupt, which Utils.deleteRecursively rethrows, replaces the result or
the error of the
+ * add: the artifact may already be at its path. The interrupt status is
kept.
+ */
+ private def deleteTemporaryDirectory(dir: File): Unit = {
+ Utils.deleteQuietly(dir)
+ if (dir.exists()) {
+ logWarning(log"Failed to delete the temporary directory
${MDC(LogKeys.PATH, dir)}.")
+ } else {
+ ShutdownHookManager.removeShutdownDeleteDir(dir)
+ }
+ }
+
/**
* Add and prepare a staged artifact (i.e an artifact that has been rebuilt
locally from bytes
- * over the wire) for use.
+ * over the wire) for use. `addLocalArtifacts` calls it for the artifacts
that it adds, except for
+ * a local native data source package that Spark checks, while
+ * `spark.sql.dataSource.native.enabled` is true, which it adds through a
private overload, so
+ * that the errors of its check name the file of the user.
*
* @param remoteRelativePath
* @param serverLocalStagingPath
@@ -218,6 +331,22 @@ class ArtifactManager(session: SparkSession) extends
AutoCloseable with Logging
serverLocalStagingPath: Path,
fragment: Option[String],
deleteStagedFile: Boolean = true
+ ): Unit = {
+ addArtifact(remoteRelativePath, serverLocalStagingPath, fragment,
deleteStagedFile, None)
+ }
+
+ /**
+ * Adds an artifact, as [[addArtifact]] above. `sourceName` is the name of
the file that the
+ * user added, if `serverLocalStagingPath` is that file, which the errors of
the check of a
+ * native data source package then give instead of the artifact, rather than
a file in which
+ * Spark staged the artifact.
+ */
+ private def addArtifact(
+ remoteRelativePath: Path,
+ serverLocalStagingPath: Path,
+ fragment: Option[String],
+ deleteStagedFile: Boolean,
+ sourceName: Option[String]
): Unit = JobArtifactSet.withActiveJobArtifactState(state) {
require(!remoteRelativePath.isAbsolute)
val normalizedRemoteRelativePath = normalizePath(remoteRelativePath)
@@ -273,19 +402,19 @@ class ArtifactManager(session: SparkSession) extends
AutoCloseable with Logging
cachedClassLoader = None
} else {
val target = ArtifactUtils.concatenatePaths(artifactPath,
normalizedRemoteRelativePath)
- // Disallow overwriting with modified version
- if (Files.exists(target)) {
- // makes the query idempotent
- if (Utils.contentEquals(target.toFile, serverLocalStagingPath.toFile))
{
+ if (isCheckedPackage(normalizedRemoteRelativePath)) {
+ // Fails if another file is the artifact, or if Spark cannot use the
package.
+ if (!addNativeDataSourcePackage(normalizedRemoteRelativePath,
serverLocalStagingPath,
+ target, deleteStagedFile, sourceName)) {
return
}
-
- throw new SparkRuntimeException(
- "ARTIFACT_ALREADY_EXISTS",
- Map("normalizedRemoteRelativePath" ->
normalizedRemoteRelativePath.toString)
- )
+ } else {
+ // Disallow overwriting with modified version: fails if another file
is the artifact.
+ if (artifactAlreadyAdded(normalizedRemoteRelativePath,
serverLocalStagingPath, target)) {
+ return
+ }
+ transferFile(serverLocalStagingPath, target, deleteSource =
deleteStagedFile)
}
- transferFile(serverLocalStagingPath, target, deleteSource =
deleteStagedFile)
// This URI is for Spark file server that starts with "spark://".
val uri = s"$artifactURI/${Utils.encodeRelativeUnixPathToURIRawPath(
@@ -334,6 +463,14 @@ class ArtifactManager(session: SparkSession) extends
AutoCloseable with Logging
artifacts.foreach { artifact =>
try {
artifact.storage match {
+ case d: Artifact.LocalFile if isCheckedPackage(artifact.path) =>
+ // The errors of its check name the file of the user.
+ addArtifact(
+ artifact.path,
+ d.path,
+ fragment = None,
+ deleteStagedFile = false,
+ sourceName = Some(d.path.toString))
case d: Artifact.LocalFile =>
addArtifact(
artifact.path,
@@ -348,13 +485,14 @@ class ArtifactManager(session: SparkSession) extends
AutoCloseable with Logging
d.stream.transferTo(outStream)
addArtifact(artifact.path, tempFile, fragment = None)
}(finallyBlock = {
- outStream.close()
+ // addArtifact moves the file only when it adds the artifact, so
delete what is left.
+
Utils.tryWithSafeFinally(outStream.close())(deleteTemporaryDirectory(tempDir.toFile))
})
case _ =>
throw SparkException.internalError(s"Unsupported artifact storage:
${artifact.storage}")
}
} catch {
- case e: SparkRuntimeException if e.getCondition ==
"ARTIFACT_ALREADY_EXISTS" =>
+ case e: SparkRuntimeException if ArtifactManager.isRejectedArtifact(e)
=>
failedArtifactExceptions += e
}
}
@@ -608,6 +746,15 @@ object ArtifactManager extends Logging {
val JAR, FILE, ARCHIVE = Value
}
+ /**
+ * Whether an error of adding an artifact only rejects that artifact, so
that the call that adds
+ * several artifacts adds the others, and then reports all such errors.
+ */
+ private[sql] def isRejectedArtifact(e: SparkRuntimeException): Boolean = {
+ e.getCondition == "ARTIFACT_ALREADY_EXISTS" ||
+
Option(e.getCondition).exists(_.startsWith("INVALID_NATIVE_DATA_SOURCE_PACKAGE."))
+ }
+
private class StateCleanupRunner(cleanupState: ArtifactStateForCleanup)
extends Runnable {
override def run(): Unit = {
ArtifactManager.cleanUpGlobalResources(cleanupState)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourcePackage.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourcePackage.scala
index cc49dbc0cd73..9d45abaedcb9 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourcePackage.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourcePackage.scala
@@ -132,6 +132,62 @@ object NativeDataSourcePackage extends Logging {
/** Reads the package in the given file. The result is cached until the file
changes. */
def read(file: File): NativeDataSourcePackage = tryRead(file).fold(p =>
throw p.error, identity)
+ /**
+ * Checks a package that a session adds with `spark.addArtifact`, in the
given file, before the
+ * session gets it: fails if the file is not a valid package, if Spark does
not support its
+ * version of the interface, or if it has no library for the platform of the
driver, which loads
+ * the library to plan a query. The package that a session added last takes
precedence, and a
+ * session cannot remove an artifact. Ignoring such a package, as Spark
ignores an unusable
+ * configured one, would silently keep the session on the package or library
that it used
+ * before, and accepting a package without a library for this platform would
fail every query
+ * on its data sources, so Spark rejects both. The library is not loaded, so
a library that
+ * fails to load fails the queries that use it, and neither is it checked
that the driver can
+ * run native data sources at all: before Java 22, a valid package is added
and every query on
+ * its data sources fails, and on a platform where the C data interface of
Arrow Java does not
+ * load, such as Linux with musl, only a package with a library for that
platform is added, and
+ * the queries fail too. The file is not the artifact itself, so the result
is not cached, and
+ * errors call it `path`.
+ */
+ def checkAdded(file: File, path: String): Unit = {
+ val manifest = readManifest(file, path).fold(invalid => throw
invalid.error, identity)
+ if (!isSupported(manifest)) {
+ throw unsupportedAbiVersionError(path, manifest)
+ }
+ if (!manifest.libraries.contains(NativePlatform.current)) {
+ throw unsupportedPlatformError(path, manifest)
+ }
+ }
+
+ /** Whether Spark supports the version of the interface of a package. */
+ private[ffi] def isSupported(manifest: NativeDataSourceManifest): Boolean = {
+ // The version of a package is the oldest that its library implements.
+ manifest.abiVersion <= NativeLibrary.ABI_VERSION
+ }
+
+ /** The error for a package whose version of the interface Spark does not
support. */
+ private[ffi] def unsupportedAbiVersionError(
+ path: String,
+ manifest: NativeDataSourceManifest): Throwable = {
+ QueryExecutionErrors.invalidNativeDataSourcePackageError(
+ path,
+ "UNSUPPORTED_ABI_VERSION",
+ Map(
+ "version" -> manifest.abiVersion.toString,
+ "supported" -> NativeLibrary.ABI_VERSION.toString))
+ }
+
+ /** The error for a package without a library for the platform of this JVM.
*/
+ private[ffi] def unsupportedPlatformError(
+ path: String,
+ manifest: NativeDataSourceManifest): Throwable = {
+ QueryExecutionErrors.invalidNativeDataSourcePackageError(
+ path,
+ "UNSUPPORTED_PLATFORM",
+ Map(
+ "platform" -> NativePlatform.current,
+ "platforms" -> manifest.libraries.keys.toSeq.sorted.mkString(", ")))
+ }
+
/**
* Reads the package in the given file, or returns why it is invalid. The
result is cached until
* the file changes, and an invalid package, or a package of a version of
the interface that
@@ -141,9 +197,7 @@ object NativeDataSourcePackage extends Logging {
val version = try {
FileVersion(file.getCanonicalFile)
} catch {
- case e: IOException =>
- return Left(InvalidNativeDataSourcePackage(
- invalidManifestError(file, s"The package cannot be read: $e", e),
Nil))
+ case e: IOException => return Left(cannotRead(file, e))
}
Option(packages.getIfPresent(version)).getOrElse {
val canonical = new File(version.path)
@@ -152,7 +206,7 @@ object NativeDataSourcePackage extends Logging {
case Left(invalid) =>
logWarning(log"Invalid native data source package ${MDC(PATH,
canonical)}.",
invalid.error)
- case Right(pkg) if pkg.manifest.abiVersion > NativeLibrary.ABI_VERSION
=>
+ case Right(pkg) if !isSupported(pkg.manifest) =>
logWarning(log"The native data source package ${MDC(PATH,
canonical)} implements " +
log"version ${MDC(VERSION_NUM, pkg.manifest.abiVersion)} of the
native data source " +
log"interface, which this version of Spark does not support, so it
is not used.")
@@ -177,18 +231,61 @@ object NativeDataSourcePackage extends Logging {
packages.asMap().keySet().asScala.exists(_.path == canonicalPath)
}
- private def invalidManifestError(file: File, reason: String, cause:
Throwable = null) = {
+ /** The invalid package in a file that cannot be read, whose manifest lists
`dataSources`. */
+ private def cannotRead(
+ file: File,
+ e: IOException,
+ dataSources: Seq[String] = Nil): InvalidNativeDataSourcePackage = {
+ invalidPackage(file.getPath, s"The package cannot be read: $e",
dataSources, e)
+ }
+
+ /** An invalid package, which errors call `path`, whose manifest lists
`dataSources`. */
+ private def invalidPackage(
+ path: String,
+ reason: String,
+ dataSources: Seq[String],
+ cause: Throwable): InvalidNativeDataSourcePackage = {
+ InvalidNativeDataSourcePackage(invalidManifestError(path, reason, cause),
dataSources)
+ }
+
+ private def invalidManifestError(path: String, reason: String, cause:
Throwable = null) = {
QueryExecutionErrors.invalidNativeDataSourcePackageError(
- file.getPath, "INVALID_MANIFEST", Map("reason" -> reason), cause)
+ path, "INVALID_MANIFEST", Map("reason" -> reason), cause)
}
+ /**
+ * Reads the manifest of the package in the given file and computes its
checksum, without the
+ * cache and the log of [[tryRead]].
+ */
private def parse(file: File): ReadResult = {
+ readManifest(file, file.getPath).flatMap { manifest =>
+ try {
+ Right(NativeDataSourcePackage(file.getPath, sha256(file), manifest))
+ } catch {
+ case e: IOException => Left(cannotRead(file, e, manifest.dataSources))
+ }
+ }
+ }
+
+ /**
+ * Reads the manifest of the package in the given file, which errors call
`path`, and checks
+ * that the package contains the libraries that it lists. Only reads the
central directory of
+ * the zip file and the manifest.
+ */
+ private def readManifest(
+ file: File,
+ path: String): Either[InvalidNativeDataSourcePackage,
NativeDataSourceManifest] = {
def invalid(reason: String, dataSources: Seq[String] = Nil, cause:
Throwable = null) = {
- Left(InvalidNativeDataSourcePackage(invalidManifestError(file, reason,
cause), dataSources))
+ Left(invalidPackage(path, reason, dataSources, cause))
+ }
+ if (file.isDirectory) {
+ // The error of ZipFile would name the file, which can be a copy of the
one at `path`.
+ return invalid("The package is a directory.")
}
try {
- val manifest = Utils.tryWithResource(new ZipFile(file)) { zip =>
- Option(zip.getEntry(MANIFEST_NAME)) match {
+ Utils.tryWithResource(new ZipFile(file)) { zip =>
+ // getEntry also finds the directory entry `<name>/`, which is not a
file.
+ Option(zip.getEntry(MANIFEST_NAME)).filterNot(_.isDirectory) match {
case None => invalid(s"The package does not contain the manifest
$MANIFEST_NAME.")
case Some(entry) =>
val json = Utils.tryWithResource(zip.getInputStream(entry)) { in =>
@@ -197,7 +294,9 @@ object NativeDataSourcePackage extends Logging {
parseManifest(json) match {
case Left(reason) => invalid(reason)
case Right(manifest) =>
- manifest.libraries.values.find(zip.getEntry(_) == null) match {
+ manifest.libraries.values.find { library =>
+ Option(zip.getEntry(library)).forall(_.isDirectory)
+ } match {
case Some(library) => invalid(
s"The package does not contain the library $library.",
manifest.dataSources)
case None => Right(manifest)
@@ -205,7 +304,6 @@ object NativeDataSourcePackage extends Logging {
}
}
}
- manifest.map(NativeDataSourcePackage(file.getPath, sha256(file), _))
} catch {
case e: IOException => invalid(s"The package is not a valid zip file:
$e", cause = e)
}
@@ -399,12 +497,7 @@ object NativeLibraries extends Logging {
private def loadPackaged(pkg: NativeDataSourcePackage): Try[NativeLibrary] =
{
val platform = NativePlatform.current
val libraryPath = pkg.manifest.libraries.getOrElse(platform, {
- throw QueryExecutionErrors.invalidNativeDataSourcePackageError(
- pkg.path,
- "UNSUPPORTED_PLATFORM",
- Map(
- "platform" -> platform,
- "platforms" -> pkg.manifest.libraries.keys.toSeq.sorted.mkString(",
")))
+ throw NativeDataSourcePackage.unsupportedPlatformError(pkg.path,
pkg.manifest)
})
val packageFile = locate(pkg)
val libraryFile = new File(new File(extractionDir, pkg.sha256), new
File(libraryPath).getName)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceRegistry.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceRegistry.scala
index f58f056fb8dd..6630fb6b7534 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceRegistry.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceRegistry.scala
@@ -26,7 +26,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys.{DATA_SOURCE, PATH, PATHS}
import org.apache.spark.sql.Artifact
import org.apache.spark.sql.classic.SparkSession
-import org.apache.spark.sql.errors.{QueryCompilationErrors,
QueryExecutionErrors}
+import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
import org.apache.spark.util.Utils
@@ -62,10 +62,10 @@ object NativeDataSourceRegistry extends Logging {
val artifacts = artifactPackageFiles()
val (invalid, valid) = (configuredPackageFiles(conf) ++ artifacts)
.map(NativeDataSourcePackage.tryRead).partitionMap(identity)
- // A package of a version of the interface that Spark does not support is
unusable too.
+ // A package of a version of the interface that Spark does not support is
unusable too. Spark
+ // checks the packages that a session adds when it adds them, so these are
configured ones.
val (unsupported, usable) = valid.partitionMap { pkg =>
- // The version of a package is the oldest that its library implements.
- if (pkg.manifest.abiVersion <= NativeLibrary.ABI_VERSION) Right(pkg)
else Left(pkg)
+ if (NativeDataSourcePackage.isSupported(pkg.manifest)) Right(pkg) else
Left(pkg)
}
// The packages are read from their canonical paths, in the order that the
session added them.
val addedPaths = artifacts.map(_.getCanonicalPath).toSet
@@ -78,12 +78,7 @@ object NativeDataSourceRegistry extends Logging {
// provides the data source: a stale copy does not hide a valid one.
An invalid package
// whose manifest cannot be read is only logged, when it is read.
unsupported.find(_.manifest.dataSources.exists(_.equalsIgnoreCase(name))).foreach
{ pkg =>
- throw QueryExecutionErrors.invalidNativeDataSourcePackageError(
- pkg.path,
- "UNSUPPORTED_ABI_VERSION",
- Map(
- "version" -> pkg.manifest.abiVersion.toString,
- "supported" -> NativeLibrary.ABI_VERSION.toString))
+ throw NativeDataSourcePackage.unsupportedAbiVersionError(pkg.path,
pkg.manifest)
}
invalid.find(_.dataSources.exists(_.equalsIgnoreCase(name))).foreach(p
=> throw p.error)
None
@@ -146,8 +141,8 @@ object NativeDataSourceRegistry extends Logging {
packages: Seq[NativeDataSourcePackage]): Option[(String,
NativeDataSourcePackage)] = {
val found = packages
.flatMap(pkg =>
pkg.manifest.dataSources.find(_.equalsIgnoreCase(name)).map(_ -> pkg))
- // The same package can be found more than once, for example when it is
in a configured
- // directory and was also added as an artifact.
+ // The same package can be found more than once, for example when it is
listed in
+ // spark.sql.dataSource.native.paths both as a file and through its
directory.
.groupBy(_._2.sha256).values.map(_.head).toSeq
found match {
case Seq() => None
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/artifact/ArtifactManagerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/artifact/ArtifactManagerSuite.scala
index 0391103aa283..45d8921f56d4 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/artifact/ArtifactManagerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/artifact/ArtifactManagerSuite.scala
@@ -20,6 +20,8 @@ import java.io.File
import java.nio.charset.StandardCharsets
import java.nio.file.{Files, Path, Paths}
+import scala.collection.mutable.ArrayBuffer
+
import org.apache.spark.{SparkConf, SparkException, SparkRuntimeException}
import org.apache.spark.metrics.source.CodegenMetrics
import org.apache.spark.sql.{AnalysisException, Artifact}
@@ -431,6 +433,31 @@ class ArtifactManagerSuite extends SharedSparkSession {
}
}
+ test("Add local artifacts through the public addArtifact") {
+ // A subclass that overrides addArtifact, for example to audit what a
session adds, sees the
+ // local files that addLocalArtifacts adds, other than a native data
source package that Spark
+ // checks, which goes through a private overload, as it sees the in-memory
ones.
+ val added = ArrayBuffer.empty[Path]
+ val manager = new ArtifactManager(spark.newSession()) {
+ override def addArtifact(
+ remoteRelativePath: Path,
+ serverLocalStagingPath: Path,
+ fragment: Option[String],
+ deleteStagedFile: Boolean): Unit = {
+ added += remoteRelativePath
+ super.addArtifact(remoteRelativePath, serverLocalStagingPath,
fragment, deleteStagedFile)
+ }
+ }
+ try {
+ val file =
Files.writeString(Utils.createTempDir().toPath.resolve("audited.txt"),
"audited")
+ manager.addLocalArtifacts(
+ Artifact.newFileArtifact(Paths.get("audited.txt"), new
Artifact.LocalFile(file)) :: Nil)
+ assert(added.toSeq == Seq(Paths.get("files", "audited.txt")))
+ } finally {
+ manager.close()
+ }
+ }
+
test("Add multiple artifacts to local session and check if all are added
despite exception") {
val copyDir = Utils.createTempDir().toPath
Utils.copyDirectory(artifactPath.toFile, copyDir.toFile)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
index 86d8c0376df3..905a819aec01 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/ffi/NativeDataSourceSuite.scala
@@ -18,13 +18,15 @@ package org.apache.spark.sql.execution.datasources.v2.ffi
import java.io.File
import java.nio.charset.StandardCharsets.UTF_8
-import java.nio.file.Files
+import java.nio.file.{AccessDeniedException, Files, NoSuchFileException, Paths}
import java.nio.file.StandardCopyOption.REPLACE_EXISTING
+import java.nio.file.attribute.{PosixFileAttributeView, PosixFilePermission}
import java.sql.{Date, Timestamp}
import java.time.{Instant, LocalDate}
import java.util.UUID
import java.util.concurrent.{Callable, CountDownLatch, ExecutionException,
TimeUnit}
import java.util.concurrent.atomic.AtomicReference
+import java.util.zip.{ZipException, ZipFile}
import scala.collection.mutable.ArrayBuffer
import scala.jdk.CollectionConverters._
@@ -1604,8 +1606,12 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
parameters = Map(
"name" -> "native_range",
"packages" -> Seq(first, second).map(_.getCanonicalPath).mkString(",
")))
- // The same package found twice is not a conflict: here, in the configured
paths and in the
- // artifacts of the session.
+ // The same package found twice is not a conflict: here, listed both as a
file and through
+ // its directory.
+ val only = Utils.createTempDir()
+ val pkg = createPackage(library, dataSources(""), dir = only, fileName =
"c.sparkpkg")
+ checkRows(range(newSession(pkg, only), "end" -> "2"), Seq(expectedRow(0L),
expectedRow(1L)))
+ // Nor is the same package, configured and also added by the session.
val session = newSession(first)
session.addArtifact(first.getPath)
checkRows(range(session, "end" -> "2"), Seq(expectedRow(0L),
expectedRow(1L)))
@@ -1637,6 +1643,184 @@ class NativeDataSourceSuite extends QueryTest with
SharedSparkSession {
assert(logged.forall(_.contains("for the data source native_range")),
logged)
}
+ nativeTest("reject a package that a session adds when Spark cannot use it") {
+ // The package that a session added last takes precedence, so Spark
rejects a package that it
+ // cannot use when the session adds it, rather than keep using what the
session used before.
+ val session = newSession(defaultPackage)
+ def ids(): DataFrame = range(session, "end" -> "3").select("id")
+ val dir = Utils.createTempDir()
+ def check(pkg: File, subClass: String, params: Map[String, String]): Unit
= {
+ // Also when the same file is added again.
+ (1 to 2).foreach { _ =>
+ checkError(
+ exception =
intercept[SparkRuntimeException](session.addArtifact(pkg.getPath)),
+ condition = s"INVALID_NATIVE_DATA_SOURCE_PACKAGE.$subClass",
+ parameters = Map("path" -> pkg.getPath) ++ params)
+ }
+ // The session does not get it, and keeps using the configured package.
+ assert(session.artifactManager.getNativeDataSourcePackages._1.isEmpty)
+ assert(!session.artifactManager.hasFileArtifact(pkg.getName))
+ assert(!session.sparkContext.listFiles().exists(_.endsWith("/" +
pkg.getName)))
+ checkRows(ids(), Seq(Row(0L), Row(1L), Row(2L)))
+ }
+ check(createPackage(renamedNegatingLibrary, dataSources(""), dir = dir,
+ fileName = "range-2.0.sparkpkg", abiVersion = 2),
+ "UNSUPPORTED_ABI_VERSION", Map("version" -> "2", "supported" -> "1"))
+ check(createZip(dir, "range-2.1.sparkpkg",
+ NativeDataSourcePackage.MANIFEST_NAME -> manifestJson(
+ dataSources(""), Map(NativePlatform.current ->
"lib.so")).getBytes(UTF_8)),
+ "INVALID_MANIFEST", Map("reason" -> "The package does not contain the
library lib.so."))
+ // ZipFile.getEntry("lib") also finds the directory entry "lib/", as for
the manifest below.
+ check(createZip(dir, "range-2.7.sparkpkg",
+ NativeDataSourcePackage.MANIFEST_NAME -> manifestJson(
+ dataSources(""), Map(NativePlatform.current ->
"lib")).getBytes(UTF_8),
+ "lib/" -> Array.emptyByteArray, "lib/x" -> "x".getBytes(UTF_8)),
+ "INVALID_MANIFEST", Map("reason" -> "The package does not contain the
library lib."))
+ check(createZip(dir, "range-2.8.sparkpkg",
+ s"${NativeDataSourcePackage.MANIFEST_NAME}/" -> Array.emptyByteArray),
+ "INVALID_MANIFEST", Map("reason" ->
+ s"The package does not contain the manifest
${NativeDataSourcePackage.MANIFEST_NAME}."))
+ val notAZip = new File(dir, "range-2.2.sparkpkg")
+ Files.writeString(notAZip.toPath, "not a zip")
+ // The message of the ZipException names no file.
+ val notAZipReason =
+ s"The package is not a valid zip file: ${intercept[ZipException](new
ZipFile(notAZip))}"
+ check(notAZip, "INVALID_MANIFEST", Map("reason" -> notAZipReason))
+ val directory = new File(dir, "range-2.3.sparkpkg")
+ assert(directory.mkdir())
+ check(directory, "INVALID_MANIFEST", Map("reason" -> "The package is a
directory."))
+ // The driver loads the library to plan a query.
+ check(createPackage(library, dataSources(""), dir = dir, fileName =
"range-2.4.sparkpkg",
+ platform = "other-platform"),
+ "UNSUPPORTED_PLATFORM", Map("platform" -> NativePlatform.current,
+ "platforms" -> "other-platform"))
+ // A file that does not exist, or that cannot be read, fails as for the
other artifacts.
+ intercept[NoSuchFileException](session.addArtifact(new File(dir,
"missing.sparkpkg").getPath))
+ val unreadable = createPackage(library, dataSources(""), dir = dir,
+ fileName = "range-2.5.sparkpkg")
+ try {
+ // Not when the user can read any file, as root.
+ if (unreadable.setReadable(false) &&
!Files.isReadable(unreadable.toPath)) {
+
intercept[AccessDeniedException](session.addArtifact(unreadable.getPath))
+ }
+ } finally {
+ unreadable.setReadable(true)
+ }
+ // A package that Spark staged, such as one added from bytes, is named
after the artifact.
+ val staged = intercept[SparkRuntimeException] {
+ session.addArtifact(Files.readAllBytes(createPackage(library,
dataSources(""),
+ abiVersion = 2).toPath), "range-2.6.sparkpkg")
+ }
+ checkError(
+ exception = staged,
+ condition = "INVALID_NATIVE_DATA_SOURCE_PACKAGE.UNSUPPORTED_ABI_VERSION",
+ parameters = Map(
+ "path" -> Paths.get("files", "range-2.6.sparkpkg").toString,
+ "version" -> "2",
+ "supported" -> "1"))
+ assert(session.artifactManager.getNativeDataSourcePackages._1.isEmpty)
+
+ // A valid package with the same file name as a rejected one can be added
then.
+ val added = createPackage(renamedNegatingLibrary, dataSources(""),
+ fileName = "range-2.0.sparkpkg")
+ session.addArtifact(added.getPath)
+ checkRows(ids(), Seq(Row(0L), Row(-1L), Row(-2L)))
+ // Adding it again, from the file or from the same bytes, does nothing.
+ session.addArtifact(added.getPath)
+ session.addArtifact(Files.readAllBytes(added.toPath), added.getName)
+
assert(session.artifactManager.getNativeDataSourcePackages._1.map(_.getName) ==
+ Seq("range-2.0.sparkpkg"))
+ checkRows(ids(), Seq(Row(0L), Row(-1L), Row(-2L)))
+ // Another file with the name of an artifact fails as such, also if it is
not a package, and
+ // the error names the artifact.
+ def checkExisting(e: SparkRuntimeException, fileName: String =
"range-2.0.sparkpkg"): Unit = {
+ checkError(
+ exception = e,
+ condition = "ARTIFACT_ALREADY_EXISTS",
+ parameters = Map("normalizedRemoteRelativePath" -> Paths.get("files",
fileName).toString))
+ }
+ val sameName = new File(Utils.createTempDir(), "range-2.0.sparkpkg")
+ Files.copy(notAZip.toPath, sameName.toPath)
+
checkExisting(intercept[SparkRuntimeException](session.addArtifact(sameName.getPath)))
+ val sameNameDirectory = new File(Utils.createTempDir(),
"range-2.0.sparkpkg")
+ assert(sameNameDirectory.mkdir())
+
checkExisting(intercept[SparkRuntimeException](session.addArtifact(sameNameDirectory.getPath)))
+ val missing = new File(Utils.createTempDir(), "range-2.0.sparkpkg")
+
checkExisting(intercept[SparkRuntimeException](session.addArtifact(missing.getPath)))
+ // Also for a package that Spark staged, which is compared with the
artifact before the check.
+ checkExisting(intercept[SparkRuntimeException] {
+ session.addArtifact("not a zip".getBytes(UTF_8), "range-2.0.sparkpkg")
+ })
+ // And for an artifact that is a directory, as files/range-5.0.sparkpkg is
after this.
+ val inner = new File(Utils.createTempDir(), "inner.txt")
+ Files.writeString(inner.toPath, "inner")
+ session.artifactManager.addArtifact(Paths.get("files",
"range-5.0.sparkpkg", inner.getName),
+ inner.toPath, None, deleteStagedFile = false)
+ checkExisting(intercept[SparkRuntimeException] {
+ session.addArtifact(createPackage(library, dataSources(""),
+ fileName = "range-5.0.sparkpkg").getPath)
+ }, "range-5.0.sparkpkg")
+
+ // A call that adds several artifacts adds the others, and reports all the
rejected ones.
+ val valid = createPackage(library, dataSources(""), dir = dir, fileName =
"range-3.0.sparkpkg")
+ val rejected = intercept[SparkRuntimeException] {
+ session.addArtifacts(sameName.toURI, notAZip.toURI, valid.toURI)
+ }
+ checkExisting(rejected)
+ assert(rejected.getSuppressed.length == 1)
+ checkError(
+ exception =
rejected.getSuppressed.head.asInstanceOf[SparkRuntimeException],
+ condition = "INVALID_NATIVE_DATA_SOURCE_PACKAGE.INVALID_MANIFEST",
+ parameters = Map("path" -> notAZip.getPath, "reason" -> notAZipReason))
+ checkRows(ids(), Seq(Row(0L), Row(1L), Row(2L)))
+
+ // Only the file artifacts (files/) with the extension are packages, which
Spark checks, and
+ // not other kinds of artifacts with it, which a Spark Connect client can
add.
+ val pyFile = createPackage(library, dataSources(""), dir = dir,
+ fileName = "range-4.0.sparkpkg", abiVersion = 2)
+ session.artifactManager.addArtifact(
+ Paths.get("pyfiles", pyFile.getName), pyFile.toPath, None,
deleteStagedFile = false)
+
assert(session.artifactManager.getNativeDataSourcePackages._1.map(_.getName) ==
+ Seq("range-2.0.sparkpkg", "range-3.0.sparkpkg"))
+ }
+
+ test("add any file with the extension when native data sources are
disabled") {
+ val session = spark.newSession()
+ sessions += session
+ // A static configuration, which a session cannot set otherwise.
+
session.sessionState.conf.setConf(StaticSQLConf.NATIVE_DATA_SOURCE_ENABLED,
false)
+ val file = new File(Utils.createTempDir(), "not_a_package.sparkpkg")
+ Files.writeString(file.toPath, "not a zip")
+ session.addArtifact(file.getPath)
+ assert(session.artifactManager.hasFileArtifact(file.getName))
+ }
+
+ test("add a read-only package as a writable artifact") {
+ // Spark checks a copy of the file of the user, which keeps its
permissions, or on Windows its
+ // read-only attribute, so Spark makes it writable: the artifact, moved
from it, is writable.
+ val session = spark.newSession()
+ sessions += session
+ val pkg = createZip(Utils.createTempDir(), "read_only.sparkpkg",
+ NativeDataSourcePackage.MANIFEST_NAME -> manifestJson(
+ dataSources(""), Map(NativePlatform.current ->
"lib.so")).getBytes(UTF_8),
+ "lib.so" -> "lib".getBytes(UTF_8))
+ assert(pkg.setWritable(false))
+ try {
+ session.addArtifact(pkg.getPath)
+ val artifact =
session.artifactManager.getNativeDataSourcePackages._1.head.toPath
+ // The permissions rather than Files.isWritable, which is true for root
whatever they are.
+ Option(Files.getFileAttributeView(artifact,
classOf[PosixFileAttributeView])) match {
+ case Some(view) =>
+
assert(view.readAttributes().permissions().contains(PosixFilePermission.OWNER_WRITE))
+ case None =>
+ // On Windows, Files.isWritable reads the read-only attribute.
+ assert(Files.isWritable(artifact))
+ }
+ } finally {
+ pkg.setWritable(true)
+ }
+ }
+
nativeTest("Java data sources take precedence") {
val session = newSession(createPackage(library, Seq("json")))
val dir = Utils.createTempDir().getPath + "/json"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]