dongjoon-hyun commented on PR #46026:
URL: https://github.com/apache/spark/pull/46026#issuecomment-2054155242
Thank you for updating.
BTW, I have only Apple Silicon Mac and I realized that the pre-built Apache
Hadoop 3.4.0 seems not to support Apple Silicon. I'm blocked by this still. The
following is the result when I build your PR branch.
```
$ ./bin/spark-sql --conf
spark.driver.extraLibraryPath=/Users/dongjoon/APACHE/hadoop-releases/hadoop-3.4.0/lib/native
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
{"ts":"2024-04-14T11:59:40.118-0700","level":"WARN","msg":"Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable","logger":"org.apache.hadoop.util.NativeCodeLoader"}
{"ts":"2024-04-14T11:59:41.809-0700","level":"WARN","msg":"Version
information not found in metastore. hive.metastore.schema.verification is not
enabled so recording the schema version
2.3.0","logger":"org.apache.hadoop.hive.metastore.ObjectStore"}
{"ts":"2024-04-14T11:59:41.809-0700","level":"WARN","msg":"setMetaStoreSchemaVersion
called but recording version is disabled: version = 2.3.0, comment = Set by
MetaStore
[email protected]","logger":"org.apache.hadoop.hive.metastore.ObjectStore"}
{"ts":"2024-04-14T11:59:41.814-0700","level":"WARN","msg":"Failed to get
database default, returning
NoSuchObjectException","logger":"org.apache.hadoop.hive.metastore.ObjectStore"}
Spark Web UI available at http://localhost:4040
Spark master: local[*], Application Id: local-1713121180569
spark-sql (default)> CREATE TABLE tmp_compressed_datasource (key STRING)
USING TEXT OPTIONS (compression 'zstd');
{"ts":"2024-04-14T11:59:47.251-0700","level":"WARN","msg":"Couldn't find
corresponding Hive SerDe for data source provider TEXT. Persisting data source
table `spark_catalog`.`default`.`tmp_compressed_datasource` into Hive metastore
in Spark SQL specific format, which is NOT compatible with
Hive.","logger":"org.apache.spark.sql.hive.HiveExternalCatalog"}
{"ts":"2024-04-14T11:59:47.293-0700","level":"WARN","msg":"METASTORE_FILTER_HOOK
will be ignored, since hive.security.authorization.manager is set to instance
of
HiveAuthorizerFactory.","logger":"org.apache.hadoop.hive.ql.session.SessionState"}
{"ts":"2024-04-14T11:59:47.348-0700","level":"WARN","msg":"HiveConf of name
hive.internal.ss.authz.settings.applied.marker does not
exist","logger":"org.apache.hadoop.hive.conf.HiveConf"}
Time taken: 1.165 seconds
spark-sql (default)> INSERT INTO TABLE tmp_compressed_datasource values
('a'),('b'),('c');
{"ts":"2024-04-14T11:59:52.348-0700","level":"ERROR","msg":"Exception in
task 2.0 in stage 0.0 (TID 2)","context":{"mdc.taskName":"task 2.0 in stage 0.0
(TID 2)","task_name":"task 2.0 in stage 0.0 (TID 2)"},"exception":
{"class":"java.lang.RuntimeException",
"msg":"native zStandard library not available: this version of libhadoop was
built without zstd support.","stacktrace":"java.lang.RuntimeException: native
zStandard library not available: this version of libhadoop was built without
zstd support.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
