vshinde-medacist opened a new issue, #4527:
URL: https://github.com/apache/iceberg/issues/4527
We are doing POC on Iceberg and evaluating it first time.
Spark Environment:
- Spark: spark-3.1.2-bin-hadoop3.2
- Scala: 2.12.10
- Java: 1.8.0_321
- Hadoop: 3.2.0
- Iceberg 0.13.1
As suggested in Iceberg's official documentation, to add support for Iceberg
in Spark shell, we are adding Iceberg dependency while launching the Spark
shell as below,
```cmd
spark-shell --packages
org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:0.13.1
````
After launching the Spark shell with the above command, we are not able to
use the Spark shell at all. For all the commands (even non Iceberg) we are
getting the same exception as below,
```cmd
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/spark/sql/catalyst/plans/logical/BinaryCommand
```
Below simple command also throwing same exception.
```scala
val df : DataFrame =
spark.read.json("/spark-3.1.2-bin-hadoop3.2/examples/src/main/resources/people.json")
df.show()
```
In Spark source code, `BinaryCommand` class belongs to `Spark SQL` module,
so tried explicitly adding Spark SQL dependency while launching Spark shell as
below, but still getting same exception.
```cmd
spark-shell --packages
org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:0.13.1,org.apache.spark:spark-sql_2.12:3.1.2
```
**When we launch `spark-shell` normally i.e. without Iceberg dependency,
then it is working properly.**
Any pointer in the right direction for troubleshooting would be really
helpful.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
