org.apache.spark.sql.AnalysisException: Detected implicit cartesian product

[kyunam]

"left join" complains and tells me I need to turn on
"spark.sql.crossJoin.enabled=true".

But when I persist one dataframe, it runs fine.

Why do you have to "persist"?

org.apache.spark.sql.AnalysisException: Detected implicit cartesian product
for INNER join between logical plans

SELECT * FROM LHS left join RHS on LHS.R = RHS.R

The above happens in both Spark 2.3.3 and 2.4.4.

Thanks,
Kyunam



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

How to query StructField's metadata in spark sql?

[kyunam]

Using SQL, is it possible to query a column's metadata?

Thanks,
Kyunam



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Possible to specify returnType: DataType in UDFRegistration.register()?

[kyunam]

Scala API
<https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.UDFRegistration@register(name:String,f:org.apache.spark.sql.api.java.UDF2[_,_,_],returnType:org.apache.spark.sql.types.DataType):Unit>
  
says that register(..) accepts returnType.But when I do and compile, I get
the following error.error: overloaded method value register with
alternatives:[ERROR]   (name: String,f:
org.apache.spark.sql.api.java.UDF22[_, _, _, _, _, _, _, _, _, _, _, _, _,
_, _, _, _, _, _, _, _, _, _],returnType:
org.apache.spark.sql.types.DataType)Unit [ERROR]   (name: String,f:
org.apache.spark.sql.api.java.UDF21[_, _, _, _, _, _, _, _, _, _, _, _, _,
_, _, _, _, _, _, _, _, _],returnType:
org.apache.spark.sql.types.DataType)Unit Why can't I pass in
returnType?Thanks,Kyunam



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

how to call database specific function when reading writing thru jdbc

[Kyunam Kim]

For example, in SQL Server, when reading, I want to call a built-in function: 
STAsText()

SELECT id, shape.STAsText() FROM SpatialTable

val df = _sparkSession
.read
.jdbc(url, "dbo.SpatialTable", props)
.select("shape.STAsText()")  // No, this doesn't work.
.as("shape")

Also when writing, I want to be able to call a built-in function: 
STGeomFromText like this:
INSERT INTO SpatialTable (GeomCol1) VALUES (geometry::STGeomFromText('POLYGON 
((0 0, 150 0, 150 150, 0 150, 0 0))', 0));

How would go about doing this?

Thanks,
Kyunam