I didn't post the SO issue, I've just found the same exception I'm facing for Spark 3.2. Almaren Framework has a concept of create temporary views with the name "__TABLE__".
Example, if you want to use SQL dialect to a DataFrame to join a table/aggregation/apply a function whatever. Instead of you create a temporary table you just use the "__TABLE__" alias. You don't really care about the name of the table. You may use this "__TABLE__" approach in different parts of your code. Why can't I create or replace temporary views in different DataFrame with the same name as before ? On Mon, Dec 13, 2021 at 1:27 PM Sean Owen <sro...@gmail.com> wrote: > If the issue is what you posted in SO, I think the stack trace explains it > already. You want to avoid this recursive definition, which in general > can't work. > I think it's simply explicitly disallowed in all cases now, but, you > should not be depending on this anyway - why can't this just be avoided? > > On Mon, Dec 13, 2021 at 10:06 AM Daniel de Oliveira Mantovani < > daniel.oliveira.mantov...@gmail.com> wrote: > >> Sean, >> >> https://github.com/music-of-the-ainur/almaren-framework/tree/spark-3.2 >> >> Just executing "sbt test" will reproduce the error, the same code works >> for spark 2.3.x, 2.4.x and 3.1.x why doesn't it work for spark 3.2 ? >> >> Thank you so much >> >> >> >> On Mon, Dec 13, 2021 at 12:59 PM Sean Owen <sro...@gmail.com> wrote: >> >>> ... but the error is not "because that already exists". See your stack >>> trace. It's because the definition is recursive. You define temp view >>> test1, create a second DF from it, and then redefine test1 as that result. >>> test1 depends on test1. >>> >>> On Mon, Dec 13, 2021 at 9:58 AM Daniel de Oliveira Mantovani < >>> daniel.oliveira.mantov...@gmail.com> wrote: >>> >>>> Sean, >>>> >>>> The method name is very clear "createOrReplaceTempView" doesn't make >>>> any sense to throw an exception because this view already exists. Spark >>>> 3.2.x is breaking back compatibility with no reason or sense. >>>> >>>> >>>> On Mon, Dec 13, 2021 at 12:53 PM Sean Owen <sro...@gmail.com> wrote: >>>> >>>>> The error looks 'valid' - you define a temp view in terms of its own >>>>> previous version, which doesn't quite make sense - somewhere the new >>>>> definition depends on the old definition. I think it just correctly >>>>> surfaces as an error now,. >>>>> >>>>> On Mon, Dec 13, 2021 at 9:41 AM Daniel de Oliveira Mantovani < >>>>> daniel.oliveira.mantov...@gmail.com> wrote: >>>>> >>>>>> Hello team, >>>>>> >>>>>> I've found this issue while I was porting my project from Apache >>>>>> Spark 3.1.x to 3.2.x. >>>>>> >>>>>> >>>>>> https://stackoverflow.com/questions/69937415/spark-3-2-0-the-different-dataframe-createorreplacetempview-the-same-name-tempvi >>>>>> >>>>>> Do we have a bug for that in apache-spark or I need to create one ? >>>>>> >>>>>> Thank you so much >>>>>> >>>>>> [info] com.github.music.of.the.ainur.almaren.Test *** ABORTED *** >>>>>> [info] org.apache.spark.sql.AnalysisException: Recursive view >>>>>> `__TABLE__` detected (cycle: `__TABLE__` -> `__TABLE__`) >>>>>> [info] at >>>>>> org.apache.spark.sql.errors.QueryCompilationErrors$.recursiveViewDetectedError(QueryCompilationErrors.scala:2045) >>>>>> [info] at >>>>>> org.apache.spark.sql.execution.command.ViewHelper$.checkCyclicViewReference(views.scala:515) >>>>>> [info] at >>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2(views.scala:522) >>>>>> [info] at >>>>>> org.apache.spark.sql.execution.command.ViewHelper$.$anonfun$checkCyclicViewReference$2$adapted(views.scala:522) >>>>>> [info] at scala.collection.Iterator.foreach(Iterator.scala:941) >>>>>> [info] at scala.collection.Iterator.foreach$(Iterator.scala:941) >>>>>> [info] at >>>>>> scala.collection.AbstractIterator.foreach(Iterator.scala:1429) >>>>>> [info] at >>>>>> scala.collection.IterableLike.foreach(IterableLike.scala:74) >>>>>> [info] at >>>>>> scala.collection.IterableLike.foreach$(IterableLike.scala:73) >>>>>> [info] at >>>>>> scala.collection.AbstractIterable.foreach(Iterable.scala:56) >>>>>> >>>>>> -- >>>>>> >>>>>> -- >>>>>> Daniel Mantovani >>>>>> >>>>>> >>>> >>>> -- >>>> >>>> -- >>>> Daniel Mantovani >>>> >>>> >> >> -- >> >> -- >> Daniel Mantovani >> >> -- -- Daniel Mantovani
