Re: Spark-sql can replace Hive ?

OK you mean use spark.sql as opposed to HiveContext.sql? val HiveContext = new org.apache.spark.sql.hive.HiveContext(sc) HiveContext.sql("") replace with spark.sql("") ? view my Linkedin profile *Disclaimer:* Use it at

Re: Spark-sql can replace Hive ?

Currently I am using hive sql engine for adhoc queries. As spark-sql also supports this, I want migrate from hive. From: Mich Talebzadeh Date: Thursday, 10 June 2021 at 8:12 PM To: Battula, Brahma Reddy Cc: ayan guha , d...@spark.apache.org , user@spark.apache.org Subject: Re: Spark-sql

Re: Missing stack function from SQL functions API

Hi David If you need alternative way to do it you can use below: df.select(expr("stack(2, 1,2,3)")) Or df.withColumn('stacked', expr("stack(2, 1,2,3)")) Thanks Khalid On Mon, 14 Jun 2021, 10:14 , wrote: > I noticed that the stack SQL function >

What happens if a random forest max bins is set too high?

What happens if a random forest "max bins" hyperparameter is set too high? When training a sparkml random forest ( https://spark.apache.org/docs/latest/ml-classification-regression.html#random-forest-classifier ) with maxBins set roughly equal to the max number of distinct categorical values for

Re: Apply window function on data consumed from Kafka topic

Hi, If you want help I'd suggest copying the full code, you just shared the config part. On the other hand, if you are doing a project *now* I'd also suggest using Structured Streaming , I'm sure you would get

Does Rollups work with spark structured streaming with state.

Hi Spark-Users, Hope you are all doing well. Recently I was looking into rollup operations in spark. As we know state based aggregation is supported in spark structured streaming. I was wondering if rollup operations are also supported? Like the state of previous aggregation on the rollups are

RE: Apply window function on data consumed from Kafka topic

Hi All, Can some one help me how to resolve this?. It is very important to achieve my project objective Regards, Favas From: Muhammed Favas Sent: Thursday, June 10, 2021 15:26 PM To: user@spark.apache.org Subject: Apply window function on data consumed from Kafka topic Hi, I