What's the expected Spark 3.1.4 release date ?

Hi Team, wondering what's the expected release date for spark 3.1.4?

pyspark.ml Pipeline stages are corrupted under multi-threaded access - is this a bug?

ages() in each object matches that supplied during pipeline object construction. Observe that there are occasions when this does not hold true. This occurs with both Spark 1.6 and 2.x Regards, Vinayak Joshi

Re: Spark 2.x Pyspark Spark SQL createDataframe Error

Thanks Michal. I have submitted a Spark issue and PR based on my understanding of why this changed in Spark 2.0. If interested you can follow it on https://issues.apache.org/jira/browse/SPARK-18687 Regards, Vinayak. From: Michal Šenkýř <bina...@gmail.com> To: Vinayak Joshi5/Ind

Re: Spark 2.x Pyspark Spark SQL createDataframe Error

) at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 111 more Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /Users/vinayak/devel/spark-stc/git_repo/spark-master-x/spark/metastore_db

Spark 2.x Pyspark Spark SQL createDataframe Error

The error goes away if sqlContext2 is replaced with sqlContext in the error line. Is this a regression, or has something changed that makes this the expected behavior in Spark 2.x ? Regards, Vinayak

Re: spark-submit hive connection through spark Initial job has not accepted any resources

Hi Herman, This error comes when you have started your master but no worker has been added to your cluster, please check through spark master UI is there any worker added in master? Also check in your driver code have you set configuration.setmaster(local[]) if added remove it and give spark

Re: My notes on Spark Performance & Tuning Guide

Please include me too. Vinayak Agrawal Big Data Analytics IBM "To Strive, To Seek, To Find and Not to Yield!" ~Lord Alfred Tennyson > On May 17, 2016, at 2:15 PM, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Hi all, > > Many thanks for your trem

Do I need to install Cassandra node on Spark Master node to work with Cassandra?

need to install cassandra node on my Spark Master node so that Spark can connect with cassandra or Cassandra only needs to be on Spark worker nodes? It seemss logical considering data locality. Thanks -- Vinayak Agrawal "To Strive, To Seek, To Find and Not to Yield!" ~Lord Alfred Tennyson

Saving a pipeline model ?

are the spark users currently working around this? Is there a way to convert a pipelinemodel to mllib model and save ? Thanks - Vinayak Agrawal "To Strive, To Seek, To Find and Not to Yield!" ~Lord Alfred Tennyson

Getting Co-oefficients of a logistic regression model for a pipelinemodel Spark ML library

+model+statistics%22=newest=1 Any suggestions? -- Vinayak Agrawal "To Strive, To Seek, To Find and Not to Yield!" ~Lord Alfred Tennyson

Re: can we create dummy variables from categorical variables, using sparkR

for categorical variables in sparkR like we > do using "dummies" package in R > > -- > Warm regards, > Devesh. > -- Vinayak Agrawal Big Data Analytics IBM "To Strive, To Seek, To Find and Not to Yield!" ~Lord Alfred Tennyson

Re: spark-submit hive connection through spark Initial job has not accepted any resources

Java code which I am trying to invoke. import org.apache.spark.SparkContext; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.sql.DataFrame; import org.apache.spark.sql.hive.HiveContext; public class SparkHiveInsertor { public static void main(String[]

spark-submit hive connection through spark Initial job has not accepted any resources

Hi, I am able to fetch data, create table, put data from spark shell (scala command line) from spark to hive but when I create java code to do same and submitting it through spark-submit i am getting *"Initial job has not accepted any resources; check your cluster UI to ensure that workers are

spark sql through java code facing issue

Hi All, *I am new to spark and trying to execute spark sql through java code as below* package com.ce.sql; import java.util.List; import org.apache.spark.SparkConf; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.api.java.JavaSparkContext; import