Re: Spark SQL in R?

2019-06-08 Thread Felix Cheung
I don’t think you should get a hive-xml from the internet. It should have connection information about a running hive metastore - if you don’t have a hive metastore service as you are running locally (from a laptop?) then you don’t really need it. You can get spark to work with it’s own.

[ANNOUNCE] Apache Bahir 2.3.3 Released

2019-06-08 Thread Luciano Resende
Apache Bahir provides extensions to multiple distributed analytic platforms, extending their reach with a diversity of streaming connectors and SQL data sources. The Apache Bahir community is pleased to announce the release of Apache Bahir 2.3.3 which provides the following extensions for Apache Sp

[ANNOUNCE] Apache Bahir 2.2.3 Released

2019-06-08 Thread Luciano Resende
Apache Bahir provides extensions to multiple distributed analytic platforms, extending their reach with a diversity of streaming connectors and SQL data sources. The Apache Bahir community is pleased to announce the release of Apache Bahir 2.2.3 which provides the following extensions for Apache Sp

Re: Spark 2.2 With Column usage

2019-06-08 Thread anbutech
Thanks Jacek Laskowski Sir.but i didn't get the point here please advise the below one are you expecting: dataset1.as("t1) join(dataset3.as("t2"), col(t1.col1) === col(t2.col1), JOINTYPE.Inner ) .join(dataset4.as("t3"), col(t3.col1) === col(t1.col1), JOINTYPE.Inner) .select("id",lit(refe

Re: Spark logging questions

2019-06-08 Thread Jacek Laskowski
Hi, What are "the spark driver and executor threads information" and "spark application logging"? Spark uses log4j so set up logging levels appropriately and you should be done. Pozdrawiam, Jacek Laskowski https://about.me/JacekLaskowski The Internals of Spark SQL https://bit.ly/spark-sql-i

Re: Spark 2.2 With Column usage

2019-06-08 Thread Jacek Laskowski
Hi, > val referenceFiltered = dataset2.filter(.dataDate == date).filter.someColumn).select("id").toString > .withColumn("new_column",lit(referenceFiltered)) That won't work since lit is a function (adapter) to convert Scala values to Catalyst expressions. Unless I'm mistaken, in your case, what