Re: SparkSQL: StringType for numeric comparison

: > Hi, > > I am using SparkSQL 1.1.0. > > Actually, I have a table as following: > > root > |-- account_id: string (nullable = false) > |-- Birthday: string (nullable = true) > |-- preferstore: string (nullable = true) > |-- registstore: string (nullable = true

SparkSQL: select syntax

Hi all, A quick question on SparkSql *SELECT* syntax. Does it support queries like: *SELECT t1.*, t2.d, t2.e FROM t1 LEFT JOIN t2 on t1.a = t2.a* It always ends with the exception: *Exception in thread "main" java.lang.RuntimeException: [2.12] failure: string literal expected SELEC

SparkSQL: StringType for numeric comparison

Hi, I am using SparkSQL 1.1.0. Actually, I have a table as following: root |-- account_id: string (nullable = false) |-- Birthday: string (nullable = true) |-- preferstore: string (nullable = true) |-- registstore: string (nullable = true) |-- gender: string (nullable = true

Re: Nested Query using SparkSQL 1.1.0

Hi Shahab, Do you mean queries with group by and aggregation functions? Once you register the json dataset as a table, you can write queries like querying a regular table. You can join it with other tables and do aggregations. Is it what you were asking for? If not, can you give me a more concrete

Re: Nested Query using SparkSQL 1.1.0

Thanks Yin. I trued HiveQL and and it solved that problem. But now I have second query requirement : But since you are main developer behind JSON-Spark integration (I saw your presentation on youtube "Easy JSON Data Manipulation in Spark"), is it possible to perform aggregation kind queries, for e

Re: Nested Query using SparkSQL 1.1.0

Hi Shahab, Can you try to use HiveContext? Its should work in 1.1. For SQLContext, this issues was not fixed in 1.1 and you need to use master branch at the moment. Thanks, Yin On Sun, Oct 12, 2014 at 5:20 PM, shahab wrote: > Hi, > > Apparently is it is possible to query nested json using sp

Re: SparkSQL LEFT JOIN problem

#65279(or U+FEFF). As a result, the first field has a leading #65279 char. When querying, I just used account_id, so SparkSQL cannot find the given field in AST, while the one in AST is #65279account_id. So the solution this to convert input file to UTF-8 Unicode (*without* BOM), that will remove

Re: Setting SparkSQL configuration

Currently Spark SQL doesn’t support reading SQL specific configurations via system properties. But for |HiveContext|, you can put them in |hive-site.xml|. On 10/13/14 4:28 PM, Kevin Paul wrote: Hi all, I tried to set the configuration spark.sql.inMemoryColumnarStorage.compressed, and spark.s

Setting SparkSQL configuration

Hi all, I tried to set the configuration spark.sql.inMemoryColumnarStorage.compressed, and spark.sql.inMemoryColumnarStorage.batchSize in spark.executor.extraJavaOptions but it does not work, my spark.executor.extraJavaOptions contains "Dspark.sql.inMemoryColumnarStorage.compressed=true -Dspark.sql

Re: SparkSQL on Hive error

Thanks Michael, your patch works for me :) Regards, Kelvin Paul On Fri, Oct 3, 2014 at 3:52 PM, Michael Armbrust wrote: > Are you running master? There was briefly a regression here that is > hopefully fixed by spark#2635 . > > On Fri, Oct 3, 2014 at 1

Nested Query using SparkSQL 1.1.0

Hi, Apparently is it is possible to query nested json using spark SQL, but , mainly due to lack of proper documentation/examples, I did not manage to make it working. I do appreciate if you could point me to any example or help with this issue, Here is my code: val anotherPeopleRDD = sc.paral

Re: Blog post: An Absolutely Unofficial Way to Connect Tableau to SparkSQL (Spark 1.1)

Very cool Denny, thanks for sharing this! Matei On Oct 11, 2014, at 9:46 AM, Denny Lee wrote: > https://www.concur.com/blog/en-us/connect-tableau-to-sparksql > > If you're wondering how to connect Tableau to SparkSQL - here are the steps > to connect Tableau to SparkSQL. > > > > Enjoy! >

Re: How to do broadcast join in SparkSQL

It works fine, thanks for the help Michael. Liancheng also told me a trick, using a subquery with LIMIT n. It works in latest 1.2.0 BTW, looks like the broadcast optimization won't be recognized if I do a left join instead of a inner join. Is that true? How can I make it work for left joins? Che

Re: SparkSQL LEFT JOIN problem

Hi Can you try select birthday from customer left join profile on customer.account_id = profile.account_id to see if the problems remains on your entire data? Thanks, Liquan On Fri, Oct 10, 2014 at 8:20 AM, invkrh wrote: > Hi, > > I am exploring SparkSQL 1.1.0, I have a problem on

SparkSQL LEFT JOIN problem

Hi, I am exploring SparkSQL 1.1.0, I have a problem on LEFT JOIN. Here is the request: select * from customer left join profile on customer.account_id = profile.account_id The two tables' schema are shown as following: // Table: customer root |-- account_id: string (nullable =

How to patch sparkSQL on EC2?

Hi I have written a few extensions for sparkSQL (for version 1.1.0) and I am trying to deploy my new jar files (one for catalyst and one for sql/core) on ec2. My approach was to create a new spark/lib/spark-assembly-1.1.0-hadoop1.0.4.jar that merged the contents of the old one with the

Re: sparksql connect remote hive cluster

k_1073763904_23146 > file=/user/tja01/datasets/00ab46fa4d6711e4afb70003ff41ebbf/part-3 > > not sure if some of the ports are not open or it needs access to additional > things. > > thanks, > > > > > -- > View this message in context: > http://apache-spark-use

Re: How to do broadcast join in SparkSQL

Thanks for the input. We purposefully made sure that the config option did not make it into a release as it is not something that we are willing to support long term. That said we'll try and make this easier in the future either through hints or better support for statistics. In this particular

sparksql connect remote hive cluster

. thanks, -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/sparksql-connect-remote-hive-cluster-tp15928.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To

Re: How to do broadcast join in SparkSQL

Ok, currently there's cost-based optimization however Parquet statistics is not implemented... What's the good way if I want to join a big fact table with several tiny dimension tables in Spark SQL (1.1)? I wish we can allow user hint for the join. Jianshi On Wed, Oct 8, 2014 at 2:18 PM, Jiansh

Re: How to do broadcast join in SparkSQL

Looks like https://issues.apache.org/jira/browse/SPARK-1800 is not merged into master? I cannot find spark.sql.hints.broadcastTables in latest master, but it's in the following patch. https://github.com/apache/spark/commit/76ca4341036b95f71763f631049fdae033990ab5 Jianshi On Mon, Sep 29, 2014

HiveServer1 and SparkSQL

Hi Shark supported both the HiveServer1 and HiveServer2 thrift interfaces (using $ bin/shark -service sharkserver[1 or 2]). SparkSQL seems to support only HiveServer2. I was wondering what is involved to add support for HiveServer1. Is this something straightforward to do that I can embark on

Re: [ANN] SparkSQL support for Cassandra with Calliope

> > > >Hi All, > > >An year ago we started this journey and laid the path for Spark + Cassandra >stack. We established the ground work and direction for Spark Cassandra >connectors and we have been happy seeing the results. > > >With Spark 1.1.0 and SparkSQL r

Re: [SparkSQL] Function parity with Shark?

I have created https://issues.apache.org/jira/browse/SPARK-3814 https://issues.apache.org/jira/browse/SPARK-3815 Will probably try my hand at 3814, seems like a good place to get started... On Fri, Oct 3, 2014 at 3:06 PM, Michael Armbrust wrote: > Thanks for digging in! These both look like t

Re: [ANN] SparkSQL support for Cassandra with Calliope

Tian > > > On Friday, October 3, 2014 11:15 AM, Rohit Rai > wrote: > > > Hi All, > > An year ago we started this journey and laid the path for Spark + > Cassandra stack. We established the ground work and direction for Spark > Cassandra connectors and we have been

Re: [ANN] SparkSQL support for Cassandra with Calliope

ey and laid the path for Spark + Cassandra stack. We established the ground work and direction for Spark Cassandra connectors and we have been happy seeing the results. With Spark 1.1.0 and SparkSQL release, we its time to take Calliope to the logical next level also paving the way for much more a

Re: [SparkSQL] Function parity with Shark?

Thanks for digging in! These both look like they should have JIRAs. On Fri, Oct 3, 2014 at 8:14 AM, Yana Kadiyska wrote: > Thanks -- it does appear that I misdiagnosed a bit: case works generally > but it doesn't seem to like the bit operation, which does not seem to work > (type of bit_field i

[ANN] SparkSQL support for Cassandra with Calliope

Hi All, An year ago we started this journey and laid the path for Spark + Cassandra stack. We established the ground work and direction for Spark Cassandra connectors and we have been happy seeing the results. With Spark 1.1.0 and SparkSQL release, we its time to take Calliope <h

Re: [SparkSQL] Function parity with Shark?

Thanks -- it does appear that I misdiagnosed a bit: case works generally but it doesn't seem to like the bit operation, which does not seem to work (type of bit_field in Hive is bigint): Error: java.lang.RuntimeException: Unsupported language features in query: select (case when bit_field & 1=1 th

Re: SparkSQL on Hive error

Also make sure to call |hiveContext.sql| within the same thread where |hiveContext| is created, because Hive uses thread-local variable to initialize the |Driver.conf|. On 10/3/14 4:52 PM, Michael Armbrust wrote: Are you running master? There was briefly a regression here that is hopefully f

Re: SparkSQL on Hive error

hi, I have just tested the same command, it works here, can you please provide your create table command? regards Arthur scala> hiveContext.hql("show tables") warning: there were 1 deprecation warning(s); re-run with -deprecation for details 2014-10-03 17:14:33,575 INFO [main] parse.ParseDriv

Re: SparkSQL on Hive error

Are you running master? There was briefly a regression here that is hopefully fixed by spark#2635 . On Fri, Oct 3, 2014 at 1:43 AM, Kevin Paul wrote: > Hi all, I tried to launch my application with spark-submit, the command I > use is: > > bin/spark-su

SparkSQL on Hive error

Hi all, I tried to launch my application with spark-submit, the command I use is: bin/spark-submit --class ${MY_CLASS} --jars ${MY_JARS} --master local myApplicationJar.jar I've buillt spark with SPARK_HIVE=true, and was able to start HiveContext, and was able to run command like, hiveContext.sql

Re: SparkSQL DataType mappings

Hi Yin, Thanks for the reply. I've found the section as well, a couple of days ago and managed to integrate es-hadoop with Spark SQL [1] Cheers, [1] http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/master/spark.html On 10/2/14 6:32 PM, Yin Huai wrote: Hi Costin, I am answering yo

Re: [SparkSQL] Function parity with Shark?

What are the errors you are seeing. All of those functions should work. On Thu, Oct 2, 2014 at 6:56 AM, Yana Kadiyska wrote: > Hi, in an effort to migrate off of Shark I recently tried the Thrift JDBC > server that comes with Spark 1.1.0. > > However I observed that conditional functions do not

Re: SparkSQL DataType mappings

Hi Costin, I am answering your questions below. 1. You can find Spark SQL data type reference at here . It explains the underlying data type for a Spark SQL data type for Scala, Java, and Python APIs. For

[SparkSQL] Function parity with Shark?

Hi, in an effort to migrate off of Shark I recently tried the Thrift JDBC server that comes with Spark 1.1.0. However I observed that conditional functions do not work (I tried 'case' and 'coalesce') some string functions like 'concat' also did not work. Is there a list of what's missing or a ro

How to read just specified columns from parquet file using SparkSQL.

Hi, I am new to SparkSQL. I want to read the specified columns from the parquet, not all the columns defined in the parquet file. For instance, the schema of the parquet file would look like this: { "type": "record", "name": "ElectricPowerUsage",

Re: Unresolved attributes: SparkSQL on the schemaRDD

message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339p15376.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --

SparkSQL DataType mappings

Hi, I'm working on supporting SchemaRDD in Elasticsearch Hadoop [1] but I'm having some issues with the SQL API, in particular in what the DataTypes translate to. 1. A SchemaRDD is composed of a Row and StructType - I'm using the latter to decompose a Row into primitives. I'm not clear howeve

Re: Unresolved attributes: SparkSQL on the schemaRDD

I'm using the latest version i.e. Spark 1.1.0 Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-SparkSQL-on-the-schemaRDD-tp15339p15376.html Sent from the Apache Spark User List mailing list archive at Nabbl

Re: Unresolved attributes: SparkSQL on the schemaRDD

issing something? I'm trying to understand how the RDD is stored. >>> >>> Thanks in advance. >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-att

Re: Unresolved attributes: SparkSQL on the schemaRDD

>> >> Am I missing something? I'm trying to understand how the RDD is stored. >> >> Thanks in advance. >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attribut

Re: Unresolved attributes: SparkSQL on the schemaRDD

> > > Am I missing something? I'm trying to understand how the RDD is stored. > > Thanks in advance. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-

Re: Unresolved attributes: SparkSQL on the schemaRDD

but identifier .age found SELECT data[0].name FROM people where data[0].age >=13* Am I missing something? I'm trying to understand how the RDD is stored. Thanks in advance. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Unresolved-attributes-Spa

Re: Unresolved attributes: SparkSQL on the schemaRDD

In your case, the table has only one row, whose contents is “data”, which is an array. You need something like |SELECT data[0].name FROM json_table| to access the |name| field. On 9/29/14 11:08 PM, vdiwakar.malladi wrote: Hello, I'm exploring SparkSQL and I'm facing issue while

Unresolved attributes: SparkSQL on the schemaRDD

Hello, I'm exploring SparkSQL and I'm facing issue while using the queries. Any help on this is appreciated. I have the following schema once loaded as RDD. root |-- data: array (nullable = true) ||-- element: struct (containsNull = false) |||-- age: integer (nulla

Re: SparkSQL: map type MatchError when inserting into Hive table

It turned out a bug in my code. In the select clause the list of fields is misaligned with the schema of the target table. As a consequence the map data couldn’t be cast to some other type in the schema. Thanks anyway. On 9/26/14, 8:08 PM, "Cheng Lian" wrote: >Would you mind to provide the DDL

Re: How to do broadcast join in SparkSQL

Yes, looks like it can only be controlled by the parameter spark.sql.autoBroadcastJoinThreshold, which is a little bit weird to me. How am I suppose to know the exact bytes of a table? Let me specify the join algorithm is preferred I think. Jianshi On Sun, Sep 28, 2014 at 11:57 PM, Ted Yu wrote

Re: How to do broadcast join in SparkSQL

Have you looked at SPARK-1800 ? e.g. see sql/core/src/test/scala/org/apache/spark/sql/JoinSuite.scala Cheers On Sun, Sep 28, 2014 at 1:55 AM, Jianshi Huang wrote: > I cannot find it in the documentation. And I have a dozen dimension tables > to (left) join... > > > Cheers, > -- > Jianshi Huang

How to do broadcast join in SparkSQL

I cannot find it in the documentation. And I have a dozen dimension tables to (left) join... Cheers, -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/

Re: SparkSQL: map type MatchError when inserting into Hive table

Would you mind to provide the DDL of this partitioned table together with the query you tried? The stacktrace suggests that the query was trying to cast a map into something else, which is not supported in Spark SQL. And I doubt whether Hive support casting a complex type to some other type.

Re: SparkSQL: map type MatchError when inserting into Hive table

Would you mind to provide the DDL of this partitioned table together with the query you tried? The stacktrace suggests that the query was trying to cast a map into something else, which is not supported in Spark SQL. And I doubt whether Hive support casting a complex type to some other type.

Re: SparkSQL: map type MatchError when inserting into Hive table

It might be a problem when inserting into a partitioned table. It worked fine to when the target table was unpartitioned. Can you confirm this? Thanks, Du On 9/26/14, 4:48 PM, "Du Li" wrote: >Hi, > >I was loading data into a partitioned table on Spark 1.1.0 >beeline-thriftserver. The table

SparkSQL: map type MatchError when inserting into Hive table

Hi, I was loading data into a partitioned table on Spark 1.1.0 beeline-thriftserver. The table has complex data types such as map and array>. The query is like ³insert overwrite table a partition (Š) select Š² and the select clause worked if run separately. However, when running the insert query,

Re: SparkSQL Thriftserver in Mesos

1/14 1:16 AM, John Omernik wrote: I am running the Thrift server in SparkSQL, and running it on the node I compiled spark on. When I run it, tasks only work if they landed on that node, other executors started on nodes I didn't compile spark on (and thus don't have the compile

Re: SparkSQL: Freezing while running TPC-H query 5

be added to the discussion > below: > > http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-Freezing-while-running-TPC-H-query-5-tp14902p14921.html > To unsubscribe from SparkSQL: Freezing while running TPC-H query 5, click > here > <http://apache-spark-user-list.1001560.n3

Re: Exception with SparkSql and Avro

Thanks, I didn't create the tables myself as I have no control over that process. However these tables are read just fund using the Jdbc connection to the hiveserver2 so it should be possible On Sep 24, 2014 12:48 AM, Michael Armbrust wrote: Can you show me the DDL you are using? Here is an exa

Re: Exception with SparkSql and Avro

Can you show me the DDL you are using? Here is an example of a way I got the avro serde to work: https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala#L246 Also, this isn't ready for primetime yet, but a quick plug for some ongoing work: http

SparkSQL: Freezing while running TPC-H query 5

Hi, I am trying to run TPC-H queries with SparkSQL 1.1.0 CLI with 1 r3.4xlarge master + 20 r3.4xlarge slave machines on EC2 (each machine has 16vCPUs, 122GB memory). The TPC-H scale factor I am using is 1000 (i.e. 1000GB of total data). When I try to run TPC-H query 5, the query hangs for a

RE: Exception with SparkSql and Avro

Hello, I am trying to read a hive table that is stored in Avro DEFLATE files. something simple like "SELECT * FROM X LIMIT 10" I get 2 exceptions in the logs: 2014-09-23 09:27:50,157 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 10.0 in stage 1.0 (TID 10, cl.local): org.apache.avro.A

Re: SparkSQL Thriftserver in Mesos

//polyglotprogramming.com On Mon, Sep 22, 2014 at 2:35 PM, John Omernik wrote: > Any thoughts on this? > > On Sat, Sep 20, 2014 at 12:16 PM, John Omernik wrote: > >> I am running the Thrift server in SparkSQL, and running it on the node I >> compiled spark on. When I run

Re: SparkSQL Thriftserver in Mesos

Any thoughts on this? On Sat, Sep 20, 2014 at 12:16 PM, John Omernik wrote: > I am running the Thrift server in SparkSQL, and running it on the node I > compiled spark on. When I run it, tasks only work if they landed on that > node, other executors started on nodes I didn't co

SparkSQL: Key not valid while running TPC-H

Hi, I am trying to run TPC-H queries with SparkSQL 1.1.0 CLI with 1 r3.4xlarge master + 20 r3.4xlarge slave machines on EC2 (each machine has 16vCPUs, 122GB memory). The TPC-H scale factor I am using is 1000 (i.e. 1000GB of total data). When I try to run TPC-H query 3 i.e. select l_orderkey, sum

SparkSQL Thriftserver in Mesos

I am running the Thrift server in SparkSQL, and running it on the node I compiled spark on. When I run it, tasks only work if they landed on that node, other executors started on nodes I didn't compile spark on (and thus don't have the compile directory) fail. Should spark be d

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

Thank you Yin Huai. This is probably true. I saw in the hive-site.xml, Liu has changed the entry, which is default should be false. hive.support.concurrency Enable Hive's Table Lock Manager Service true Someone is working on upgrading the Hive to 0.13 for SparkSQL (

Re: SparkSQL 1.1 hang when "DROP" or "LOAD"

sting the steps, I will look at this, hopefully come out >> with a solution soon. >> >> -Original Message- >> From: linkpatrickliu [mailto:linkpatrick...@live.com] >> Sent: Tuesday, September 16, 2014 3:17 PM >> To: u...@spark.incubator.apache.org >&

Re: SparkSQL 1.1 hang when "DROP" or "LOAD"

lto:linkpatrick...@live.com] > Sent: Tuesday, September 16, 2014 3:17 PM > To: u...@spark.incubator.apache.org > Subject: RE: SparkSQL 1.1 hang when "DROP" or "LOAD" > > Hi, Hao Cheng. > > I have done other tests. And the result shows the thriftServer ca

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

Thank you for pasting the steps, I will look at this, hopefully come out with a solution soon. -Original Message- From: linkpatrickliu [mailto:linkpatrick...@live.com] Sent: Tuesday, September 16, 2014 3:17 PM To: u...@spark.incubator.apache.org Subject: RE: SparkSQL 1.1 hang when "

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

cceed. So I think the reason is that the thriftServer cannot release locks correctly in Zookeeper. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-1-hang-when-DROP-or-LOAD-tp14222p14339.ht

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

Seems like the thriftServer cannot connect to Zookeeper, so it cannot get lock. This is how it the log looks when I run SparkSQL: "load data inpath "kv1.txt" into table src;" log: 14/09/16 14:40:47 INFO Driver: 14/09/16 14:40:47 INFO ClientCnxn: Opening socket connection to

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

USTOM) -Original Message- From: linkpatrickliu [mailto:linkpatrick...@live.com] Sent: Tuesday, September 16, 2014 1:10 PM To: u...@spark.incubator.apache.org Subject: RE: SparkSQL 1.1 hang when "DROP" or "LOAD" Besides， When I use bin/spark-sql, I can Load data and drop table

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

Besides， When I use bin/spark-sql, I can Load data and drop table freely. Only when I use sbin/start-thriftserver.sh and connect with beeline, the client will hang! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-1-hang-when-DROP-or-LOAD

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

DskipTests package -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-1-hang-when-DROP-or-LOAD-tp14222p14325.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

From: linkpatrickliu [mailto:linkpatrick...@live.com] Sent: Tuesday, September 16, 2014 12:14 PM To: u...@spark.incubator.apache.org Subject: RE: SparkSQL 1.1 hang when "DROP" or "LOAD" Hi， Hao Cheng， Here is the Spark\Hadoop version: Spark version = 1.1.0 Hadoop version = 2.0.0-cd

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

ill grant select and drop privilege to the owner of the table -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-1-hang-when-DROP-or-LOAD-tp14222p14320.html Sent from the Apache Spark User List mailing l

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

ill grant select and drop privilege to the owner of the table -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-1-hang-when-DROP-or-LOAD-tp14222p14319.html Sent from the Apache Spark User List mailing l

RE: SparkSQL 1.1 hang when "DROP" or "LOAD"

To: u...@spark.incubator.apache.org Subject: SparkSQL 1.1 hang when "DROP" or "LOAD" I started sparkSQL thrift server: "sbin/start-thriftserver.sh" Then I use beeline to connect to it: "bin/beeline" "!connect jdbc:hive2://localhost:1 op1 op1" I have created

Re: About SparkSQL 1.1.0 join between more than two table

1.0.1 does not have the support on outer joins (added in 1.1). Your query should be fine in 1.1. On Mon, Sep 15, 2014 at 5:35 AM, Yanbo Liang wrote: > Spark SQL can support SQL and HiveSQL which used SQLContext and > HiveContext separate. > As far as I know, SQLContext of Spark SQL 1.1.0 can not

Re: About SparkSQL 1.1.0 join between more than two table

Spark SQL can support SQL and HiveSQL which used SQLContext and HiveContext separate. As far as I know, SQLContext of Spark SQL 1.1.0 can not support three table join directly. However you can modify your query with subquery such as SELECT * FROM (SELECT * FROM youhao_data left join youhao_age on

SparkSQL 1.1 hang when "DROP" or "LOAD"

I started sparkSQL thrift server: "sbin/start-thriftserver.sh" Then I use beeline to connect to it: "bin/beeline" "!connect jdbc:hive2://localhost:1 op1 op1" I have created a database for user op1. "create database dw_op1"; And grant all privileges to

About SparkSQL 1.1.0 join between more than two table

Hi: When I use spark SQL (1.0.1), I found it not support join between three tables,eg: sql("SELECT * FROM youhao_data left join youhao_age on (youhao_data.rowkey=youhao_age.rowkey) left join youhao_totalKiloMeter on (youhao_age.rowkey=youhao_totalKiloMeter.rowkey)") I take the Exception: Excep

Re: SparkSQL hang due to

What is in your hive-site.xml? On Thu, Sep 11, 2014 at 11:04 PM, linkpatrickliu wrote: > I am running Spark Standalone mode with Spark 1.1 > > I started SparkSQL thrift server as follows: > ./sbin/start-thriftserver.sh > > Then I use beeline to connect to it. > Now, I

SparkSQL hang due to

I am running Spark Standalone mode with Spark 1.1 I started SparkSQL thrift server as follows: ./sbin/start-thriftserver.sh Then I use beeline to connect to it. Now, I can "CREATE", "SELECT", "SHOW" the databases or the tables; But when I "DROP" or &q

Re: Table not found: using jdbc console to query sparksql hive thriftserver

@yahoo-inc.com>> Cc: "u...@spark.incubator.apache.org<mailto:u...@spark.incubator.apache.org>" mailto:u...@spark.incubator.apache.org>>, alexandria1101 mailto:alexandria.shea...@gmail.com>> Subject: Re: Table not found: using jdbc console to query sparksql hive

Re: Table not found: using jdbc console to query sparksql hive thriftserver

text to >>JDBC >> somehow? >> >> >> >> -- >> View this message in context: >>http://apache-spark-user-list.1001560.n3.nabble.com/Table-not-found-using >>-jdbc-consol

Re: SparkSQL HiveContext TypeTag compile error

PM To: "user@spark.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Subject: Re: SparkSQL HiveContext TypeTag compile error Solved it. The problem occurred because the case class was defined within a test case in FunSuite. Moving the case class defi

Re: SparkSQL HiveContext TypeTag compile error

.apache.org<mailto:user@spark.apache.org>" mailto:user@spark.apache.org>> Subject: SparkSQL HiveContext TypeTag compile error Hi, I have the following code snippet. It works fine on spark-shell but in a standalone app it reports "No TypeTag available for MySchema” at compile time

SparkSQL HiveContext TypeTag compile error

Hi, I have the following code snippet. It works fine on spark-shell but in a standalone app it reports "No TypeTag available for MySchema” at compile time when calling hc.createScheamaRdd(rdd). Anybody knows what might be missing? Thanks, Du -- Import org.apache.spark.sql.hive.HiveContext

Re: Table not found: using jdbc console to query sparksql hive thriftserver

Thank you!! I can do this using saveAsTable with the schemaRDD, right? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Table-not-found-using-jdbc-console-to-query-sparksql-hive-thriftserver-tp13840p13979.html Sent from the Apache Spark User List mailing

Re: Table not found: using jdbc console to query sparksql hive thriftserver

t;JDBC >> somehow? >> >> >> >> -- >> View this message in context: >>http://apache-spark-user-list.1001560.n3.nabble.com/Table-not-found-using >>-jdbc-console-to-query-sparksql-hive-thriftserver-tp13840p13922.html >> Sent from the Apache Spark U

Re: Table not found: using jdbc console to query sparksql hive thriftserver

ng found by the thrift server. Do I have to pass the hiveContext to JDBC > somehow? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Table-not-found-using-jdbc-console-to-query-sparksql-hive-thriftserver-tp13840p13922.html &g

Re: Table not found: using jdbc console to query sparksql hive thriftserver

-sparksql-hive-thriftserver-tp13840p13922.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h

Re: Table not found: using jdbc console to query sparksql hive thriftserver

n context: >http://apache-spark-user-list.1001560.n3.nabble.com/Table-not-found-using- >jdbc-console-to-query-sparksql-hive-thriftserver-tp13840p13851.html >Sent from the Apache Spark User List mailing list archive at Nabble.com. > >-

Re: Table not found: using jdbc console to query sparksql hive thriftserver

.n3.nabble.com/Table-not-found-using-jdbc-console-to-query-sparksql-hive-thriftserver-tp13840p13851.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsub

Re: Table not found: using jdbc console to query sparksql hive thriftserver

/Table-not-found-using-jdbc-console-to-query-sparksql-hive-thriftserver-tp13840p13847.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For

Re: Table not found: using jdbc console to query sparksql hive thriftserver

Your tables were registered in the SqlContext, whereas the thrift server works with HiveContext. They seem to be in two different worlds today. On 9/9/14, 5:16 PM, "alexandria1101" wrote: >Hi, > >I want to use the sparksql thrift server in my application and make sure >

Table not found: using jdbc console to query sparksql hive thriftserver

Hi, I want to use the sparksql thrift server in my application and make sure everything is loading and working. I built Spark 1.1 SNAPSHOT and ran the thrift server using ./sbin/start-thrift-server. In my application I load tables into schemaRDDs and I expect that the thrift-server should pick

Error while running sparkSQL application in the cluster-mode environment

Hello, I tried to execute a simple spark application using sparkSQL. At first try, it worked as I exepcted but after then, it doesn't run and shows an stderr like below: Spark Executor Command: "java" "-cp" "::/opt/spark-1.0.2-bin-hadoop2/conf:/opt/spa

Re: TimeStamp selection with SparkSQL

f the SQLContext. >>> >>> >>> >>> In this particular query you showed, seems datatime is the type Date, >>> unfortunately, neither of those SQL dialect supports Date, but Timestamp. >>> >>> >>> >>> Cheng Hao >>>

<    5   6   7   8   9   10   11   12   >