Re: Mathematical functions in spark sql

2015-01-26 Thread Alexey Romanchuk
I have tried "select ceil(2/3)", but got "key not found: floor" On Tue, Jan 27, 2015 at 11:05 AM, Ted Yu wrote: > Have you tried floor() or ceil() functions ? > > According to http://spark.apache.org/sql/, Spark SQL is compatible with > Hive SQL. > > Cheers > > On Mon, Jan 26, 2015 at 8:29 PM, 1

Re: akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down

2014-12-02 Thread Alexey Romanchuk
Any ideas? Anyone got the same error? On Mon, Dec 1, 2014 at 2:37 PM, Alexey Romanchuk wrote: > Hello spark users! > > I found lots of strange messages in driver log. Here it is: > > 2014-12-01 11:54:23,849 [sparkDriver-akka.actor.default-dispatcher-25] > ERROR > akka.remot

akka.remote.transport.Transport$InvalidAssociationException: The remote system terminated the association because it is shutting down

2014-12-01 Thread Alexey Romanchuk
Hello spark users! I found lots of strange messages in driver log. Here it is: 2014-12-01 11:54:23,849 [sparkDriver-akka.actor.default-dispatcher-25] ERROR akka.remote.EndpointWriter[akka://sparkDriver/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FsparkExecutor%40data1.hadoop%3A1

Re: Delayed hotspot optimizations in Spark

2014-10-10 Thread Alexey Romanchuk
show > whether the upfront compilation really helps. I doubt it. > > However is this almost surely due to caching somewhere, in Spark SQL > or HDFS? I really doubt hotspot makes a difference compared to these > much larger factors. > > On Fri, Oct 10, 2014 at 8:49 AM, Alexe

Delayed hotspot optimizations in Spark

2014-10-10 Thread Alexey Romanchuk
Hello spark users and developers! I am using hdfs + spark sql + hive schema + parquet as storage format. I have lot of parquet files - one files fits one hdfs block for one day. The strange thing is very slow first query for spark sql. To reproduce situation I use only one core and I have 97sec f

Re: Log hdfs blocks sending

2014-09-26 Thread Alexey Romanchuk
ocal and does remote reads which really kills > performance. > > Hope that helps! > Andrew > > On Thu, Sep 25, 2014 at 12:09 AM, Alexey Romanchuk < > alexey.romanc...@gmail.com> wrote: > >> Hello again spark users and developers! >> >> I have st

Log hdfs blocks sending

2014-09-25 Thread Alexey Romanchuk
Hello again spark users and developers! I have standalone spark cluster (1.1.0) and spark sql running on it. My cluster consists of 4 datanodes and replication factor of files is 3. I use thrift server to access spark sql and have 1 table with 30+ partitions. When I run query on whole table (some