Hi Ayan,
Even with SQL query like:
SQLContext sqlContext = new SQLContext(jsc);
DataFrame df_2_sql = sqlContext.sql("select assetId, row_number() over (
partition by " +
"assetId order by assetId) as " +
"serial from df1");
df_2_sql.show(false);
It fails with:
Exception in thread "main" java.lang.RuntimeException: [1.35] failure:
``union'' expected but `(' found
select assetId, row_number() over ( partition by assetId order by assetId) as
serial from df1
^
at scala.sys.package$.error(package.scala:27)
at
org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
at
org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
at
org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:211)
at
org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:211)
Thanks
From: ayan guha [mailto:guha.a...@gmail.com]
Sent: 01 September 2016 11:12
To: saurabh3d
Cc: user
Subject: Re: Window Functions with SQLContext
I think you can write the SQL query and run it usin sqlContext.
like
select *,row_number() over(partitin by assetid order by assetid) rn from t
On Thu, Sep 1, 2016 at 3:16 PM, saurabh3d <HYPERLINK
"mailto:saurabh.s.du...@oracle.com"; \nsaurabh.s.du...@oracle.com> wrote:
Hi All,
As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001> ,
Window functions should be supported by SQLContext. But when i try to run
SQLContext sqlContext = new SQLContext(jsc);
WindowSpec w = Window.partitionBy("assetId").orderBy("assetId");
DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
df_2.show(false);
it fails with:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Could not
resolve window function 'row_number'. Note that, using window functions
currently requires a HiveContext;
This code runs fine with HiveContext.
Any idea what’s going on? Is this a known issue and is there a workaround
to make Window function work without HiveContext.
Thanks,
Saurabh
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: HYPERLINK
"mailto:user-unsubscr...@spark.apache.org"user-unsubscr...@spark.apache.org
--
Best Regards,
Ayan Guha
