Hi Adline,
rowNumber and row_number are same functions:
@scala.deprecated("Use row_number. This will be removed in Spark 2.0.")
def rowNumber() : org.apache.spark.sql.Column = { /* compiled code */ }
def row_number() : org.apache.spark.sql.Column = { /* compiled code */ }
but the issue here is about using SQLContext instead of HiveContext.
Thanks,
Saurabh
-----Original Message-----
From: Adline Dsilva [mailto:adline.dsi...@mimos.my]
Sent: 01 September 2016 11:05
To: saurabh3d; user@spark.apache.org
Subject: RE: Window Functions with SQLContext
Hi,
Use function rowNumber instead of row_number
df1.withColumn("row_number", rowNumber.over(w));
Regards,
Adline
________________________________________
From: saurabh3d [saurabh.s.du...@oracle.com]
Sent: 01 September 2016 13:16
To: user@spark.apache.org
Subject: Window Functions with SQLContext
Hi All,
As per SPARK-11001 <https://issues.apache.org/jira/browse/SPARK-11001> ,
Window functions should be supported by SQLContext. But when i try to run
SQLContext sqlContext = new SQLContext(jsc); WindowSpec w =
Window.partitionBy("assetId").orderBy("assetId");
DataFrame df_2 = df1.withColumn("row_number", row_number().over(w));
df_2.show(false);
it fails with:
Exception in thread "main" org.apache.spark.sql.AnalysisException: Could not
resolve window function 'row_number'. Note that, using window functions
currently requires a HiveContext;
This code runs fine with HiveContext.
Any idea what's going on? Is this a known issue and is there a workaround to
make Window function work without HiveContext.
Thanks,
Saurabh
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Window-Functions-with-SQLContext-tp27636.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
________________________________
DISCLAIMER:
This e-mail (including any attachments) is for the addressee(s) only and may be
confidential, especially as regards personal data. If you are not the intended
recipient, please note that any dealing, review, distribution, printing,
copying or use of this e-mail is strictly prohibited. If you have received this
email in error, please notify the sender immediately and delete the original
message (including any attachments).
MIMOS Berhad is a research and development institution under the purview of the
Malaysian Ministry of Science, Technology and Innovation. Opinions, conclusions
and other information in this e-mail that do not relate to the official
business of MIMOS Berhad and/or its subsidiaries shall be understood as neither
given nor endorsed by MIMOS Berhad and/or its subsidiaries and neither MIMOS
Berhad nor its subsidiaries accepts responsibility for the same. All liability
arising from or in connection with computer viruses and/or corrupted e-mails is
excluded to the fullest extent permitted by law.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
