[jira] [Commented] (SPARK-24497) ANSI SQL: Recursive query

2020-02-18 Thread Daniel Mateus Pires (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-24497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039498#comment-17039498
 ] 

Daniel Mateus Pires commented on SPARK-24497:
-

Hey! the PR linked to this issue has merge conflicts and reviewers didn't come 
back to it for another round of reviews, just wanted to notify on this thread 
that this feature would be very useful :+1:

> ANSI SQL: Recursive query
> -
>
> Key: SPARK-24497
> URL: https://issues.apache.org/jira/browse/SPARK-24497
> Project: Spark
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Yuming Wang
>Priority: Major
>
> h3. *Examples*
> Here is an example for {{WITH RECURSIVE}} clause usage. Table "department" 
> represents the structure of an organization as an adjacency list.
> {code:sql}
> CREATE TABLE department (
> id INTEGER PRIMARY KEY,  -- department ID
> parent_department INTEGER REFERENCES department, -- upper department ID
> name TEXT -- department name
> );
> INSERT INTO department (id, parent_department, "name")
> VALUES
>  (0, NULL, 'ROOT'),
>  (1, 0, 'A'),
>  (2, 1, 'B'),
>  (3, 2, 'C'),
>  (4, 2, 'D'),
>  (5, 0, 'E'),
>  (6, 4, 'F'),
>  (7, 5, 'G');
> -- department structure represented here is as follows:
> --
> -- ROOT-+->A-+->B-+->C
> --  | |
> --  | +->D-+->F
> --  +->E-+->G
> {code}
>  
>  To extract all departments under A, you can use the following recursive 
> query:
> {code:sql}
> WITH RECURSIVE subdepartment AS
> (
> -- non-recursive term
> SELECT * FROM department WHERE name = 'A'
> UNION ALL
> -- recursive term
> SELECT d.*
> FROM
> department AS d
> JOIN
> subdepartment AS sd
> ON (d.parent_department = sd.id)
> )
> SELECT *
> FROM subdepartment
> ORDER BY name;
> {code}
> More details:
> [http://wiki.postgresql.org/wiki/CTEReadme]
> [https://info.teradata.com/htmlpubs/DB_TTU_16_00/index.html#page/SQL_Reference/B035-1141-160K/lqe1472241402390.html]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-23194) from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls

2018-09-28 Thread Daniel Mateus Pires (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-23194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631652#comment-16631652
 ] 

Daniel Mateus Pires commented on SPARK-23194:
-

Any news on this ? not being able to set the from_json mode and use the 
columnNameOfCorruptRecord option is pretty limiting, and the documentation of 
"from_json" suggests that all the spark.read.json options are available


{code:java}
   * @param options options to control how the json is parsed. accepts the same 
options and the json data source.
{code}


> from_json in FAILFAST mode doesn't fail fast, instead it just returns nulls
> ---
>
> Key: SPARK-23194
> URL: https://issues.apache.org/jira/browse/SPARK-23194
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Burak Yavuz
>Priority: Major
>
> from_json accepts Json parsing options such as being PERMISSIVE to parsing 
> errors or failing fast. It seems from the code that even though the default 
> option is to fail fast, we catch that exception and return nulls.
>  
> In order to not change behavior, we should remove that try-catch block and 
> change the default to permissive, but allow failfast mode to indeed fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-25480) Dynamic partitioning + saveAsTable with multiple partition columns create empty directory

2018-09-20 Thread Daniel Mateus Pires (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-25480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622814#comment-16622814
 ] 

Daniel Mateus Pires commented on SPARK-25480:
-

* Didn't try accessing S3 without EMR
* EMRFS consistent view:Disabled

Release label:emr-5.13.0
Hadoop distribution:Amazon 2.8.3
Applications:Spark 2.3.0, Zeppelin 0.7.3, Hive 2.3.2

I'll try and reproduce without using EMR, and without using S3 and will update 
the ticket 

> Dynamic partitioning + saveAsTable with multiple partition columns create 
> empty directory
> -
>
> Key: SPARK-25480
> URL: https://issues.apache.org/jira/browse/SPARK-25480
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Daniel Mateus Pires
>Priority: Minor
> Attachments: dynamic_partitioning.json
>
>
> We use .saveAsTable and dynamic partitioning as our only way to write data to 
> S3 from Spark.
> When only 1 partition column is defined for a table, .saveAsTable behaves as 
> expected:
>  - with Overwrite mode it will create a table if it doesn't exist and write 
> the data
>  - with Append mode it will append to a given partition
>  - with Overwrite mode if the table exists it will overwrite the partition
> If 2 partition columns are used however, the directory is created on S3 with 
> the SUCCESS file, but no data is actually written
> our solution is to check if the table doesn't exist, and in that case, set 
> the partitioning mode back to static before running saveAsTable:
> {code}
> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> df.write.mode("overwrite").partitionBy("year", "month").option("path", 
> "s3://hbc-data-warehouse/integration/users_test").saveAsTable("users_test")
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-25480) Dynamic partitioning + saveAsTable with multiple partition columns create empty directory

2018-09-20 Thread Daniel Mateus Pires (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-25480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621771#comment-16621771
 ] 

Daniel Mateus Pires commented on SPARK-25480:
-

I've added an exported Zeppelin notebook to reproduce the issue

> Dynamic partitioning + saveAsTable with multiple partition columns create 
> empty directory
> -
>
> Key: SPARK-25480
> URL: https://issues.apache.org/jira/browse/SPARK-25480
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Daniel Mateus Pires
>Priority: Minor
> Attachments: dynamic_partitioning.json
>
>
> We use .saveAsTable and dynamic partitioning as our only way to write data to 
> S3 from Spark.
> When only 1 partition column is defined for a table, .saveAsTable behaves as 
> expected:
>  - with Overwrite mode it will create a table if it doesn't exist and write 
> the data
>  - with Append mode it will append to a given partition
>  - with Overwrite mode if the table exists it will overwrite the partition
> If 2 partition columns are used however, the directory is created on S3 with 
> the SUCCESS file, but no data is actually written
> our solution is to check if the table doesn't exist, and in that case, set 
> the partitioning mode back to static before running saveAsTable:
> {code}
> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> df.write.mode("overwrite").partitionBy("year", "month").option("path", 
> "s3://hbc-data-warehouse/integration/users_test").saveAsTable("users_test")
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-25480) Dynamic partitioning + saveAsTable with multiple partition columns create empty directory

2018-09-20 Thread Daniel Mateus Pires (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-25480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Mateus Pires updated SPARK-25480:

Attachment: dynamic_partitioning.json

> Dynamic partitioning + saveAsTable with multiple partition columns create 
> empty directory
> -
>
> Key: SPARK-25480
> URL: https://issues.apache.org/jira/browse/SPARK-25480
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Daniel Mateus Pires
>Priority: Minor
> Attachments: dynamic_partitioning.json
>
>
> We use .saveAsTable and dynamic partitioning as our only way to write data to 
> S3 from Spark.
> When only 1 partition column is defined for a table, .saveAsTable behaves as 
> expected:
>  - with Overwrite mode it will create a table if it doesn't exist and write 
> the data
>  - with Append mode it will append to a given partition
>  - with Overwrite mode if the table exists it will overwrite the partition
> If 2 partition columns are used however, the directory is created on S3 with 
> the SUCCESS file, but no data is actually written
> our solution is to check if the table doesn't exist, and in that case, set 
> the partitioning mode back to static before running saveAsTable:
> {code}
> spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
> df.write.mode("overwrite").partitionBy("year", "month").option("path", 
> "s3://hbc-data-warehouse/integration/users_test").saveAsTable("users_test")
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-25480) Dynamic partitioning + saveAsTable with multiple partition columns create empty directory

2018-09-20 Thread Daniel Mateus Pires (JIRA) 
Daniel Mateus Pires created SPARK-25480:
---

 Summary: Dynamic partitioning + saveAsTable with multiple 
partition columns create empty directory
 Key: SPARK-25480
 URL: https://issues.apache.org/jira/browse/SPARK-25480
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 2.3.0
Reporter: Daniel Mateus Pires


We use .saveAsTable and dynamic partitioning as our only way to write data to 
S3 from Spark.

When only 1 partition column is defined for a table, .saveAsTable behaves as 
expected:
 - with Overwrite mode it will create a table if it doesn't exist and write the 
data
 - with Append mode it will append to a given partition
 - with Overwrite mode if the table exists it will overwrite the partition

If 2 partition columns are used however, the directory is created on S3 with 
the SUCCESS file, but no data is actually written

our solution is to check if the table doesn't exist, and in that case, set the 
partitioning mode back to static before running saveAsTable:
{code}
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
df.write.mode("overwrite").partitionBy("year", "month").option("path", 
"s3://hbc-data-warehouse/integration/users_test").saveAsTable("users_test")
{code}
 




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24798) sortWithinPartitions(xx) will failed in java.lang.NullPointerException

2018-07-15 Thread Daniel Mateus Pires (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544528#comment-16544528
 ] 

Daniel Mateus Pires commented on SPARK-24798:
-

+1 it's solved by using "Option"

> sortWithinPartitions(xx) will failed in java.lang.NullPointerException
> --
>
> Key: SPARK-24798
> URL: https://issues.apache.org/jira/browse/SPARK-24798
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: shengyao piao
>Priority: Minor
>
> I have some issue in Spark 2.3 when I run bellow code in spark-shell or 
> spark-submit 
> I already figured out the reason of error is the name field contains 
> Some(null),
> But I believe this code will run successfully in Spark 2.2
> Is it an expected behavior in Spark 2.3 ?
>  
> ・Spark code
> {code}
> case class Hoge (id : Int,name : Option[String])
>  val ds = 
> spark.createDataFrame(Array((1,"John"),(2,null))).withColumnRenamed("_1", 
> "id").withColumnRenamed("_2", "name").map(row => 
> Hoge(row.getAs[Int]("id"),Some(row.getAs[String]("name"
>  
> ds.sortWithinPartitions("id").foreachPartition(iter => println(iter.isEmpty))
> {code}
> ・Error
> {code}
> java.lang.NullPointerException
> at 
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter.write(UnsafeRowWriter.java:194)
> at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.serializefromobject_doConsume$(Unknown
>  Source)
> at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.mapelements_doConsume$(Unknown
>  Source)
> at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.deserializetoobject_doConsume$(Unknown
>  Source)
> at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.sort_addToSorter$(Unknown
>  Source)
> at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
>  Source)
> at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$class.isEmpty(Iterator.scala:330)
> at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1336)
> at 
> $line37.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(:26)
> at 
> $line37.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$anonfun$1.apply(:26)
> at 
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at 
> org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
> at 
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
> at 
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:109)
> at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Issue Comment Deleted] (SPARK-24702) Unable to cast to calendar interval in spark sql.

2018-07-03 Thread Daniel Mateus Pires (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-24702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Mateus Pires updated SPARK-24702:

Comment: was deleted

(was: https://github.com/apache/spark/pull/21706)

> Unable to cast to calendar interval in spark sql.
> -
>
> Key: SPARK-24702
> URL: https://issues.apache.org/jira/browse/SPARK-24702
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Priyanka Garg
>Priority: Major
>
> when I am trying to cast string type to calendar interval type, I am getting 
> the following error:
> spark.sql("select cast(cast(interval '1' day as string) as 
> calendarinterval)").show()
> ^^^
>  
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitPrimitiveDataType$1.apply(AstBuilder.scala:1673)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitPrimitiveDataType$1.apply(AstBuilder.scala:1651)
>   at 
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitPrimitiveDataType(AstBuilder.scala:1651)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitPrimitiveDataType(AstBuilder.scala:49)
>   at 
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$PrimitiveDataTypeContext.accept(SqlBaseParser.java:13779)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.typedVisit(AstBuilder.scala:55)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.org$apache$spark$sql$catalyst$parser$AstBuilder$$visitSparkDataType(AstBuilde



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24702) Unable to cast to calendar interval in spark sql.

2018-07-03 Thread Daniel Mateus Pires (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531480#comment-16531480
 ] 

Daniel Mateus Pires commented on SPARK-24702:
-

https://github.com/apache/spark/pull/21706

> Unable to cast to calendar interval in spark sql.
> -
>
> Key: SPARK-24702
> URL: https://issues.apache.org/jira/browse/SPARK-24702
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Priyanka Garg
>Priority: Major
>
> when I am trying to cast string type to calendar interval type, I am getting 
> the following error:
> spark.sql("select cast(cast(interval '1' day as string) as 
> calendarinterval)").show()
> ^^^
>  
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitPrimitiveDataType$1.apply(AstBuilder.scala:1673)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitPrimitiveDataType$1.apply(AstBuilder.scala:1651)
>   at 
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitPrimitiveDataType(AstBuilder.scala:1651)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitPrimitiveDataType(AstBuilder.scala:49)
>   at 
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$PrimitiveDataTypeContext.accept(SqlBaseParser.java:13779)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.typedVisit(AstBuilder.scala:55)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.org$apache$spark$sql$catalyst$parser$AstBuilder$$visitSparkDataType(AstBuilde



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24702) Unable to cast to calendar interval in spark sql.

2018-07-03 Thread Daniel Mateus Pires (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531474#comment-16531474
 ] 

Daniel Mateus Pires commented on SPARK-24702:
-

Got it working, I'll open a PR

> Unable to cast to calendar interval in spark sql.
> -
>
> Key: SPARK-24702
> URL: https://issues.apache.org/jira/browse/SPARK-24702
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Priyanka Garg
>Priority: Major
>
> when I am trying to cast string type to calendar interval type, I am getting 
> the following error:
> spark.sql("select cast(cast(interval '1' day as string) as 
> calendarinterval)").show()
> ^^^
>  
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitPrimitiveDataType$1.apply(AstBuilder.scala:1673)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitPrimitiveDataType$1.apply(AstBuilder.scala:1651)
>   at 
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitPrimitiveDataType(AstBuilder.scala:1651)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitPrimitiveDataType(AstBuilder.scala:49)
>   at 
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$PrimitiveDataTypeContext.accept(SqlBaseParser.java:13779)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.typedVisit(AstBuilder.scala:55)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.org$apache$spark$sql$catalyst$parser$AstBuilder$$visitSparkDataType(AstBuilde



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24702) Unable to cast to calendar interval in spark sql.

2018-07-03 Thread Daniel Mateus Pires (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531379#comment-16531379
 ] 

Daniel Mateus Pires commented on SPARK-24702:
-

The error is:

DataType calendarinterval is not supported.(line 1, pos 48)

== SQL ==
select cast(cast(interval '1' day as string) as calendarinterval)

I was able to reproduce it in 2.4.0, I can see CalendarIntervalType inside 
org.apache.sql.types so it is definitely supported, looking into it more (but 
I'm very new to the codebase) 

> Unable to cast to calendar interval in spark sql.
> -
>
> Key: SPARK-24702
> URL: https://issues.apache.org/jira/browse/SPARK-24702
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Priyanka Garg
>Priority: Major
>
> when I am trying to cast string type to calendar interval type, I am getting 
> the following error:
> spark.sql("select cast(cast(interval '1' day as string) as 
> calendarinterval)").show()
> ^^^
>  
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitPrimitiveDataType$1.apply(AstBuilder.scala:1673)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitPrimitiveDataType$1.apply(AstBuilder.scala:1651)
>   at 
> org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:108)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitPrimitiveDataType(AstBuilder.scala:1651)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.visitPrimitiveDataType(AstBuilder.scala:49)
>   at 
> org.apache.spark.sql.catalyst.parser.SqlBaseParser$PrimitiveDataTypeContext.accept(SqlBaseParser.java:13779)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.typedVisit(AstBuilder.scala:55)
>   at 
> org.apache.spark.sql.catalyst.parser.AstBuilder.org$apache$spark$sql$catalyst$parser$AstBuilder$$visitSparkDataType(AstBuilde



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24703) Unable to multiply calender interval with long/int

2018-07-03 Thread Daniel Mateus Pires (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-24703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531363#comment-16531363
 ] 

Daniel Mateus Pires commented on SPARK-24703:
-

was able to reproduce in 2.4.0

> Unable to multiply calender interval with long/int
> --
>
> Key: SPARK-24703
> URL: https://issues.apache.org/jira/browse/SPARK-24703
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.1
>Reporter: Priyanka Garg
>Priority: Major
>
> When i am trying to multiply calender interval with long/int , I am getting 
> below error. The same syntax is supported in Postgres.
>  spark.sql("select 3 *  interval '1' day").show()
> org.apache.spark.sql.AnalysisException: cannot resolve '(3 * interval 1 
> days)' due to data type mismatch: differing types in '(3 * interval 1 days)' 
> (int and calendarinterval).; line 1 pos 7;
> 'Project [unresolvedalias((3 * interval 1 days), None)]
> +- OneRowRelation
>  
>   at 
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:93)
>   at 
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:85)
>   at 
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:289)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org