[jira] [Updated] (SPARK-18340) Inconsistent error messages in launching scripts and hanging in sparkr script for wrong options

2016-11-07 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18340: - Description: It seems there are some problems with handling wrong options as below:

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591417#comment-15591417 ] Hyukjin Kwon commented on SPARK-17916: -- FWIW, the codes I ran in Spark (equalvalent with the last

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591408#comment-15591408 ] Hyukjin Kwon commented on SPARK-17916: -- [~falaki] I just have thought about this more but I started

[jira] [Commented] (SPARK-14428) [SQL] Allow more flexibility when parsing dates and timestamps in json datasources

2016-10-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15577994#comment-15577994 ] Hyukjin Kwon commented on SPARK-14428: -- For 1. I guess this was fixed in

[jira] [Created] (SPARK-17955) Use the same read path in DataFrameReader.jdbc and DataFrameReader.format("jdbc")

2016-10-15 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-17955: Summary: Use the same read path in DataFrameReader.jdbc and DataFrameReader.format("jdbc") Key: SPARK-17955 URL: https://issues.apache.org/jira/browse/SPARK-17955

[jira] [Commented] (SPARK-18064) Spark SQL can't load default config file

2016-10-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15600483#comment-15600483 ] Hyukjin Kwon commented on SPARK-18064: -- Could you please fill up the JIRA description? I would like

[jira] [Created] (SPARK-17989) Check ascendingOrder type in sort_array function ahead

2016-10-18 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-17989: Summary: Check ascendingOrder type in sort_array function ahead Key: SPARK-17989 URL: https://issues.apache.org/jira/browse/SPARK-17989 Project: Spark Issue

[jira] [Comment Edited] (SPARK-18068) Spark SQL doesn't parse some ISO 8601 formatted dates

2016-10-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604377#comment-15604377 ] Hyukjin Kwon edited comment on SPARK-18068 at 10/25/16 7:01 AM:

[jira] [Commented] (SPARK-18068) Spark SQL doesn't parse some ISO 8601 formatted dates

2016-10-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604377#comment-15604377 ] Hyukjin Kwon commented on SPARK-18068: -- [~stephane.maa...@gmail.com] As a workaround, we might be

[jira] [Commented] (SPARK-18068) Spark SQL doesn't parse some ISO 8601 formatted dates

2016-10-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604437#comment-15604437 ] Hyukjin Kwon commented on SPARK-18068: -- Oh, it is documented only in codes -

[jira] [Comment Edited] (SPARK-18068) Spark SQL doesn't parse some ISO 8601 formatted dates

2016-10-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604437#comment-15604437 ] Hyukjin Kwon edited comment on SPARK-18068 at 10/25/16 9:48 AM: Oh, it is

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593271#comment-15593271 ] Hyukjin Kwon commented on SPARK-17916: -- Oh, yes sure. I just thought the root problem is to

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-10-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593300#comment-15593300 ] Hyukjin Kwon commented on SPARK-17916: -- Could I please ask what you think? cc [~felixcheung] > CSV

[jira] [Updated] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce

2016-10-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-14393: - Affects Version/s: 2.0.0 2.0.1 > monotonicallyIncreasingId not

[jira] [Created] (SPARK-18423) ReceiverTracker should close checkpoint dir when stopped even if it was not started

2016-11-12 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18423: Summary: ReceiverTracker should close checkpoint dir when stopped even if it was not started Key: SPARK-18423 URL: https://issues.apache.org/jira/browse/SPARK-18423

[jira] [Updated] (SPARK-18423) ReceiverTracker should close checkpoint dir when stopped even if it was not started

2016-11-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18423: - Description: {code} Running org.apache.spark.streaming.JavaAPISuite Tests run: 53, Failures: 0,

[jira] [Updated] (SPARK-18422) Fix wholeTextFiles test to pass on Windows in JavaAPISuite

2016-11-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18422: - Component/s: Spark Core > Fix wholeTextFiles test to pass on Windows in JavaAPISuite >

[jira] [Created] (SPARK-18422) Fix wholeTextFiles test to pass on Windows in JavaAPISuite

2016-11-12 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18422: Summary: Fix wholeTextFiles test to pass on Windows in JavaAPISuite Key: SPARK-18422 URL: https://issues.apache.org/jira/browse/SPARK-18422 Project: Spark

[jira] [Created] (SPARK-18437) Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations

2016-11-14 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18437: Summary: Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations Key: SPARK-18437 URL: https://issues.apache.org/jira/browse/SPARK-18437

[jira] [Commented] (SPARK-18437) Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations

2016-11-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663968#comment-15663968 ] Hyukjin Kwon commented on SPARK-18437: -- Hi [~felixcheung], what do you think about R documentation?

[jira] [Updated] (SPARK-18437) Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations

2016-11-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18437: - Description: It seems in Scala/Java, - {{Note:}} - {{NOTE:}} - {{Note that}} -

[jira] [Updated] (SPARK-18437) Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations

2016-11-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18437: - Description: It seems in Scala/Java, - {{Note:}} - {{NOTE:}} - {{Note that}} -

[jira] [Updated] (SPARK-18437) Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations

2016-11-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18437: - Description: It seems in Scala/Java, - {{Note:}} - {{NOTE:}} - {{Note that}} -

[jira] [Commented] (SPARK-18437) Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations

2016-11-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664057#comment-15664057 ] Hyukjin Kwon commented on SPARK-18437: -- Ah, thank you. Let me try to do this separately for Scala,

[jira] [Updated] (SPARK-18437) Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations

2016-11-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18437: - Description: It seems in Scala/Java, - {{Note:}} - {{NOTE:}} - {{Note that}} -

[jira] [Commented] (SPARK-18437) Inconsistent mark-down for `Note:` across Scala/Java/R/Python in API documentations

2016-11-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665547#comment-15665547 ] Hyukjin Kwon commented on SPARK-18437: -- Sounds good and safe. Thanks! > Inconsistent mark-down for

[jira] [Created] (SPARK-18610) greatest/leatest fails to run with string aginst date/timestamp

2016-11-28 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18610: Summary: greatest/leatest fails to run with string aginst date/timestamp Key: SPARK-18610 URL: https://issues.apache.org/jira/browse/SPARK-18610 Project: Spark

[jira] [Commented] (SPARK-18610) greatest/leatest fails to run with string aginst date/timestamp

2016-11-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702032#comment-15702032 ] Hyukjin Kwon commented on SPARK-18610: -- Ah, I believe it is a bit complicated more than I know but

[jira] [Commented] (SPARK-17212) TypeCoercion support widening conversion between DateType and TimestampType

2016-11-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699731#comment-15699731 ] Hyukjin Kwon commented on SPARK-17212: -- Oh, [~hvanhovell], I might be wrong but I guess it is

[jira] [Commented] (SPARK-17212) TypeCoercion support widening conversion between DateType and TimestampType

2016-11-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15699751#comment-15699751 ] Hyukjin Kwon commented on SPARK-17212: -- Oh wait, I am sorry. I rushed the comments. The problem

[jira] [Updated] (SPARK-18610) greatest/leatest fails to run with string aginst date/timestamp

2016-11-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18610: - Description: It seems Spark SQL fails to implicitly cast (or detect widen type) from string

[jira] [Commented] (SPARK-18610) greatest/leatest fails to run with string aginst date/timestamp

2016-11-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702047#comment-15702047 ] Hyukjin Kwon commented on SPARK-18610: -- So, I understood (by myself) that we might need to

[jira] [Commented] (SPARK-18610) greatest/leatest fails to run with string aginst date/timestamp

2016-11-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702083#comment-15702083 ] Hyukjin Kwon commented on SPARK-18610: -- Hi [~hvanhovell] and [~cloud_fan], After rethinking, I think

[jira] [Commented] (SPARK-18610) greatest/leatest fails to run with string aginst date/timestamp

2016-11-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702057#comment-15702057 ] Hyukjin Kwon commented on SPARK-18610: -- Actually, in case of MySQL, it seems converting the datatype

[jira] [Created] (SPARK-18615) Switch to multi-line doc to avoid a genjavadoc bug for backticks

2016-11-28 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18615: Summary: Switch to multi-line doc to avoid a genjavadoc bug for backticks Key: SPARK-18615 URL: https://issues.apache.org/jira/browse/SPARK-18615 Project: Spark

[jira] [Commented] (SPARK-12307) ParquetFormat options should be exposed through the DataFrameReader/Writer options API

2016-11-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704121#comment-15704121 ] Hyukjin Kwon commented on SPARK-12307: -- Hi [~holdenk], I guess we are able to set this via

[jira] [Commented] (SPARK-3359) `sbt/sbt unidoc` doesn't work with Java 8

2016-11-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15673792#comment-15673792 ] Hyukjin Kwon commented on SPARK-3359: - Hi [~srowen], It seems I met this issue while trying to build

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

2016-11-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675441#comment-15675441 ] Hyukjin Kwon commented on SPARK-18352: -- Hi [~rxin], I think it seems this can be simply done after

[jira] [Commented] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

2016-11-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675466#comment-15675466 ] Hyukjin Kwon commented on SPARK-18352: -- Ah, you meant producing each row while parsing the whole

[jira] [Comment Edited] (SPARK-18352) Parse normal, multi-line JSON files (not just JSON Lines)

2016-11-17 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675441#comment-15675441 ] Hyukjin Kwon edited comment on SPARK-18352 at 11/18/16 1:53 AM: Hi

[jira] [Commented] (SPARK-18393) DataFrame pivot output column names should respect aliases

2016-11-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676315#comment-15676315 ] Hyukjin Kwon commented on SPARK-18393: -- Hi [~ekhliang], It seems it was fixed in SPARK-17458 {code}

[jira] [Created] (SPARK-18514) Fix `Note:`/`NOTE:`/`Note that` across R API documentation

2016-11-20 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18514: Summary: Fix `Note:`/`NOTE:`/`Note that` across R API documentation Key: SPARK-18514 URL: https://issues.apache.org/jira/browse/SPARK-18514 Project: Spark

[jira] [Commented] (SPARK-18437) Inconsistent mark-down for `Note:`/`NOTE:`/`Note that` across Scala/Java/R/Python in API documentations

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668697#comment-15668697 ] Hyukjin Kwon commented on SPARK-18437: -- Yup, I don't mind but let us be very sure on the changes to

[jira] [Created] (SPARK-18445) Fix `Note:` across Scala API documentation

2016-11-15 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18445: Summary: Fix `Note:` across Scala API documentation Key: SPARK-18445 URL: https://issues.apache.org/jira/browse/SPARK-18445 Project: Spark Issue Type:

[jira] [Updated] (SPARK-18445) Fix `Note:`/`NOTE:`/`Note that` across Scala/Java API documentation

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18445: - Description: Same issue with the parent but the scope is within Scala/Java API documentation.

[jira] [Updated] (SPARK-18437) Inconsistent mark-down for `Note:`/`NOTE:`/`Note that` across Scala/Java/R/Python in API documentations

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18437: - Summary: Inconsistent mark-down for `Note:`/`NOTE:`/`Note that` across Scala/Java/R/Python in

[jira] [Updated] (SPARK-18445) Fix `Note:`/`NOTE:`/`Note that` across Scala API documentation

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18445: - Summary: Fix `Note:`/`NOTE:`/`Note that` across Scala API documentation (was: Fix `Note:`

[jira] [Updated] (SPARK-18445) Fix `Note:`/`NOTE:`/`Note that` across Scala/Java API documentation

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18445: - Summary: Fix `Note:`/`NOTE:`/`Note that` across Scala/Java API documentation (was: Fix

[jira] [Commented] (SPARK-18457) ORC and other columnar formats using HiveShim read all columns when doing a simple count

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669369#comment-15669369 ] Hyukjin Kwon commented on SPARK-18457: -- Ah, I see {{HiveShim.appendReadColumns}} is not called when

[jira] [Comment Edited] (SPARK-18457) ORC and other columnar formats using HiveShim read all columns when doing a simple count

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669150#comment-15669150 ] Hyukjin Kwon edited comment on SPARK-18457 at 11/16/16 2:39 AM: Hm, don't

[jira] [Comment Edited] (SPARK-18457) ORC and other columnar formats using HiveShim read all columns when doing a simple count

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669150#comment-15669150 ] Hyukjin Kwon edited comment on SPARK-18457 at 11/16/16 2:39 AM: Hm, don't

[jira] [Commented] (SPARK-18457) ORC and other columnar formats using HiveShim read all columns when doing a simple count

2016-11-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669150#comment-15669150 ] Hyukjin Kwon commented on SPARK-18457: -- Hm, don't we push down no column when performing `count()`?

[jira] [Commented] (SPARK-17591) Fix/investigate the failure of tests in Scala On Windows

2016-11-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15682578#comment-15682578 ] Hyukjin Kwon commented on SPARK-17591: -- I will close this when I am able to proceed further the

[jira] [Updated] (SPARK-18514) Fix `Note:`/`NOTE:`/`Note that` across R API documentation

2016-11-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18514: - Priority: Trivial (was: Minor) > Fix `Note:`/`NOTE:`/`Note that` across R API documentation >

[jira] [Commented] (SPARK-18420) Fix the compile errors caused by checkstyle

2016-11-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662832#comment-15662832 ] Hyukjin Kwon commented on SPARK-18420: -- It seems {code} [ERROR]

[jira] [Closed] (SPARK-18190) Fix R version to not the latest in AppVeyor

2016-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-18190. Resolution: Not A Problem This is not a problem as R 3.3.2 for Windows is now released and it

[jira] [Created] (SPARK-18190) Fix R version to not the latest in AppVeyor

2016-10-31 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18190: Summary: Fix R version to not the latest in AppVeyor Key: SPARK-18190 URL: https://issues.apache.org/jira/browse/SPARK-18190 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-13913) DataFrame.withColumn fails when trying to replace existing column with dot in name

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13913. -- Resolution: Cannot Reproduce I am resolving this as Cannot Reproduce as I can't against

[jira] [Commented] (SPARK-14008) Cleanup/Extend the Vectorized Parquet Reader

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628400#comment-15628400 ] Hyukjin Kwon commented on SPARK-14008: -- Hi [~sameerag], I just wonder if this might be supposed to

[jira] [Created] (SPARK-18179) Throws analysis exception with a proper message for unsupported argument types in reflect/java_method function

2016-10-31 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18179: Summary: Throws analysis exception with a proper message for unsupported argument types in reflect/java_method function Key: SPARK-18179 URL:

[jira] [Created] (SPARK-18246) Throws an exception before execution for unsupported types in Json, CSV and text functionailities

2016-11-03 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-18246: Summary: Throws an exception before execution for unsupported types in Json, CSV and text functionailities Key: SPARK-18246 URL: https://issues.apache.org/jira/browse/SPARK-18246

[jira] [Updated] (SPARK-18246) Throws an exception before execution for unsupported types in Json, CSV and text functionailities

2016-11-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18246: - Description: * Case 1 {code} val rdd = spark.sparkContext.parallelize(1 to 100).map(i =>

[jira] [Updated] (SPARK-18246) Throws an exception before execution for unsupported types in Json, CSV and text functionailities

2016-11-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-18246: - Description: * Case 1 - {{read.json(rdd)}} {code} val rdd = spark.sparkContext.parallelize(1 to

[jira] [Comment Edited] (SPARK-14195) Error: org.apache.spark.sql.AnalysisException: cannot recognize input near 'select' 'MAC' 'from' in expression specification; line 1 pos 16 (state=,code=0)

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631071#comment-15631071 ] Hyukjin Kwon edited comment on SPARK-14195 at 11/3/16 12:52 AM: I am

[jira] [Closed] (SPARK-14196) Error: org.apache.spark.sql.AnalysisException: cannot resolve 'query_alias_fix_conflicts_0._c0'; (state=,code=0)

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-14196. Resolution: Cannot Reproduce I am closing this as Cannot Reproduce as I can't against the current

[jira] [Closed] (SPARK-14195) Error: org.apache.spark.sql.AnalysisException: cannot recognize input near 'select' 'MAC' 'from' in expression specification; line 1 pos 16 (state=,code=0)

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-14195. Resolution: Cannot Reproduce I am closing this as Cannot Reproduce as I can't against the current

[jira] [Closed] (SPARK-14197) Error: org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to name on unresolved object, tree: unresolvedalias(if ((imei#33365 = 1AA10007)) imei#3336

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-14197. Resolution: Cannot Reproduce I am closing this as Cannot Reproduce as I can't reproduce this

[jira] [Commented] (SPARK-14319) Speed up group-by aggregates

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631091#comment-15631091 ] Hyukjin Kwon commented on SPARK-14319: -- Hey [~sameerag], I guess we might be able to take an action

[jira] [Commented] (SPARK-14922) Alter Table Drop Partition Using Predicate-based Partition Spec

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631137#comment-15631137 ] Hyukjin Kwon commented on SPARK-14922: -- Hi [~smilegator], would this be supposed to be closed as a

[jira] [Closed] (SPARK-15174) DataFrame does not have correct number of rows after dropDuplicates

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-15174. Resolution: Cannot Reproduce I can't reproduce this in the current master. So, I am going to mark

[jira] [Resolved] (SPARK-16210) DataFrame.drop(colName) fails if another column has a period in its name

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-16210. -- Resolution: Cannot Reproduce I can't reproduce this too {code} scala> val rdd =

[jira] [Resolved] (SPARK-5151) Parquet Predicate Pushdown Does Not Work with Nested Structures.

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-5151. - Resolution: Duplicate I am going to resolve this as a duplicate. The reason why I resolve this

[jira] [Commented] (SPARK-17727) PySpark SQL arrays are not immutable, .remove and .pop cause issues

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631228#comment-15631228 ] Hyukjin Kwon commented on SPARK-17727: -- Hi [~srowen], would this JIRA be resolved with an action? >

[jira] [Commented] (SPARK-14443) parse_url() does not escape query parameters

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631103#comment-15631103 ] Hyukjin Kwon commented on SPARK-14443: -- I just tried this too. It seems we can do this via manually

[jira] [Commented] (SPARK-14840) Cannot drop a table which has the name starting with 'or'

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631134#comment-15631134 ] Hyukjin Kwon commented on SPARK-14840: -- Please revoke my action if this is inappropriate. > Cannot

[jira] [Resolved] (SPARK-14840) Cannot drop a table which has the name starting with 'or'

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14840. -- Resolution: Duplicate I can't reproduce this against the current master. {code} scala>

[jira] [Resolved] (SPARK-17049) LAG function fails when selecting all columns

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17049. -- Resolution: Cannot Reproduce I am resolving this JIRA as Cannot Reproduce as described in the

[jira] [Commented] (SPARK-18269) NumberFormatException when reading csv for a nullable column

2016-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636055#comment-15636055 ] Hyukjin Kwon commented on SPARK-18269: -- Could you try set {{nullValue}} to {{null}}? >

[jira] [Comment Edited] (SPARK-18269) NumberFormatException when reading csv for a nullable column

2016-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636055#comment-15636055 ] Hyukjin Kwon edited comment on SPARK-18269 at 11/4/16 11:18 AM: Could you

[jira] [Commented] (SPARK-18269) NumberFormatException when reading csv for a nullable column

2016-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636069#comment-15636069 ] Hyukjin Kwon commented on SPARK-18269: -- BTW, if I remember this correctly, it seems always forcing

[jira] [Commented] (SPARK-18269) NumberFormatException when reading csv for a nullable column

2016-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636353#comment-15636353 ] Hyukjin Kwon commented on SPARK-18269: -- Ah, yes. it seems a bug. I can reproduce this. Thanks for

[jira] [Commented] (SPARK-18270) Users schema with non-nullable properties is overidden with true

2016-11-04 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15636046#comment-15636046 ] Hyukjin Kwon commented on SPARK-18270: -- I am pretty sure this is a subset of SPARK-16472 and I

[jira] [Commented] (SPARK-12677) Lazy file discovery for parquet

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629114#comment-15629114 ] Hyukjin Kwon commented on SPARK-12677: -- Ah, I see. IMHO, it might not be an issue as long as it

[jira] [Commented] (SPARK-10840) SparkSQL doesn't work well with JSON

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627929#comment-15627929 ] Hyukjin Kwon commented on SPARK-10840: -- I guess this is a duplicate of SPARK-17969 (I am not too

[jira] [Commented] (SPARK-4549) Support BigInt -> Decimal in convertToCatalyst in SparkSQL

2016-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627695#comment-15627695 ] Hyukjin Kwon commented on SPARK-4549: - Could we maybe close this for now if no one can't explain when

[jira] [Comment Edited] (SPARK-4549) Support BigInt -> Decimal in convertToCatalyst in SparkSQL

2016-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627695#comment-15627695 ] Hyukjin Kwon edited comment on SPARK-4549 at 11/2/16 4:41 AM: -- Could we maybe

[jira] [Commented] (SPARK-6825) Data sources implementation to support `sequenceFile`

2016-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627714#comment-15627714 ] Hyukjin Kwon commented on SPARK-6825: - Hi [~shivaram], do we still need this? If so, I can maybe give

[jira] [Commented] (SPARK-7755) MetadataCache.refresh does not take into account _SUCCESS

2016-11-01 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627754#comment-15627754 ] Hyukjin Kwon commented on SPARK-7755: - Hi [~liancheng], didn't we remove

[jira] [Comment Edited] (SPARK-12677) Lazy file discovery for parquet

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15629114#comment-15629114 ] Hyukjin Kwon edited comment on SPARK-12677 at 11/2/16 2:23 PM: --- Ah, I see.

[jira] [Closed] (SPARK-12483) Data Frame as() does not work in Java

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-12483. Resolution: Invalid I am closing this as Invalid because {{Dataset.as}} API refers the alias for

[jira] [Resolved] (SPARK-12354) Implement unhandledFilter interface

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12354. -- Resolution: Done marking this Umbrella as "Done" as it is done. > Implement unhandledFilter

[jira] [Resolved] (SPARK-12355) Implement unhandledFilter interface for Parquet

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12355. -- Resolution: Not A Problem >From the past discussion, I guess it seems we want to use

[jira] [Resolved] (SPARK-11868) wrong results returned from dataframe create from Rows without consistent schma on pyspark

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-11868. -- Resolution: Cannot Reproduce I still can't repdocue this in the master. It seems this is

[jira] [Commented] (SPARK-12677) Lazy file discovery for parquet

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628301#comment-15628301 ] Hyukjin Kwon commented on SPARK-12677: -- I am pretty sure you meant reading schema when executing

[jira] [Resolved] (SPARK-13342) Cannot run INSERT statements in Spark

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13342. -- Resolution: Cannot Reproduce I am resolving this as Cannot Reproduce as I can't reproduce.

[jira] [Commented] (SPARK-13417) SQL subquery support

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628324#comment-15628324 ] Hyukjin Kwon commented on SPARK-13417: -- Hi [~rxin], I just happened to look at this JIRA. I just

[jira] [Resolved] (SPARK-13480) Regression with percentile() + function in GROUP BY

2016-11-02 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13480. -- Resolution: Cannot Reproduce I am resolving this issue as Cannot Reproduce because it seems

[jira] [Resolved] (SPARK-17656) Decide on the variant of @scala.annotation.varargs and use consistently

2016-10-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17656. -- Resolution: Fixed This was fixed in the PR together. > Decide on the variant of

[jira] [Commented] (SPARK-15577) Java can't import DataFrame type alias

2016-10-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564103#comment-15564103 ] Hyukjin Kwon commented on SPARK-15577: -- Cool, then I guess we might be able to take an action on the

[jira] [Comment Edited] (SPARK-15577) Java can't import DataFrame type alias

2016-10-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564115#comment-15564115 ] Hyukjin Kwon edited comment on SPARK-15577 at 10/11/16 1:32 AM: WDYT? -

<    5   6   7   8   9   10   11   12   13   14   >