[jira] [Closed] (SPARK-11995) Partitioning Parquet by DateType

[ https://issues.apache.org/jira/browse/SPARK-11995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-11995. Resolution: Duplicate As SPARK-17388 has a PR, I will mark this as duplicate. Please reopen if you

[jira] [Commented] (SPARK-12916) Support Row.fromSeq and Row.toSeq methods in pyspark

[ https://issues.apache.org/jira/browse/SPARK-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557964#comment-15557964 ] Hyukjin Kwon commented on SPARK-12916: -- I am pretty sure we don't need this but I would like to cc

[jira] [Commented] (SPARK-5446) Parquet column pruning should work for Map and Struct

[ https://issues.apache.org/jira/browse/SPARK-5446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557659#comment-15557659 ] Hyukjin Kwon commented on SPARK-5446: - How about SPARK-4502? It seems both issues referring column

[jira] [Commented] (SPARK-6587) Inferring schema for case class hierarchy fails with mysterious message

[ https://issues.apache.org/jira/browse/SPARK-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557724#comment-15557724 ] Hyukjin Kwon commented on SPARK-6587: - >From the discussion above, I am closing this as Not a Problem.

[jira] [Closed] (SPARK-6587) Inferring schema for case class hierarchy fails with mysterious message

[ https://issues.apache.org/jira/browse/SPARK-6587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-6587. --- Resolution: Not A Problem > Inferring schema for case class hierarchy fails with mysterious message >

[jira] [Closed] (SPARK-6982) Data Frame and Spark SQL should allow filtering on key portion of incremental parquet files

[ https://issues.apache.org/jira/browse/SPARK-6982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-6982. --- Resolution: Cannot Reproduce It seems working find now. Please reopen this if you still face this

[jira] [Resolved] (SPARK-8144) For PySpark SQL, automatically convert values provided in readwriter options to string

[ https://issues.apache.org/jira/browse/SPARK-8144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8144. - Resolution: Fixed This was fixed in https://github.com/apache/spark/pull/7926 > For PySpark SQL,

[jira] [Commented] (SPARK-8842) Spark SQL - Insert into table Issue

[ https://issues.apache.org/jira/browse/SPARK-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557779#comment-15557779 ] Hyukjin Kwon commented on SPARK-8842: - Then, would you set Affects Version/s correctly too? > Spark

[jira] [Commented] (SPARK-5511) [SQL] Possible optimisations for predicate pushdowns from Spark SQL to Parquet

[ https://issues.apache.org/jira/browse/SPARK-5511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557667#comment-15557667 ] Hyukjin Kwon commented on SPARK-5511: - 1. I agree it needs a change on Spark. 2. We already supported

[jira] [Commented] (SPARK-5936) Automatically convert a StructType to a MapType when the number of fields exceed a threshold.

[ https://issues.apache.org/jira/browse/SPARK-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557709#comment-15557709 ] Hyukjin Kwon commented on SPARK-5936: - Hi [~yhuai], I just happened to look at this JIRA. Would this

[jira] [Commented] (SPARK-6678) select count(DISTINCT C_UID) from parquetdir may be can optimize

[ https://issues.apache.org/jira/browse/SPARK-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557729#comment-15557729 ] Hyukjin Kwon commented on SPARK-6678: - Hi [~cnstar9988] Is this issue describing the slow performance

[jira] [Commented] (SPARK-7101) Spark SQL should support java.sql.Time

[ https://issues.apache.org/jira/browse/SPARK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557745#comment-15557745 ] Hyukjin Kwon commented on SPARK-7101: - I guess we have {{TimestampType}}, {{DateType}} and

[jira] [Closed] (SPARK-7301) Issue with duplicated fields in interpreted json schemas

[ https://issues.apache.org/jira/browse/SPARK-7301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-7301. --- Resolution: Duplicate It seems a duplicate of SPARK-13493 Since that JIRA has a PR open, I am closing

[jira] [Commented] (SPARK-8403) Pruner partition won't effective when partition field and fieldSchema exist in sql predicate

[ https://issues.apache.org/jira/browse/SPARK-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557771#comment-15557771 ] Hyukjin Kwon commented on SPARK-8403: - Hm, IIRC, this was fixed somewhere. Could you please try to

[jira] [Closed] (SPARK-9405) approximateCountDiscting does not work with GroupBy

[ https://issues.apache.org/jira/browse/SPARK-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-9405. --- Resolution: Not A Problem > approximateCountDiscting does not work with GroupBy >

[jira] [Commented] (SPARK-9405) approximateCountDiscting does not work with GroupBy

[ https://issues.apache.org/jira/browse/SPARK-9405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557790#comment-15557790 ] Hyukjin Kwon commented on SPARK-9405: - I am closing this from the comment above. >

[jira] [Commented] (SPARK-9783) Use SqlNewHadoopRDD in JSONRelation to eliminate extra refresh() call

[ https://issues.apache.org/jira/browse/SPARK-9783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557810#comment-15557810 ] Hyukjin Kwon commented on SPARK-9783: - [~lian cheng][~yhuai] I believe this is closable. Do you mind

[jira] [Commented] (SPARK-10501) support UUID as an atomic type

[ https://issues.apache.org/jira/browse/SPARK-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557820#comment-15557820 ] Hyukjin Kwon commented on SPARK-10501: -- Can we use {{monotonically_increasing_id}} instead? >

[jira] [Commented] (SPARK-10681) DateTimeUtils needs a method to parse string to SQL's timestamp value

[ https://issues.apache.org/jira/browse/SPARK-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557832#comment-15557832 ] Hyukjin Kwon commented on SPARK-10681: -- Hm, doesn't {{DateTimeUtils.stringToTimestamp}} handle this?

[jira] [Resolved] (SPARK-13699) Spark SQL drops the table in "overwrite" mode while writing into table

[ https://issues.apache.org/jira/browse/SPARK-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13699. -- Resolution: Fixed We introduced {{truncate}} option in

[jira] [Commented] (SPARK-13699) Spark SQL drops the table in "overwrite" mode while writing into table

[ https://issues.apache.org/jira/browse/SPARK-13699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15557996#comment-15557996 ] Hyukjin Kwon commented on SPARK-13699: -- Thank you. I will try to follow it. > Spark SQL drops the

[jira] [Commented] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce

[ https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558029#comment-15558029 ] Hyukjin Kwon commented on SPARK-14393: -- Still happens in the current master/2.0 {code} scala>

[jira] [Commented] (SPARK-14621) add oracle hint optimizer

[ https://issues.apache.org/jira/browse/SPARK-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558034#comment-15558034 ] Hyukjin Kwon commented on SPARK-14621: -- +1 for closing this. > add oracle hint optimizer >

[jira] [Resolved] (SPARK-14766) Attribute reference mismatch with Dataset filter + mapPartitions

[ https://issues.apache.org/jira/browse/SPARK-14766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14766. -- Resolution: Cannot Reproduce {code} scala> Seq((1, 1)).toDS().filter(_._1 != 0).mapPartitions

[jira] [Commented] (SPARK-15577) Java can't import DataFrame type alias

[ https://issues.apache.org/jira/browse/SPARK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558049#comment-15558049 ] Hyukjin Kwon commented on SPARK-15577: -- If I were a naive user from Spark 1.x, I think I would

[jira] [Comment Edited] (SPARK-16386) SQLContext and HiveContext parse a query string differently

[ https://issues.apache.org/jira/browse/SPARK-16386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558062#comment-15558062 ] Hyukjin Kwon edited comment on SPARK-16386 at 10/8/16 2:29 PM: --- I can;t

[jira] [Resolved] (SPARK-16386) SQLContext and HiveContext parse a query string differently

[ https://issues.apache.org/jira/browse/SPARK-16386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-16386. -- Resolution: Cannot Reproduce I can;t reproduce the problematic case 2 {code}

[jira] [Commented] (SPARK-16720) Loading CSV file with 2k+ columns fails during attribute resolution on action

[ https://issues.apache.org/jira/browse/SPARK-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15558067#comment-15558067 ] Hyukjin Kwon commented on SPARK-16720: -- Hi [~holdenk] Do you mind if I ask to close this? I tried to

[jira] [Updated] (SPARK-17840) Add some pointers for wiki/CONTRIBUTING.md in README.md and some warnings in PULL_REQUEST_TEMPLATE

[ https://issues.apache.org/jira/browse/SPARK-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17840: - Description: It seems it'd better to give best efforts for describing the guides for creating

[jira] [Created] (SPARK-17840) Add some pointers for wiki/CONTRIBUTING.md in README.md and some warnings in PULL_REQUEST_TEMPLATE

Hyukjin Kwon created SPARK-17840: Summary: Add some pointers for wiki/CONTRIBUTING.md in README.md and some warnings in PULL_REQUEST_TEMPLATE Key: SPARK-17840 URL:

[jira] [Updated] (SPARK-17840) Add some pointers for wiki/CONTRIBUTING.md in README.md and some warnings in PULL_REQUEST_TEMPLATE

[ https://issues.apache.org/jira/browse/SPARK-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17840: - Description: It seems it'd better to give best efforts for describing the guides for creating

[jira] [Commented] (SPARK-17537) Improve performance for reading parquet schema

[ https://issues.apache.org/jira/browse/SPARK-17537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492475#comment-15492475 ] Hyukjin Kwon commented on SPARK-17537: -- This should be a duplicate of SPARK-17071. > Improve

[jira] [Created] (SPARK-17583) Remove unused rowSeparator variable and set auto-expanding buffer as default for maxCharsPerColumn option in CSV

Hyukjin Kwon created SPARK-17583: Summary: Remove unused rowSeparator variable and set auto-expanding buffer as default for maxCharsPerColumn option in CSV Key: SPARK-17583 URL:

[jira] [Created] (SPARK-17586) Use Static member not via instance reference

Hyukjin Kwon created SPARK-17586: Summary: Use Static member not via instance reference Key: SPARK-17586 URL: https://issues.apache.org/jira/browse/SPARK-17586 Project: Spark Issue Type:

[jira] [Updated] (SPARK-17586) Use Static member not via instance reference

[ https://issues.apache.org/jira/browse/SPARK-17586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17586: - Component/s: (was: Spark Core) Build > Use Static member not via instance

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501666#comment-15501666 ] Hyukjin Kwon commented on SPARK-17545: -- I left my comment as I am confirmed that the fixed version

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501682#comment-15501682 ] Hyukjin Kwon commented on SPARK-17545: -- I think maybe, we should just change this to

[jira] [Created] (SPARK-17591) Fix/investigate the failure of tests in Scala On Windows

Hyukjin Kwon created SPARK-17591: Summary: Fix/investigate the failure of tests in Scala On Windows Key: SPARK-17591 URL: https://issues.apache.org/jira/browse/SPARK-17591 Project: Spark

[jira] [Commented] (SPARK-17591) Fix/investigate the failure of tests in Scala On Windows

[ https://issues.apache.org/jira/browse/SPARK-17591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502252#comment-15502252 ] Hyukjin Kwon commented on SPARK-17591: -- Please cc me if any of you is interested in and would like

[jira] [Updated] (SPARK-17591) Fix/investigate the failure of tests in Scala On Windows

[ https://issues.apache.org/jira/browse/SPARK-17591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17591: - Description: {code} Tests run: 90, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.53 sec

[jira] [Updated] (SPARK-17591) Fix/investigate the failure of tests in Scala On Windows

[ https://issues.apache.org/jira/browse/SPARK-17591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17591: - Description: {code} Tests run: 90, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 17.53 sec

[jira] [Commented] (SPARK-17597) HiveContext cannot create a table named sort

[ https://issues.apache.org/jira/browse/SPARK-17597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505316#comment-15505316 ] Hyukjin Kwon commented on SPARK-17597: -- BTW, I can't reproduce this against master. It seems this

[jira] [Comment Edited] (SPARK-17597) HiveContext cannot create a table named sort

[ https://issues.apache.org/jira/browse/SPARK-17597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505316#comment-15505316 ] Hyukjin Kwon edited comment on SPARK-17597 at 9/20/16 2:15 AM: --- BTW, I

[jira] [Commented] (SPARK-17597) HiveContext cannot create a table named sort

[ https://issues.apache.org/jira/browse/SPARK-17597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505340#comment-15505340 ] Hyukjin Kwon commented on SPARK-17597: -- [~saif.a.ellafi] Do you mind if I ask to confirm that it

[jira] [Comment Edited] (SPARK-17597) HiveContext cannot create a table named sort

[ https://issues.apache.org/jira/browse/SPARK-17597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505316#comment-15505316 ] Hyukjin Kwon edited comment on SPARK-17597 at 9/20/16 2:16 AM: --- BTW, I

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514853#comment-15514853 ] Hyukjin Kwon commented on SPARK-17636: -- Yea, Spark implementation currently does not push down

[jira] [Comment Edited] (SPARK-17636) Parquet filter push down doesn't handle struct fields

[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1551#comment-1551 ] Hyukjin Kwon edited comment on SPARK-17636 at 9/23/16 6:23 AM: ---

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1551#comment-1551 ] Hyukjin Kwon commented on SPARK-17636: -- Confirmation from committer -

[jira] [Comment Edited] (SPARK-17621) Accumulator value is doubled when using DataFrame.orderBy()

[ https://issues.apache.org/jira/browse/SPARK-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516190#comment-15516190 ] Hyukjin Kwon edited comment on SPARK-17621 at 9/23/16 11:35 AM: Hi

[jira] [Commented] (SPARK-17621) Accumulator value is doubled when using DataFrame.orderBy()

[ https://issues.apache.org/jira/browse/SPARK-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516190#comment-15516190 ] Hyukjin Kwon commented on SPARK-17621: -- Hi [~srowen] and [~sreelalsl], I just happened to look at

[jira] [Commented] (SPARK-17621) Accumulator value is doubled when using DataFrame.orderBy()

[ https://issues.apache.org/jira/browse/SPARK-17621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516307#comment-15516307 ] Hyukjin Kwon commented on SPARK-17621: -- Cool! I just found it is dependent on job stages and

[jira] [Commented] (SPARK-17619) To add support for pattern matching in ArrayContains Expression.

[ https://issues.apache.org/jira/browse/SPARK-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15516905#comment-15516905 ] Hyukjin Kwon commented on SPARK-17619: -- Do you mind if I ask you mean {{array_contains}} function?

[jira] [Commented] (SPARK-14536) NPE in JDBCRDD when array column contains nulls (postgresql)

[ https://issues.apache.org/jira/browse/SPARK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511837#comment-15511837 ] Hyukjin Kwon commented on SPARK-14536: -- I see. I rushed to read this and didn't noticed that this is

[jira] [Comment Edited] (SPARK-8500) Support for array types in JDBCRDD

[ https://issues.apache.org/jira/browse/SPARK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511840#comment-15511840 ] Hyukjin Kwon edited comment on SPARK-8500 at 9/22/16 2:04 AM: -- I am leaving a

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509110#comment-15509110 ] Hyukjin Kwon commented on SPARK-17545: -- Yes. This is because we introduced {{FastDateFormat}} there

[jira] [Commented] (SPARK-8500) Support for array types in JDBCRDD

[ https://issues.apache.org/jira/browse/SPARK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511840#comment-15511840 ] Hyukjin Kwon commented on SPARK-8500: - I am leaving a note that PostgreSQL is supporting {{ArrayType}}

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508261#comment-15508261 ] Hyukjin Kwon commented on SPARK-17545: -- [~srowen] and [~nbeyer]. Just a gentle reminder that this

[jira] [Comment Edited] (SPARK-17615) Getting "java.lang.RuntimeException: Distinct columns cannot exist in Aggregate "

[ https://issues.apache.org/jira/browse/SPARK-17615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508376#comment-15508376 ] Hyukjin Kwon edited comment on SPARK-17615 at 9/21/16 1:36 AM: --- It seems

[jira] [Commented] (SPARK-17615) Getting "java.lang.RuntimeException: Distinct columns cannot exist in Aggregate "

[ https://issues.apache.org/jira/browse/SPARK-17615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508376#comment-15508376 ] Hyukjin Kwon commented on SPARK-17615: -- It seems something went wrong when creating a JIRA. It seems

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508264#comment-15508264 ] Hyukjin Kwon commented on SPARK-17545: -- Fixing JIRA : SPARK-16216 Fixing PR :

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496579#comment-15496579 ] Hyukjin Kwon commented on SPARK-17545: -- Thank you both for your feedback. I am okay with fixing this

[jira] [Commented] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15499175#comment-15499175 ] Hyukjin Kwon commented on SPARK-17545: -- I took another look. This might be solved by changing the

[jira] [Comment Edited] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15499175#comment-15499175 ] Hyukjin Kwon edited comment on SPARK-17545 at 9/17/16 3:06 PM: --- I took

[jira] [Comment Edited] (SPARK-17545) Spark SQL Catalyst doesn't handle ISO 8601 date without colon in offset

[ https://issues.apache.org/jira/browse/SPARK-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15499175#comment-15499175 ] Hyukjin Kwon edited comment on SPARK-17545 at 9/17/16 3:07 PM: --- I took

[jira] [Commented] (SPARK-17601) SparkSQL vectorization cannot handle schema evolution for parquet tables when parquet files use Int whereas DataFrame uses Long

[ https://issues.apache.org/jira/browse/SPARK-17601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505066#comment-15505066 ] Hyukjin Kwon commented on SPARK-17601: -- We might have to avoid to open multiple related issues. I

[jira] [Commented] (SPARK-16896) Loading csv with duplicate column names

[ https://issues.apache.org/jira/browse/SPARK-16896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426262#comment-15426262 ] Hyukjin Kwon commented on SPARK-16896: -- Hi [~nlauchande], are you currently working on this? >

[jira] [Commented] (SPARK-17264) DataStreamWriter does not support "json" format

[ https://issues.apache.org/jira/browse/SPARK-17264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15442475#comment-15442475 ] Hyukjin Kwon commented on SPARK-17264: -- Is this a duplicate of SPARK-15472? > DataStreamWriter does

[jira] [Commented] (SPARK-17174) Provide support for Timestamp type Column in add_months function to return HH:mm:ss

[ https://issues.apache.org/jira/browse/SPARK-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431965#comment-15431965 ] Hyukjin Kwon commented on SPARK-17174: -- I just took a look for others as references. It seems Hive

[jira] [Commented] (SPARK-17195) Dealing with JDBC column nullability when it is not reliable

[ https://issues.apache.org/jira/browse/SPARK-17195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432103#comment-15432103 ] Hyukjin Kwon commented on SPARK-17195: -- I think if Teradata JDBC gives an incorrect schema (about

[jira] [Commented] (SPARK-17200) Automate building and testing on Windows

[ https://issues.apache.org/jira/browse/SPARK-17200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433739#comment-15433739 ] Hyukjin Kwon commented on SPARK-17200: -- Thank you so much! > Automate building and testing on

[jira] [Commented] (SPARK-17174) Provide support for Timestamp type Column in add_months function to return HH:mm:ss

[ https://issues.apache.org/jira/browse/SPARK-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434195#comment-15434195 ] Hyukjin Kwon commented on SPARK-17174: -- [~cloud_fan] I realised that {{Ruturns date ...}} might

[jira] [Commented] (SPARK-17200) Automate building and testing on Windows

[ https://issues.apache.org/jira/browse/SPARK-17200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432504#comment-15432504 ] Hyukjin Kwon commented on SPARK-17200: -- Could I please ask your opinion [~srowen]? I know you are an

[jira] [Commented] (SPARK-17200) Automate building and testing on Windows

[ https://issues.apache.org/jira/browse/SPARK-17200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432521#comment-15432521 ] Hyukjin Kwon commented on SPARK-17200: -- Maybe, I will try to do this and I guess this will be done

[jira] [Updated] (SPARK-17200) Automate building and testing on Windows

[ https://issues.apache.org/jira/browse/SPARK-17200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17200: - Description: It seems there is no automated tests on Windows (I am not sure this is being done

[jira] [Commented] (SPARK-17200) Automate building and testing on Windows

[ https://issues.apache.org/jira/browse/SPARK-17200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432508#comment-15432508 ] Hyukjin Kwon commented on SPARK-17200: -- cc [~felixcheung] and [~shivaram] who might be interested in

[jira] [Created] (SPARK-17200) Automate building and testing on Windows

Hyukjin Kwon created SPARK-17200: Summary: Automate building and testing on Windows Key: SPARK-17200 URL: https://issues.apache.org/jira/browse/SPARK-17200 Project: Spark Issue Type: Test

[jira] [Commented] (SPARK-17619) To add support for pattern matching in ArrayContains Expression.

[ https://issues.apache.org/jira/browse/SPARK-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531866#comment-15531866 ] Hyukjin Kwon commented on SPARK-17619: -- Hm, is this a regression and it was supported before? Could

[jira] [Commented] (SPARK-17619) To add support for pattern matching in ArrayContains Expression.

[ https://issues.apache.org/jira/browse/SPARK-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531778#comment-15531778 ] Hyukjin Kwon commented on SPARK-17619: -- I am worried if this make sense to support the pattern in

[jira] [Created] (SPARK-17719) Unify and tie up options in a single place in JDBC datasource API

Hyukjin Kwon created SPARK-17719: Summary: Unify and tie up options in a single place in JDBC datasource API Key: SPARK-17719 URL: https://issues.apache.org/jira/browse/SPARK-17719 Project: Spark

[jira] [Commented] (SPARK-17719) Unify and tie up options in a single place in JDBC datasource API

[ https://issues.apache.org/jira/browse/SPARK-17719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531745#comment-15531745 ] Hyukjin Kwon commented on SPARK-17719: -- I am working on this. > Unify and tie up options in a

[jira] [Commented] (SPARK-17694) convert DataFrame to DataSet should check columns match

[ https://issues.apache.org/jira/browse/SPARK-17694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528295#comment-15528295 ] Hyukjin Kwon commented on SPARK-17694: -- FWIW, this reminds me of SPARK-15918 > convert DataFrame to

[jira] [Commented] (SPARK-17695) Deserialization error when using DataFrameReader.json on JSON line that contains an empty JSON object

[ https://issues.apache.org/jira/browse/SPARK-17695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527900#comment-15527900 ] Hyukjin Kwon commented on SPARK-17695: -- Related with SPARK-8093. It seems empty struct type is not

[jira] [Updated] (SPARK-17665) SparkR supports options in other types consistently other APIs

[ https://issues.apache.org/jira/browse/SPARK-17665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17665: - Affects Version/s: 2.0.0 > SparkR supports options in other types consistently other APIs >

[jira] [Commented] (SPARK-17764) to_json function for parsing Structs to json Strings

[ https://issues.apache.org/jira/browse/SPARK-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15543869#comment-15543869 ] Hyukjin Kwon commented on SPARK-17764: -- Thank you for your advice. > to_json function for parsing

[jira] [Created] (SPARK-17763) JacksonParser silently parses null as 0 when the field is not nullable

Hyukjin Kwon created SPARK-17763: Summary: JacksonParser silently parses null as 0 when the field is not nullable Key: SPARK-17763 URL: https://issues.apache.org/jira/browse/SPARK-17763 Project:

[jira] [Commented] (SPARK-17764) to_json function for parsing Structs to json Strings

[ https://issues.apache.org/jira/browse/SPARK-17764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15541842#comment-15541842 ] Hyukjin Kwon commented on SPARK-17764: -- Hi [~marmbrus], if no one is working one this and it seems

[jira] [Created] (SPARK-17764) to_json function for parsing Structs to json Strings

Hyukjin Kwon created SPARK-17764: Summary: to_json function for parsing Structs to json Strings Key: SPARK-17764 URL: https://issues.apache.org/jira/browse/SPARK-17764 Project: Spark Issue

[jira] [Commented] (SPARK-17665) SparkR does not support options in other types consistently other APIs

[ https://issues.apache.org/jira/browse/SPARK-17665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15522331#comment-15522331 ] Hyukjin Kwon commented on SPARK-17665: -- You are so fast! How about numeric types? Actually, I was

[jira] [Updated] (SPARK-17665) SparkR does not support options in other types consistently other APIs

[ https://issues.apache.org/jira/browse/SPARK-17665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17665: - Summary: SparkR does not support options in other types consistently other APIs (was: SparkR

[jira] [Created] (SPARK-17674) Warnings from SparkR tests being ignored without redirecting to errors

Hyukjin Kwon created SPARK-17674: Summary: Warnings from SparkR tests being ignored without redirecting to errors Key: SPARK-17674 URL: https://issues.apache.org/jira/browse/SPARK-17674 Project:

[jira] [Updated] (SPARK-17674) Warnings from SparkR tests being ignored without redirecting to errors

[ https://issues.apache.org/jira/browse/SPARK-17674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17674: - Description: For example, _currently_ we are having warnings as below:

[jira] [Commented] (SPARK-17840) Add some pointers for wiki/CONTRIBUTING.md in README.md and some warnings in PULL_REQUEST_TEMPLATE

[ https://issues.apache.org/jira/browse/SPARK-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560968#comment-15560968 ] Hyukjin Kwon commented on SPARK-17840: -- I do agree and thank you for your prompt response. Maybe, I

[jira] [Commented] (SPARK-14393) monotonicallyIncreasingId not monotonically increasing with downstream coalesce

[ https://issues.apache.org/jira/browse/SPARK-14393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15560991#comment-15560991 ] Hyukjin Kwon commented on SPARK-14393: -- Thanks for pinging me!, in my personal opinion, we should

[jira] [Commented] (SPARK-15577) Java can't import DataFrame type alias

[ https://issues.apache.org/jira/browse/SPARK-15577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564115#comment-15564115 ] Hyukjin Kwon commented on SPARK-15577: -- WDYT - [~holdenk] > Java can't import DataFrame type alias

[jira] [Commented] (SPARK-17967) Support for list or other types as an option for datasources

[ https://issues.apache.org/jira/browse/SPARK-17967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647747#comment-15647747 ] Hyukjin Kwon commented on SPARK-17967: -- Thanks [~rxin], I made a patch for this locally but I guess

[jira] [Commented] (SPARK-17967) Support for list or other types as an option for datasources

[ https://issues.apache.org/jira/browse/SPARK-17967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15647746#comment-15647746 ] Hyukjin Kwon commented on SPARK-17967: -- Thanks [~rxin], I made a patch for this locally but I guess

[jira] [Issue Comment Deleted] (SPARK-17967) Support for list or other types as an option for datasources

[ https://issues.apache.org/jira/browse/SPARK-17967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-17967: - Comment: was deleted (was: Thanks [~rxin], I made a patch for this locally but I guess you might

[jira] [Commented] (SPARK-18359) No option in read csv for other decimal delimiter than dot

[ https://issues.apache.org/jira/browse/SPARK-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649352#comment-15649352 ] Hyukjin Kwon commented on SPARK-18359: -- I guess that should be locale specific. I guess we recently

[jira] [Created] (SPARK-18295) Match up to_json to from_json in null safety

Hyukjin Kwon created SPARK-18295: Summary: Match up to_json to from_json in null safety Key: SPARK-18295 URL: https://issues.apache.org/jira/browse/SPARK-18295 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-18340) Inconsistent error messages in launching scripts and hanging in sparkr script for wrong options

Hyukjin Kwon created SPARK-18340: Summary: Inconsistent error messages in launching scripts and hanging in sparkr script for wrong options Key: SPARK-18340 URL: https://issues.apache.org/jira/browse/SPARK-18340

<    4   5   6   7   8   9   10   11   12   13   >