[jira] [Commented] (SPARK-15245) stream API throws an exception with an incorrect message when the path is not a direcotry

[ https://issues.apache.org/jira/browse/SPARK-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277696#comment-15277696 ] Hyukjin Kwon commented on SPARK-15245: -- Oh, the main reason is, I thought {{'basePath' must be a

[jira] [Created] (SPARK-15245) stream API throws an exception with an incorrect message when the path is not a direcotry

Hyukjin Kwon created SPARK-15245: Summary: stream API throws an exception with an incorrect message when the path is not a direcotry Key: SPARK-15245 URL: https://issues.apache.org/jira/browse/SPARK-15245

[jira] [Commented] (SPARK-15245) stream API throws an exception with an incorrect message when the path is not a direcotry

[ https://issues.apache.org/jira/browse/SPARK-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277700#comment-15277700 ] Hyukjin Kwon commented on SPARK-15245: -- Thank you so much. Let me close my PR. > stream API throws

[jira] [Commented] (SPARK-15245) stream API throws an exception with an incorrect message when the path is not a direcotry

[ https://issues.apache.org/jira/browse/SPARK-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277716#comment-15277716 ] Hyukjin Kwon commented on SPARK-15245: -- (BTW, as you might already know, the reason why I thought

[jira] [Commented] (SPARK-15250) Remove deprecated json API in DataFrameReader

[ https://issues.apache.org/jira/browse/SPARK-15250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277907#comment-15277907 ] Hyukjin Kwon commented on SPARK-15250: -- [~rxin] I searched and track down the related PRs but could

[jira] [Created] (SPARK-15250) Remove deprecated json API in DataFrameReader

Hyukjin Kwon created SPARK-15250: Summary: Remove deprecated json API in DataFrameReader Key: SPARK-15250 URL: https://issues.apache.org/jira/browse/SPARK-15250 Project: Spark Issue Type:

[jira] [Commented] (SPARK-15245) stream API throws an exception with an incorrect message when the path is not a direcotry

[ https://issues.apache.org/jira/browse/SPARK-15245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15277979#comment-15277979 ] Hyukjin Kwon commented on SPARK-15245: -- Sorry for leaving comments again and again but I think this

[jira] [Created] (SPARK-14787) Upgrade Joda-Time library from 2.9 to 2.9.3

Hyukjin Kwon created SPARK-14787: Summary: Upgrade Joda-Time library from 2.9 to 2.9.3 Key: SPARK-14787 URL: https://issues.apache.org/jira/browse/SPARK-14787 Project: Spark Issue Type:

[jira] [Commented] (SPARK-14525) DataFrameWriter's save method should delegate to jdbc for jdbc datasource

[ https://issues.apache.org/jira/browse/SPARK-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251339#comment-15251339 ] Hyukjin Kwon commented on SPARK-14525: -- I think it might be great if we can cc who knows this part

[jira] [Comment Edited] (SPARK-14525) DataFrameWriter's save method should delegate to jdbc for jdbc datasource

[ https://issues.apache.org/jira/browse/SPARK-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251339#comment-15251339 ] Hyukjin Kwon edited comment on SPARK-14525 at 4/21/16 6:17 AM: --- I think it

[jira] [Comment Edited] (SPARK-14525) DataFrameWriter's save method should delegate to jdbc for jdbc datasource

[ https://issues.apache.org/jira/browse/SPARK-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247448#comment-15247448 ] Hyukjin Kwon edited comment on SPARK-14525 at 4/19/16 9:52 AM: --- Shouldn't

[jira] [Commented] (SPARK-14525) DataFrameWriter's save method should delegate to jdbc for jdbc datasource

[ https://issues.apache.org/jira/browse/SPARK-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247448#comment-15247448 ] Hyukjin Kwon commented on SPARK-14525: -- Shouldn't we then deprecate the support for

[jira] [Commented] (SPARK-14728) Add a rule to block the use of getOrElse(null) which can simply be orNull.

[ https://issues.apache.org/jira/browse/SPARK-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247451#comment-15247451 ] Hyukjin Kwon commented on SPARK-14728: -- [~rxin] Do you think it is okay to add a rule? If you are

[jira] [Commented] (SPARK-14728) Add a rule to block the use of getOrElse(null) which can simply be orNull.

[ https://issues.apache.org/jira/browse/SPARK-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247459#comment-15247459 ] Hyukjin Kwon commented on SPARK-14728: -- Oh, sorry, I noticed not all classes having {{getOrElse}}

[jira] [Created] (SPARK-14728) Add a rule to block the use of getOrElse(null) which can simply be orNull.

Hyukjin Kwon created SPARK-14728: Summary: Add a rule to block the use of getOrElse(null) which can simply be orNull. Key: SPARK-14728 URL: https://issues.apache.org/jira/browse/SPARK-14728 Project:

[jira] [Closed] (SPARK-14728) Add a rule to block the use of getOrElse(null) which can simply be orNull.

[ https://issues.apache.org/jira/browse/SPARK-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon closed SPARK-14728. Resolution: Invalid > Add a rule to block the use of getOrElse(null) which can simply be orNull. >

[jira] [Commented] (SPARK-14726) Support for sampling when inferring schema in CSV data source

[ https://issues.apache.org/jira/browse/SPARK-14726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15247378#comment-15247378 ] Hyukjin Kwon commented on SPARK-14726: -- This is currently not supported. I can work on this but I

[jira] [Created] (SPARK-14800) Dealing with null as a value in options for each internal data source

Hyukjin Kwon created SPARK-14800: Summary: Dealing with null as a value in options for each internal data source Key: SPARK-14800 URL: https://issues.apache.org/jira/browse/SPARK-14800 Project: Spark

[jira] [Commented] (SPARK-14800) Dealing with null as a value in options for each internal data source

[ https://issues.apache.org/jira/browse/SPARK-14800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251509#comment-15251509 ] Hyukjin Kwon commented on SPARK-14800: -- Let me work on this if

[jira] [Updated] (SPARK-15212) CSV file reader when read file with first line schema do not filter blank in schema column name

[ https://issues.apache.org/jira/browse/SPARK-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-15212: - Priority: Minor (was: Major) > CSV file reader when read file with first line schema do not

[jira] [Updated] (SPARK-15212) CSV file reader when read file with first line schema do not filter blank in schema column name

[ https://issues.apache.org/jira/browse/SPARK-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-15212: - Affects Version/s: (was: 1.6.2) (was: 1.6.1) > CSV file reader

[jira] [Commented] (SPARK-15212) CSV file reader when read file with first line schema do not filter blank in schema column name

[ https://issues.apache.org/jira/browse/SPARK-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275801#comment-15275801 ] Hyukjin Kwon commented on SPARK-15212: -- Since Spark does not have CSV data source as a internal data

[jira] [Commented] (SPARK-15212) CSV file reader when read file with first line schema do not filter blank in schema column name

[ https://issues.apache.org/jira/browse/SPARK-15212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15275802#comment-15275802 ] Hyukjin Kwon commented on SPARK-15212: -- Since there is a workaroud with `withColumnRenamed()` which

[jira] [Commented] (SPARK-16720) Loading CSV file with 2k+ columns fails during attribute resolution on action

[ https://issues.apache.org/jira/browse/SPARK-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393069#comment-15393069 ] Hyukjin Kwon commented on SPARK-16720: -- Hi [~holdenk], this part seems familiar to me. Do you mind

[jira] [Commented] (SPARK-14536) NPE in JDBCRDD when array column contains nulls (postgresql)

[ https://issues.apache.org/jira/browse/SPARK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393161#comment-15393161 ] Hyukjin Kwon commented on SPARK-14536: -- Hi [~jeremyrsmith], are you working on this? > NPE in

[jira] [Commented] (SPARK-16720) Loading CSV file with 2k+ columns fails during attribute resolution on action

[ https://issues.apache.org/jira/browse/SPARK-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393182#comment-15393182 ] Hyukjin Kwon commented on SPARK-16720: -- [~holdenk] I just tried to reproduce this with the codes

[jira] [Commented] (SPARK-8500) Support for array types in JDBCRDD

[ https://issues.apache.org/jira/browse/SPARK-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393282#comment-15393282 ] Hyukjin Kwon commented on SPARK-8500: - FYI, this is still happening in 2.0.0 and current master. The

[jira] [Commented] (SPARK-14536) NPE in JDBCRDD when array column contains nulls (postgresql)

[ https://issues.apache.org/jira/browse/SPARK-14536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393283#comment-15393283 ] Hyukjin Kwon commented on SPARK-14536: -- FYI, It seems {{ArrayType}} is not supported for JDBC for

[jira] [Commented] (SPARK-16698) json parsing regression - "." in keys

[ https://issues.apache.org/jira/browse/SPARK-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391231#comment-15391231 ] Hyukjin Kwon commented on SPARK-16698: -- It seems it does not work for all `FileFormat` data sources.

[jira] [Commented] (SPARK-16698) json parsing regression - "." in keys

[ https://issues.apache.org/jira/browse/SPARK-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15391217#comment-15391217 ] Hyukjin Kwon commented on SPARK-16698: -- FYI, this does not happen when it is read from json RDD. Let

[jira] [Comment Edited] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389273#comment-15389273 ] Hyukjin Kwon edited comment on SPARK-16646 at 7/23/16 1:53 AM: --- It seems

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15389145#comment-15389145 ] Hyukjin Kwon commented on SPARK-16646: -- My pleasure! Let me look into this and will bring some

[jira] [Created] (SPARK-16674) Avoid per-record type dispatch in JDBC when reading

Hyukjin Kwon created SPARK-16674: Summary: Avoid per-record type dispatch in JDBC when reading Key: SPARK-16674 URL: https://issues.apache.org/jira/browse/SPARK-16674 Project: Spark Issue

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388916#comment-15388916 ] Hyukjin Kwon commented on SPARK-16646: -- I see! please let me leave my thought as well just in case

[jira] [Created] (SPARK-16675) Avoid per-record type dispatch in JDBC when writing

Hyukjin Kwon created SPARK-16675: Summary: Avoid per-record type dispatch in JDBC when writing Key: SPARK-16675 URL: https://issues.apache.org/jira/browse/SPARK-16675 Project: Spark Issue

[jira] [Commented] (SPARK-16675) Avoid per-record type dispatch in JDBC when writing

[ https://issues.apache.org/jira/browse/SPARK-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388829#comment-15388829 ] Hyukjin Kwon commented on SPARK-16675: -- I will work on this. > Avoid per-record type dispatch in

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398650#comment-15398650 ] Hyukjin Kwon commented on SPARK-16646: -- Sure, I will close the PR for meanwhile. Then please update

[jira] [Commented] (SPARK-16777) Parquet schema converter depends on deprecated APIs

[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398781#comment-15398781 ] Hyukjin Kwon commented on SPARK-16777: -- Please let me leave a note because I actually took a look

[jira] [Commented] (SPARK-16776) Fix Kafka deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-16776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398786#comment-15398786 ] Hyukjin Kwon commented on SPARK-16776: -- Please let me leave the logs here just in case. {code}

[jira] [Comment Edited] (SPARK-16777) Parquet schema converter depends on deprecated APIs

[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398781#comment-15398781 ] Hyukjin Kwon edited comment on SPARK-16777 at 7/29/16 6:29 AM: --- Please let

[jira] [Commented] (SPARK-16777) Parquet schema converter depends on deprecated APIs

[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398872#comment-15398872 ] Hyukjin Kwon commented on SPARK-16777: -- Sure! > Parquet schema converter depends on deprecated APIs

[jira] [Commented] (SPARK-16777) Parquet schema converter depends on deprecated APIs

[ https://issues.apache.org/jira/browse/SPARK-16777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398866#comment-15398866 ] Hyukjin Kwon commented on SPARK-16777: -- Ah, I see, we can just use the builder to greate

[jira] [Commented] (SPARK-16776) Fix Kafka deprecation warnings

[ https://issues.apache.org/jira/browse/SPARK-16776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15400433#comment-15400433 ] Hyukjin Kwon commented on SPARK-16776: -- Oh, I meant to just leave the warnings but let me do this

[jira] [Commented] (SPARK-16610) When writing ORC files, orc.compress should not be overridden if users do not set "compression" in the options

[ https://issues.apache.org/jira/browse/SPARK-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405071#comment-15405071 ] Hyukjin Kwon commented on SPARK-16610: -- One thought is, we might have to document that we don't

[jira] [Created] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

Hyukjin Kwon created SPARK-16842: Summary: Concern about disallowing user-given schema for Parquet and ORC Key: SPARK-16842 URL: https://issues.apache.org/jira/browse/SPARK-16842 Project: Spark

[jira] [Commented] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403308#comment-15403308 ] Hyukjin Kwon commented on SPARK-16842: -- Thanks for your feedback. Yea, but I think it might not be

[jira] [Issue Comment Deleted] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-16842: - Comment: was deleted (was: Thanks for your feedback. Yea, but I think it might not be very heavy

[jira] [Commented] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403309#comment-15403309 ] Hyukjin Kwon commented on SPARK-16842: -- Thanks for your feedback. Yea, but I think it might not be

[jira] [Comment Edited] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403308#comment-15403308 ] Hyukjin Kwon edited comment on SPARK-16842 at 8/2/16 3:56 AM: -- Thanks for

[jira] [Comment Edited] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403308#comment-15403308 ] Hyukjin Kwon edited comment on SPARK-16842 at 8/2/16 3:56 AM: -- Thanks for

[jira] [Commented] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403315#comment-15403315 ] Hyukjin Kwon commented on SPARK-16842: -- If we don't support schema compatibility but should support

[jira] [Created] (SPARK-16847) Do not read Parquet corrupt statstics on binary

Hyukjin Kwon created SPARK-16847: Summary: Do not read Parquet corrupt statstics on binary Key: SPARK-16847 URL: https://issues.apache.org/jira/browse/SPARK-16847 Project: Spark Issue Type:

[jira] [Updated] (SPARK-16847) Do not read Parquet corrupt statstics on binary via VectorizedReader when it is corrupt

[ https://issues.apache.org/jira/browse/SPARK-16847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-16847: - Summary: Do not read Parquet corrupt statstics on binary via VectorizedReader when it is corrupt

[jira] [Updated] (SPARK-16847) Prevent to potentially read corrupt statstics on binary in Parquet via VectorizedReader

[ https://issues.apache.org/jira/browse/SPARK-16847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-16847: - Summary: Prevent to potentially read corrupt statstics on binary in Parquet via VectorizedReader

[jira] [Created] (SPARK-16848) Make jdbc() and read.format("jdbc") consistently throwing exception for user-specified schema

Hyukjin Kwon created SPARK-16848: Summary: Make jdbc() and read.format("jdbc") consistently throwing exception for user-specified schema Key: SPARK-16848 URL: https://issues.apache.org/jira/browse/SPARK-16848

[jira] [Commented] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403139#comment-15403139 ] Hyukjin Kwon commented on SPARK-16842: -- Let me cc [~liancheng], [~smilegator] [~dongjoon] and

[jira] [Updated] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-16842: - Description: If my understanding is correct, If the user-given schema is different with the

[jira] [Updated] (SPARK-16842) Concern about disallowing user-given schema for Parquet and ORC

[ https://issues.apache.org/jira/browse/SPARK-16842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-16842: - Description: If my understanding is correct, If the user-given schema is different with the

[jira] [Commented] (SPARK-16869) Wrong projection when join on columns with the same name which are derived from the same dataframe

[ https://issues.apache.org/jira/browse/SPARK-16869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405661#comment-15405661 ] Hyukjin Kwon commented on SPARK-16869: -- I can't reproduce this with the codes you provided as blow:

[jira] [Commented] (SPARK-16869) Wrong projection when join on columns with the same name which are derived from the same dataframe

[ https://issues.apache.org/jira/browse/SPARK-16869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405680#comment-15405680 ] Hyukjin Kwon commented on SPARK-16869: -- Hm Could you please double check? I think I can't find

[jira] [Commented] (SPARK-16610) When writing ORC files, orc.compress should not be overridden if users do not set "compression" in the options

[ https://issues.apache.org/jira/browse/SPARK-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407043#comment-15407043 ] Hyukjin Kwon commented on SPARK-16610: -- Actually, my initial proposal of the above was including

[jira] [Commented] (SPARK-16877) Add a rule for preventing use Java's Override annotation

[ https://issues.apache.org/jira/browse/SPARK-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405872#comment-15405872 ] Hyukjin Kwon commented on SPARK-16877: -- Also we might have to add

[jira] [Commented] (SPARK-15382) monotonicallyIncreasingId doesn't work when data is upsampled

[ https://issues.apache.org/jira/browse/SPARK-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374148#comment-15374148 ] Hyukjin Kwon commented on SPARK-15382: -- I was just looking into this but don't mind if you open a PR

[jira] [Comment Edited] (SPARK-16515) [SPARK][SQL] transformation script got failure for python script

[ https://issues.apache.org/jira/browse/SPARK-16515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374210#comment-15374210 ] Hyukjin Kwon edited comment on SPARK-16515 at 7/13/16 2:35 AM: --- Hi, do you

[jira] [Commented] (SPARK-16515) [SPARK][SQL] transformation script got failure for python script

[ https://issues.apache.org/jira/browse/SPARK-16515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374214#comment-15374214 ] Hyukjin Kwon commented on SPARK-16515: -- Oh, I will refer the PR. Thanks. > [SPARK][SQL]

[jira] [Commented] (SPARK-16515) [SPARK][SQL] transformation script got failure for python script

[ https://issues.apache.org/jira/browse/SPARK-16515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374210#comment-15374210 ] Hyukjin Kwon commented on SPARK-16515: -- Hi, do you mind if I ask narrow it down or some codes with

[jira] [Updated] (SPARK-16512) No way to load CSV data without dropping whole rows when some of data is not matched with given schema

[ https://issues.apache.org/jira/browse/SPARK-16512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-16512: - Description: Currently, there is no way to read CSV data without dropping whole rows when some

[jira] [Created] (SPARK-16512) No way to load CSV data without dropping whole rows when some of data is not matched with given schema

Hyukjin Kwon created SPARK-16512: Summary: No way to load CSV data without dropping whole rows when some of data is not matched with given schema Key: SPARK-16512 URL:

[jira] [Commented] (SPARK-16512) No way to load CSV data without dropping whole rows when some of data is not matched with given schema

[ https://issues.apache.org/jira/browse/SPARK-16512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374150#comment-15374150 ] Hyukjin Kwon commented on SPARK-16512: -- I will work on this as soon as

[jira] [Commented] (SPARK-16518) Schema Compatibility of Parquet Data Source

[ https://issues.apache.org/jira/browse/SPARK-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376097#comment-15376097 ] Hyukjin Kwon commented on SPARK-16518: -- [~dongjoon], [~smilegator] Thank you! then, I will try to

[jira] [Commented] (SPARK-15382) monotonicallyIncreasingId doesn't work when data is upsampled

[ https://issues.apache.org/jira/browse/SPARK-15382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15372673#comment-15372673 ] Hyukjin Kwon commented on SPARK-15382: -- This also happends in master branch (2.1.0) here is the

[jira] [Comment Edited] (SPARK-16518) Schema Compatibility of Parquet Data Source

[ https://issues.apache.org/jira/browse/SPARK-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374518#comment-15374518 ] Hyukjin Kwon edited comment on SPARK-16518 at 7/13/16 7:39 AM: --- Cool, this

[jira] [Commented] (SPARK-16518) Schema Compatibility of Parquet Data Source

[ https://issues.apache.org/jira/browse/SPARK-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374518#comment-15374518 ] Hyukjin Kwon commented on SPARK-16518: -- Cool, this was exactly what I have been wondering and

[jira] [Created] (SPARK-16516) Support for pushing down filters for decimal and timestamp types in ORC

Hyukjin Kwon created SPARK-16516: Summary: Support for pushing down filters for decimal and timestamp types in ORC Key: SPARK-16516 URL: https://issues.apache.org/jira/browse/SPARK-16516 Project:

[jira] [Commented] (SPARK-16518) Schema Compatibility of Parquet Data Source

[ https://issues.apache.org/jira/browse/SPARK-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374523#comment-15374523 ] Hyukjin Kwon commented on SPARK-16518: -- FYI, For case 2, I guess this is related with

[jira] [Comment Edited] (SPARK-16518) Schema Compatibility of Parquet Data Source

[ https://issues.apache.org/jira/browse/SPARK-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374523#comment-15374523 ] Hyukjin Kwon edited comment on SPARK-16518 at 7/13/16 7:33 AM: --- FYI, For

[jira] [Commented] (SPARK-16544) Support for conversion from compatible schema for Parquet data source when data types are not matched

[ https://issues.apache.org/jira/browse/SPARK-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15376717#comment-15376717 ] Hyukjin Kwon commented on SPARK-16544: -- I am working on this. > Support for conversion from

[jira] [Created] (SPARK-16544) Support for conversion from compatible schema for Parquet data source when data types are not matched

Hyukjin Kwon created SPARK-16544: Summary: Support for conversion from compatible schema for Parquet data source when data types are not matched Key: SPARK-16544 URL:

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387067#comment-15387067 ] Hyukjin Kwon commented on SPARK-16646: -- Maybe I had to ask if you are working on this. I will close

[jira] [Commented] (SPARK-16216) CSV data source does not write date and timestamp correctly

[ https://issues.apache.org/jira/browse/SPARK-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383664#comment-15383664 ] Hyukjin Kwon commented on SPARK-16216: -- JSON writes date and timestamp as a string format as below:

[jira] [Comment Edited] (SPARK-16216) CSV data source does not write date and timestamp correctly

[ https://issues.apache.org/jira/browse/SPARK-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383680#comment-15383680 ] Hyukjin Kwon edited comment on SPARK-16216 at 7/19/16 6:56 AM: --- I can close

[jira] [Comment Edited] (SPARK-16216) CSV data source does not write date and timestamp correctly

[ https://issues.apache.org/jira/browse/SPARK-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383664#comment-15383664 ] Hyukjin Kwon edited comment on SPARK-16216 at 7/19/16 6:43 AM: --- JSON writes

[jira] [Comment Edited] (SPARK-16216) CSV data source does not write date and timestamp correctly

[ https://issues.apache.org/jira/browse/SPARK-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383664#comment-15383664 ] Hyukjin Kwon edited comment on SPARK-16216 at 7/19/16 6:44 AM: --- JSON writes

[jira] [Commented] (SPARK-16216) CSV data source does not write date and timestamp correctly

[ https://issues.apache.org/jira/browse/SPARK-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383680#comment-15383680 ] Hyukjin Kwon commented on SPARK-16216: -- I can close existing PR and open another to match both

[jira] [Comment Edited] (SPARK-16610) When writing ORC files, orc.compress should not be overridden if users do not set "compression" in the options

[ https://issues.apache.org/jira/browse/SPARK-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383471#comment-15383471 ] Hyukjin Kwon edited comment on SPARK-16610 at 7/19/16 2:02 AM: --- Thank you

[jira] [Commented] (SPARK-16610) When writing ORC files, orc.compress should not be overridden if users do not set "compression" in the options

[ https://issues.apache.org/jira/browse/SPARK-16610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383471#comment-15383471 ] Hyukjin Kwon commented on SPARK-16610: -- Yea, actually I pointed this out in the PR. Check out this

[jira] [Commented] (SPARK-16216) CSV data source does not write date and timestamp correctly

[ https://issues.apache.org/jira/browse/SPARK-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384107#comment-15384107 ] Hyukjin Kwon commented on SPARK-16216: -- Please let me leave my thought here just in case it is

[jira] [Commented] (SPARK-16636) Missing documentation for CalendarIntervalType type in sql-programming-guide.md

[ https://issues.apache.org/jira/browse/SPARK-16636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385127#comment-15385127 ] Hyukjin Kwon commented on SPARK-16636: -- I will cc you, [~rxin] [~cloud_fan] just in case. > Missing

[jira] [Created] (SPARK-16636) Missing documentation for CalendarIntervalType type in sql-programming-guide.md

Hyukjin Kwon created SPARK-16636: Summary: Missing documentation for CalendarIntervalType type in sql-programming-guide.md Key: SPARK-16636 URL: https://issues.apache.org/jira/browse/SPARK-16636

[jira] [Commented] (SPARK-16646) LEAST doesn't accept numeric arguments with different data types

[ https://issues.apache.org/jira/browse/SPARK-16646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385760#comment-15385760 ] Hyukjin Kwon commented on SPARK-16646: -- Just FYI, I just tried to reproduce this. In 1.6.x, in

[jira] [Commented] (SPARK-16588) Missed API fix for a function name mismatched between FunctionRegistry and functions.scala

[ https://issues.apache.org/jira/browse/SPARK-16588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380685#comment-15380685 ] Hyukjin Kwon commented on SPARK-16588: -- [~rxin] Should we deprecate or remove this for 2.0.0/2.1.0?

[jira] [Created] (SPARK-16588) Missed API fix for a function name mismatched between FunctionRegistry and functions.scala

Hyukjin Kwon created SPARK-16588: Summary: Missed API fix for a function name mismatched between FunctionRegistry and functions.scala Key: SPARK-16588 URL: https://issues.apache.org/jira/browse/SPARK-16588

[jira] [Commented] (SPARK-16597) DataFrame DateType is written as an int(Days since epoch) by csv writer

[ https://issues.apache.org/jira/browse/SPARK-16597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381696#comment-15381696 ] Hyukjin Kwon commented on SPARK-16597: -- I guess this is a duplicated of SPARK-16216. > DataFrame

[jira] [Commented] (SPARK-15393) Writing empty Dataframes doesn't save any _metadata files

[ https://issues.apache.org/jira/browse/SPARK-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380292#comment-15380292 ] Hyukjin Kwon commented on SPARK-15393: -- Yes please, could you please close this one? And actually,

[jira] [Created] (SPARK-16562) Do not allow downcast in INT32 based types for non-vectorized Parquet reader

Hyukjin Kwon created SPARK-16562: Summary: Do not allow downcast in INT32 based types for non-vectorized Parquet reader Key: SPARK-16562 URL: https://issues.apache.org/jira/browse/SPARK-16562

[jira] [Commented] (SPARK-16250) Can't use escapeQuotes option in DataFrameWriter.csv()

[ https://issues.apache.org/jira/browse/SPARK-16250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15354496#comment-15354496 ] Hyukjin Kwon commented on SPARK-16250: -- This was fixed in SPARK-16259 together. > Can't use

[jira] [Commented] (SPARK-16371) IS NOT NULL clause gives false for nested not empty column

[ https://issues.apache.org/jira/browse/SPARK-16371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362303#comment-15362303 ] Hyukjin Kwon commented on SPARK-16371: -- I somehow modified your codes and tested this by myself with

[jira] [Commented] (SPARK-16371) IS NOT NULL clause gives false for nested not empty column

[ https://issues.apache.org/jira/browse/SPARK-16371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15362299#comment-15362299 ] Hyukjin Kwon commented on SPARK-16371: -- Could you narrow down the case with a simpler code and data?

[jira] [Commented] (SPARK-16371) IS NOT NULL clause gives false for nested not empty column

[ https://issues.apache.org/jira/browse/SPARK-16371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363548#comment-15363548 ] Hyukjin Kwon commented on SPARK-16371: -- [~maver1ck] [~proflin] I could reproduce this. I will try to

[jira] [Comment Edited] (SPARK-16371) IS NOT NULL clause gives false for nested not empty column

[ https://issues.apache.org/jira/browse/SPARK-16371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15363757#comment-15363757 ] Hyukjin Kwon edited comment on SPARK-16371 at 7/6/16 4:34 AM: -- Here is

<    1   2   3   4   5   6   7   8   9   10   >