[jira] (SPARK-6802) User Defined Aggregate Function Refactoring

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon commented on SPARK-6802

[jira] (SPARK-9652) Make reading Avro to RDDs easier

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon commented on SPARK-9652

[jira] (SPARK-15559) TopicAndPartition should provide __hash__ method

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon commented on SPARK-15559

[jira] [Resolved] (SPARK-10924) Failed to update accumulators for ShuffleMapTask: Broken pipe

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-10924. -- Resolution: Cannot Reproduce Thank you so much for checking this. Let me then resolve this as

[jira] [Resolved] (SPARK-13316) "SparkException: DStream has not been initialized" when restoring StreamingContext from checkpoint and the dstream is created afterwards

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13316. -- Resolution: Not A Problem I tried to reproduce this as below: {code} nc -lk {code}

[jira] [Commented] (SPARK-17360) PySpark can create dataframe from a Python generator

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837681#comment-15837681 ] Hyukjin Kwon commented on SPARK-17360: -- Hi [~holdenk], could we resolve this given the discussion in

[jira] [Resolved] (SPARK-12827) Configurable bind address for WebUI

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12827. -- Resolution: Duplicate > Configurable bind address for WebUI >

[jira] [Resolved] (SPARK-14955) JDBCRelation should report an IllegalArgumentException if stride equals 0

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14955. -- Resolution: Duplicate > JDBCRelation should report an IllegalArgumentException if stride

[jira] [Commented] (SPARK-13637) use more information to simplify the code in Expand builder

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837913#comment-15837913 ] Hyukjin Kwon commented on SPARK-13637: -- ([~cloud_fan] it seems this one is mistakenly not resolved)

[jira] [Resolved] (SPARK-14165) NoSuchElementException: None.get when joining DataFrames with Seq of fields of different case

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14165. -- Resolution: Fixed Ah, thanks. Let me then resolve it. > NoSuchElementException: None.get

[jira] [Commented] (SPARK-15809) PySpark SQL UDF default returnType

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839348#comment-15839348 ] Hyukjin Kwon commented on SPARK-15809: -- I don't think it is worth to do this with breaking the API

[jira] [Reopened] (SPARK-15463) Support for creating a dataframe from CSV in Dataset[String]

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-15463: -- I am reopening this as I feel we need this per the issues

[jira] [Commented] (SPARK-15505) Explode nested Array in DF Column into Multiple Columns

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839386#comment-15839386 ] Hyukjin Kwon commented on SPARK-15505: -- Ah, then, we should calculate the maximum length of that

[jira] [Commented] (SPARK-15505) Explode nested Array in DF Column into Multiple Columns

2017-01-25 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839280#comment-15839280 ] Hyukjin Kwon commented on SPARK-15505: -- Can we just {{df.selectExpr("Col1", "Col2[0]", "Col2[1]",

[jira] [Resolved] (SPARK-10842) Eliminate create duplicate stage while generate job dag

2017-01-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-10842. -- Resolution: Duplicate I am resolving this as a duplicate per the reporter's comment

[jira] [Commented] (SPARK-19015) SQL request with transformation cannot be eecuted if not run first a scan table

2017-01-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833241#comment-15833241 ] Hyukjin Kwon commented on SPARK-19015: -- If it is a question, I believe you could get a better answer

[jira] [Commented] (SPARK-10908) ClassCastException in HadoopRDD.getJobConf

2017-01-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842610#comment-15842610 ] Hyukjin Kwon commented on SPARK-10908: -- Would you be possible to provide a self-reproducer? I am

[jira] [Commented] (SPARK-19087) Numpy types fail to be casted to any other types

2017-01-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842619#comment-15842619 ] Hyukjin Kwon commented on SPARK-19087: -- Is this a duplicate of SPARK-12157? > Numpy types fail to

[jira] [Commented] (SPARK-12606) Scala/Java compatibility issue Re: how to extend java transformer from Scala UnaryTransformer ?

2017-01-27 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842630#comment-15842630 ] Hyukjin Kwon commented on SPARK-12606: -- Do you mind if I ask to fix the title and contents in more

[jira] (SPARK-12955) Spark-HiveSQL: It fail when is quering a nested structure

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon resolved as Cannot Reproduce

[jira] (SPARK-13752) JSON array type parsing error

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon resolved as Cannot Reproduce

[jira] (SPARK-13851) spark streaming web ui remains completed jobs as active jobs

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon commented on SPARK-13851

[jira] (SPARK-14854) Left outer join produces incorrect output when the join condition does not have left table key

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon resolved as Not A Problem

[jira] (SPARK-15000) Spark hangs indefinitely if you cache a dataframe, then show it, then do some further processing on it

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon resolved as Cannot Reproduce

[jira] (SPARK-15070) Data corruption when using Dataset.groupBy[K : Encoder](func: T => K) when data loaded from JSON file.

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon commented on SPARK-15070

[jira] (SPARK-15175) using csv.DictReader and existing json dataframe reader as work around to support csv reader

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon resolved as Not A Problem

[jira] (SPARK-15506) only one notebook can define a UDF; java.sql.SQLException: Another instance of Derby may have already booted the database

2017-01-29 Thread Hyukjin Kwon (JIRA)
Title: Message Title Hyukjin Kwon resolved as Invalid

[jira] [Commented] (SPARK-19666) Exception when calling createDataFrame with typed RDD

2017-02-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875604#comment-15875604 ] Hyukjin Kwon commented on SPARK-19666: -- Hm, probably the {{__Message}} is an invalid java bean.

[jira] [Commented] (SPARK-19666) Exception when calling createDataFrame with typed RDD

2017-02-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875672#comment-15875672 ] Hyukjin Kwon commented on SPARK-19666: -- Ah, yes. I just tested some cases too. I could reproduce the

[jira] [Commented] (SPARK-17636) Parquet filter push down doesn't handle struct fields

2017-02-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875193#comment-15875193 ] Hyukjin Kwon commented on SPARK-17636: -- FWIW, SPARK-4502 is about nested column pruning as a related

[jira] [Commented] (SPARK-19641) JSON schema inference in DROPMALFORMED mode produces incorrect schema

2017-02-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871196#comment-15871196 ] Hyukjin Kwon commented on SPARK-19641: -- Ah, thanks for cc'ing me. I happened to see the related

[jira] [Commented] (SPARK-19615) Provide Dataset union convenience for divergent schema

2017-02-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873189#comment-15873189 ] Hyukjin Kwon commented on SPARK-19615: -- Let me leave loosely related JIRAs - SPARK-9813 , SPARK-9874

[jira] [Commented] (SPARK-19615) Provide Dataset union convenience for divergent schema

2017-02-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873188#comment-15873188 ] Hyukjin Kwon commented on SPARK-19615: -- I remember I checked UNION operation in other DBMS and

[jira] [Commented] (SPARK-19655) select count(*) , requests 1 for each row

2017-02-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15873184#comment-15873184 ] Hyukjin Kwon commented on SPARK-19655: -- I guess the problematic line is,

[jira] [Commented] (SPARK-19709) CSV datasource fails to read empty file

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881648#comment-15881648 ] Hyukjin Kwon commented on SPARK-19709: -- Please go ahead. (but I _personally_ recommend you open a PR

[jira] [Commented] (SPARK-14079) Limit the number of queries on SQL UI

2017-02-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883966#comment-15883966 ] Hyukjin Kwon commented on SPARK-14079: -- [~shixi...@databricks.com], I am just curious if this JIRA

[jira] [Comment Edited] (SPARK-14079) Limit the number of queries on SQL UI

2017-02-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883966#comment-15883966 ] Hyukjin Kwon edited comment on SPARK-14079 at 2/25/17 2:47 AM: --- [~zsxwing],

[jira] [Commented] (SPARK-14079) Limit the number of queries on SQL UI

2017-02-24 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883968#comment-15883968 ] Hyukjin Kwon commented on SPARK-14079: -- I am adding a link. Please correct this if wrong. > Limit

[jira] [Resolved] (SPARK-9275) IsolatedClientLoader could not load shared JNI libraries

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-9275. - Resolution: Duplicate > IsolatedClientLoader could not load shared JNI libraries >

[jira] [Commented] (SPARK-6072) Enable hash joins for null-safe equality predicates

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880456#comment-15880456 ] Hyukjin Kwon commented on SPARK-6072: - Hi [~dimazhiyanov], could you confirm ^ please? > Enable hash

[jira] [Commented] (SPARK-6678) select count(DISTINCT C_UID) from parquetdir may be can optimize

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880463#comment-15880463 ] Hyukjin Kwon commented on SPARK-6678: - gentle ping [~cnstar9988] > select count(DISTINCT C_UID) from

[jira] [Commented] (SPARK-19709) CSV datasource fails to read empty file

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880241#comment-15880241 ] Hyukjin Kwon commented on SPARK-19709: -- Let me fix this soon. > CSV datasource fails to read empty

[jira] [Created] (SPARK-19709) CSV datasource fails to read empty file

2017-02-23 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-19709: Summary: CSV datasource fails to read empty file Key: SPARK-19709 URL: https://issues.apache.org/jira/browse/SPARK-19709 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-19708) delete jar unable

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-19708. -- Resolution: Duplicate > delete jar unable > - > > Key:

[jira] [Commented] (SPARK-14480) Remove meaningless StringIteratorReader for CSV data source for better performance

2017-02-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879629#comment-15879629 ] Hyukjin Kwon commented on SPARK-14480: -- This seems not blocked by any of those [~pes2009k]. I sent a

[jira] [Comment Edited] (SPARK-12264) Add a typeTag or scalaTypeTag method to DataType

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813946#comment-15813946 ] Hyukjin Kwon edited comment on SPARK-12264 at 2/23/17 3:09 PM: --- (I just

[jira] [Issue Comment Deleted] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12890: - Comment: was deleted (was: [~rxin] Could you confirm if this is an issue?) > Spark SQL query

[jira] [Issue Comment Deleted] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-12890: - Comment: was deleted (was: Actually I don't still understand what is an issue here. This might

[jira] [Resolved] (SPARK-12051) Can't register UDF from Hive thrift server

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12051. -- Resolution: Duplicate I am resolving this as a duplicate per your comment in

[jira] [Updated] (SPARK-11784) Support Timestamp filter pushdown in Parquet datasource

2017-02-23 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-11784: - Summary: Support Timestamp filter pushdown in Parquet datasource (was: enable Timestamp filter

[jira] [Resolved] (SPARK-8510) NumPy arrays and matrices as values in sequence files

2017-02-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8510. - Resolution: Won't Fix I am resolving this per comments in

[jira] [Resolved] (SPARK-15679) Passing functions do not work in classes

2017-02-20 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-15679. -- Resolution: Invalid Your test codes look not working as is. I manually modified your codes and

[jira] [Commented] (SPARK-19557) Output parameters are not present in SQL Query Plan

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862595#comment-15862595 ] Hyukjin Kwon commented on SPARK-19557: -- Could you describe "this information", input and output

[jira] [Commented] (SPARK-19442) Unable to add column to the dataset using Dataset.WithColumn() api

2017-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865256#comment-15865256 ] Hyukjin Kwon commented on SPARK-19442: -- How about something like this? {code} import

[jira] [Commented] (SPARK-19536) Improve capability to merge SQL data types

2017-02-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860665#comment-15860665 ] Hyukjin Kwon commented on SPARK-19536: -- Oh, for {{ArrayType}}, I filed a JIRA, SPARK-19435 and

[jira] [Created] (SPARK-19544) Improve error message when some column types are compatible and others are not in set/union operations

2017-02-09 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-19544: Summary: Improve error message when some column types are compatible and others are not in set/union operations Key: SPARK-19544 URL:

[jira] [Updated] (SPARK-19544) Improve error message when some column types are compatible and others are not in set/union operations

2017-02-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-19544: - Description: Currently, {code} Seq((1,("a", 1))).toDF union Seq((1L,("a", "b"))).toDF

[jira] [Commented] (SPARK-19532) [Core]`DataStreamer for file` threads of DFSOutputStream leak if set `spark.speculation` to true

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862443#comment-15862443 ] Hyukjin Kwon commented on SPARK-19532: -- Is it a blocker BTW? > [Core]`DataStreamer for file`

[jira] [Commented] (SPARK-6628) ClassCastException occurs when executing sql statement "insert into" on hbase table

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862454#comment-15862454 ] Hyukjin Kwon commented on SPARK-6628: - Hi all, then, is this not an Spark issue? > ClassCastException

[jira] [Commented] (SPARK-6678) select count(DISTINCT C_UID) from parquetdir may be can optimize

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862455#comment-15862455 ] Hyukjin Kwon commented on SPARK-6678: - [~cnstar9988], do you mind if I ask your other spark code? I

[jira] [Commented] (SPARK-7101) Spark SQL should support java.sql.Time

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862457#comment-15862457 ] Hyukjin Kwon commented on SPARK-7101: - Could anyone explain the use case of this type? > Spark SQL

[jira] [Updated] (SPARK-19595) from_json produces only a single row when input is a json array

2017-02-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-19595: - Description: Currently, {{from_json}} reads a single row when it is a json array. For example,

[jira] [Created] (SPARK-19595) from_json produces only a single row when input is a json array

2017-02-14 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-19595: Summary: from_json produces only a single row when input is a json array Key: SPARK-19595 URL: https://issues.apache.org/jira/browse/SPARK-19595 Project: Spark

[jira] [Resolved] (SPARK-14894) Python GaussianMixture summary

2017-02-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-14894. -- Resolution: Duplicate [~wangmiao1981], I guess we could take an action to JIRA too if we are

[jira] [Commented] (SPARK-19442) Unable to add column to the dataset using Dataset.WithColumn() api

2017-02-14 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15867277#comment-15867277 ] Hyukjin Kwon commented on SPARK-19442: -- ping [~Navya Krishnappa], would this satisfy your demand?

[jira] [Resolved] (SPARK-8081) Problems with Optimized Logical Plan Generation

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-8081. - Resolution: Invalid I hope there are no such JIRAs, like, copying and just pasting the production

[jira] [Commented] (SPARK-9414) HiveContext:saveAsTable creates wrong partition for existing hive table(append mode)

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862466#comment-15862466 ] Hyukjin Kwon commented on SPARK-9414: - [~xguo27], could you resolve this as {{Cannot Reproduce}} if

[jira] [Commented] (SPARK-19571) appveyor windows tests are failing

2017-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864096#comment-15864096 ] Hyukjin Kwon commented on SPARK-19571: -- Oh, I overlooked and I thought it is just because of

[jira] [Commented] (SPARK-19536) Improve capability to merge SQL data types

2017-02-09 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860816#comment-15860816 ] Hyukjin Kwon commented on SPARK-19536: -- Thank you for kindly adding a link. > Improve capability to

[jira] [Commented] (SPARK-19621) R Windows AppVeyor test should run CRAN checks

2017-02-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870089#comment-15870089 ] Hyukjin Kwon commented on SPARK-19621: -- Let me maybe try if no one tries. For me, it'd take a bit of

[jira] [Commented] (SPARK-19614) add type-preserving null function

2017-02-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870082#comment-15870082 ] Hyukjin Kwon commented on SPARK-19614: -- [~ndimiduk]] Or maybe you are referring

[jira] [Issue Comment Deleted] (SPARK-19610) multi line support for CSV

2017-02-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-19610: - Comment: was deleted (was: Sure, let me try. Thanks for cc'ing me.) > multi line support for

[jira] [Commented] (SPARK-19610) multi line support for CSV

2017-02-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868699#comment-15868699 ] Hyukjin Kwon commented on SPARK-19610: -- Sure, let me try. Thanks for cc'ing me. > multi line

[jira] [Commented] (SPARK-19610) multi line support for CSV

2017-02-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868700#comment-15868700 ] Hyukjin Kwon commented on SPARK-19610: -- Sure, let me try. Thanks for cc'ing me. > multi line

[jira] [Resolved] (SPARK-19442) Unable to add column to the dataset using Dataset.WithColumn() api

2017-02-15 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-19442. -- Resolution: Invalid Good to know. Let me then resolve it. Thanks for your input to help verify

[jira] [Commented] (SPARK-18699) Spark CSV parsing types other than String throws exception when malformed

2017-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863426#comment-15863426 ] Hyukjin Kwon commented on SPARK-18699: -- Thanks for cc'ing me. For me, it is reasonable to me too

[jira] [Commented] (SPARK-19442) Unable to add column to the dataset using Dataset.WithColumn() api

2017-02-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15863601#comment-15863601 ] Hyukjin Kwon commented on SPARK-19442: -- Could I please ask a example like I did if I misunderstood?

[jira] [Commented] (SPARK-19557) Output parameters are not present in SQL Query Plan

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862368#comment-15862368 ] Hyukjin Kwon commented on SPARK-19557: -- I am sorry that I don't follow the JIRA. Could I please ask

[jira] [Commented] (SPARK-19557) Output parameters are not present in SQL Query Plan

2017-02-11 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862371#comment-15862371 ] Hyukjin Kwon commented on SPARK-19557: -- I mean, up to my understanding, we do have path in the query

[jira] [Commented] (SPARK-19222) Limit Query Performance issue

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827619#comment-15827619 ] Hyukjin Kwon commented on SPARK-19222: -- Can you make the indentation pretty if you want to remove

[jira] [Commented] (SPARK-19217) Offer easy cast from vector to array

2017-01-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823713#comment-15823713 ] Hyukjin Kwon commented on SPARK-19217: -- Yea, as I said above, it is possible to read/write but I

[jira] [Commented] (SPARK-19248) Regex_replace works in 1.6 but not in 2.0

2017-01-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15825173#comment-15825173 ] Hyukjin Kwon commented on SPARK-19248: -- I just looked into this for my curiosity. This seems related

[jira] [Commented] (SPARK-19217) Offer easy cast from vector to array

2017-01-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824677#comment-15824677 ] Hyukjin Kwon commented on SPARK-19217: -- (That's fixed in SPARK-17765 for ORC up to my knowledge.) >

[jira] [Comment Edited] (SPARK-19217) Offer easy cast from vector to array

2017-01-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824677#comment-15824677 ] Hyukjin Kwon edited comment on SPARK-19217 at 1/16/17 11:05 PM: (That's

[jira] [Resolved] (SPARK-11562) Provide user an option to init SQLContext or HiveContext in spark shell

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-11562. -- Resolution: Not A Problem Per the comment from a committer {quote} HiveContext is now

[jira] [Resolved] (SPARK-12359) Add showString() to DataSet API.

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12359. -- Resolution: Not A Problem It seems this was added when migrating {{DataFrame}} to {{Dataset}}.

[jira] [Resolved] (SPARK-12377) Wrong implementation for Row.__call__ in pyspark

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-12377. -- Resolution: Invalid I verified this while trying to take over the PR and it seems working as

[jira] [Resolved] (SPARK-13335) Optimize Data Frames collect_list and collect_set with declarative aggregates

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-13335. -- Resolution: Duplicate I am resolving this per committer's comments -

[jira] [Commented] (SPARK-13936) PushPredicateThroughProject using Constraints

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828168#comment-15828168 ] Hyukjin Kwon commented on SPARK-13936: -- [~smilegator], I just happened to see your comment -

[jira] [Resolved] (SPARK-10220) org.apache.spark.sql.jdbc.JDBCRDD could not parse mysql table column named reserved word

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-10220. -- Resolution: Cannot Reproduce I manually fixed all the tests to use the column name {{desc}} in

[jira] [Resolved] (SPARK-10506) There exits some potential resource leak in jsonExpressions.scala

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-10506. -- Resolution: Duplicate > There exits some potential resource leak in jsonExpressions.scala >

[jira] [Commented] (SPARK-12143) When column type is binary, select occurs ClassCastExcption in Beeline.

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828119#comment-15828119 ] Hyukjin Kwon commented on SPARK-12143: -- [~srowen][~rxin], this one seems mistakenly not closed. >

[jira] [Commented] (SPARK-15023) Add support for testing against the `ProcessingTime(intervalMS > 0)` trigger and `ManualClock`

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828188#comment-15828188 ] Hyukjin Kwon commented on SPARK-15023: -- Hi [~lwlin], I just wonder if this one was resolved by the

[jira] [Commented] (SPARK-14172) Hive table partition predicate not passed down correctly

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828177#comment-15828177 ] Hyukjin Kwon commented on SPARK-14172: -- Hi [~hvanhovell], this JIRA seems mistakenly not resolved.

[jira] [Commented] (SPARK-14172) Hive table partition predicate not passed down correctly

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15828179#comment-15828179 ] Hyukjin Kwon commented on SPARK-14172: -- Oh wait, sorry the linked PR above has a different JIRA. >

[jira] [Resolved] (SPARK-15667) Throw exception if columns number of outputs mismatch the inputs

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-15667. -- Resolution: Duplicate Resolving this per

[jira] [Resolved] (SPARK-17101) Provide consistent format identifiers for TextFileFormat and ParquetFileFormat

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17101. -- Resolution: Duplicate > Provide consistent format identifiers for TextFileFormat and

[jira] [Resolved] (SPARK-15805) update the whole sql programming guide

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-15805. -- Resolution: Duplicate It seems a duplicate. please reopen this if I misunderstood. > update

[jira] [Resolved] (SPARK-16924) DataStreamReader can not support option("inferSchema", true/false) for csv and json file source

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-16924. -- Resolution: Invalid I am resolving this JIRA. Please refer the discussion in the PR and reopen

[jira] [Resolved] (SPARK-17295) Create TestHiveSessionState use reflect logic based on the setting of CATALOG_IMPLEMENTATION

2017-01-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-17295. -- Resolution: Invalid Resolving this per the discussion in

<    2   3   4   5   6   7   8   9   10   11   >