[jira] [Commented] (SPARK-22708) spark on yarn error but Final app status: SUCCEEDED, exitCode: 0

2017-12-06 Thread jinchen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281232#comment-16281232 ] jinchen commented on SPARK-22708: - This happened when I do : {code:java} df.createOrReplaceTempView

[jira] [Assigned] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22725: Assignee: Apache Spark > df.select on a Stream is broken, vs a List >

[jira] [Commented] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281353#comment-16281353 ] Apache Spark commented on SPARK-22725: -- User 'ash211' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22725: Assignee: (was: Apache Spark) > df.select on a Stream is broken, vs a List >

[jira] [Created] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Andrew Ash (JIRA)
Andrew Ash created SPARK-22725: -- Summary: df.select on a Stream is broken, vs a List Key: SPARK-22725 URL: https://issues.apache.org/jira/browse/SPARK-22725 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22687) Run spark-sql in scala-2.12 and JDK9

2017-12-06 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281467#comment-16281467 ] liyunzhang commented on SPARK-22687: {code} bin/spark-submit --class

[jira] [Comment Edited] (SPARK-22660) Use position() and limit() to fix ambiguity issue in scala-2.12

2017-12-06 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281463#comment-16281463 ] liyunzhang edited comment on SPARK-22660 at 12/7/17 7:54 AM: - [~srowen] : As

[jira] [Created] (SPARK-22726) Basic tests for Binary Comparison and ImplicitTypeCasts

2017-12-06 Thread Xiao Li (JIRA)
Xiao Li created SPARK-22726: --- Summary: Basic tests for Binary Comparison and ImplicitTypeCasts Key: SPARK-22726 URL: https://issues.apache.org/jira/browse/SPARK-22726 Project: Spark Issue Type:

[jira] [Commented] (SPARK-1153) Generalize VertexId in GraphX so that UUIDs can be used as vertex IDs.

2017-12-06 Thread Brain (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281271#comment-16281271 ] Brain commented on SPARK-1153: -- :) why do not add string or uuid support? > Generalize VertexId in GraphX

[jira] [Commented] (SPARK-22725) df.select on a Stream is broken, vs a List

2017-12-06 Thread Andrew Ash (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281354#comment-16281354 ] Andrew Ash commented on SPARK-22725: Demonstration of difference between {{.map}} on List vs Stream:

[jira] [Commented] (SPARK-22717) Postgres is assumed to cascade truncate table by default which is not true

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281341#comment-16281341 ] Dongjoon Hyun commented on SPARK-22717: --- [~bolke]. The current behavior is not a bug. It's

[jira] [Resolved] (SPARK-22717) Postgres is assumed to cascade truncate table by default which is not true

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-22717. --- Resolution: Invalid > Postgres is assumed to cascade truncate table by default which is not

[jira] [Closed] (SPARK-22717) Postgres is assumed to cascade truncate table by default which is not true

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-22717. - > Postgres is assumed to cascade truncate table by default which is not true >

[jira] [Commented] (SPARK-22717) Postgres is assumed to cascade truncate table by default which is not true

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281359#comment-16281359 ] Dongjoon Hyun commented on SPARK-22717: --- I'm looking forward to see the PR for PostgreSQL. I'll

[jira] [Commented] (SPARK-22660) Use position() and limit() to fix ambiguity issue in scala-2.12

2017-12-06 Thread liyunzhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281463#comment-16281463 ] liyunzhang commented on SPARK-22660: [~srowen] : As we have finished the review of this jira, can

[jira] [Commented] (SPARK-22680) SparkSQL scan all partitions when the specified partitions are not exists in parquet formatted table

2017-12-06 Thread Jiang Xingbo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281235#comment-16281235 ] Jiang Xingbo commented on SPARK-22680: -- Could you also post the result of EXPLAIN? Thanks! >

[jira] [Resolved] (SPARK-22715) Reuse array in CreateNamedStruct

2017-12-06 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-22715. - Resolution: Not A Problem > Reuse array in CreateNamedStruct >

[jira] [Commented] (SPARK-22711) _pickle.PicklingError: args[0] from __newobj__ args has the wrong class from cloudpickle.py

2017-12-06 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281300#comment-16281300 ] Hyukjin Kwon commented on SPARK-22711: -- Can you give a self-reproducer if you wouldn't mind and is

[jira] [Updated] (SPARK-22660) Use position() and limit() to fix ambiguity issue in scala-2.12

2017-12-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22660: -- Summary: Use position() and limit() to fix ambiguity issue in scala-2.12 (was: se position() and

[jira] [Created] (SPARK-22719) refactor ConstantPropagation

2017-12-06 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-22719: -- Summary: refactor ConstantPropagation Key: SPARK-22719 URL: https://issues.apache.org/jira/browse/SPARK-22719 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-22719) refactor ConstantPropagation

2017-12-06 Thread Gengliang Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-22719: --- Description: The current time complexity of ConstantPropagation is O(n^2), which can be slow

[jira] [Resolved] (SPARK-22710) ConfigBuilder.fallbackConf doesn't trigger onCreate function

2017-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22710. - Resolution: Fixed Fix Version/s: 2.3.0 > ConfigBuilder.fallbackConf doesn't trigger onCreate

[jira] [Resolved] (SPARK-22698) Avoid the generation of useless mutable states by GenerateUnsafeProjection

2017-12-06 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-22698. - Resolution: Invalid I am closing since I have not been able to solve the issue without

[jira] [Updated] (SPARK-22717) Postgres is assumed to cascade truncate table by default which is not true

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-22717: -- Affects Version/s: (was: 2.2.2) 2.2.1 > Postgres is assumed to

[jira] [Resolved] (SPARK-14228) Lost executor of RPC disassociated, and occurs exception: Could not find CoarseGrainedScheduler or it has been stopped

2017-12-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-14228. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19741

[jira] [Assigned] (SPARK-22699) Avoid the generation of useless mutable states by GenerateSafeProjection

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22699: Assignee: Apache Spark > Avoid the generation of useless mutable states by

[jira] [Commented] (SPARK-22699) Avoid the generation of useless mutable states by GenerateSafeProjection

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280613#comment-16280613 ] Apache Spark commented on SPARK-22699: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22699) Avoid the generation of useless mutable states by GenerateSafeProjection

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22699: Assignee: (was: Apache Spark) > Avoid the generation of useless mutable states by

[jira] [Updated] (SPARK-22713) OOM caused by the memory contention and memory leak in TaskMemoryManager

2017-12-06 Thread Lijie Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Xu updated SPARK-22713: - Description: *[Abstract]* I recently encountered an OOM error in a PageRank application

[jira] [Created] (SPARK-22712) Use `buildReaderWithPartitionValues` in native OrcFileFormat

2017-12-06 Thread Dongjoon Hyun (JIRA)
Dongjoon Hyun created SPARK-22712: - Summary: Use `buildReaderWithPartitionValues` in native OrcFileFormat Key: SPARK-22712 URL: https://issues.apache.org/jira/browse/SPARK-22712 Project: Spark

[jira] [Updated] (SPARK-22713) OOM caused by the memory contention and memory leak in TaskMemoryManager

2017-12-06 Thread Lijie Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Xu updated SPARK-22713: - Description: *[Abstract]* I recently encountered an OOM error in a PageRank application

[jira] [Updated] (SPARK-22713) OOM caused by the memory contention and memory leak in TaskMemoryManager

2017-12-06 Thread Lijie Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Xu updated SPARK-22713: - Description: *[Abstract]* I recently encountered an OOM error in a PageRank application

[jira] [Created] (SPARK-22713) OOM caused by the memory contention and memory leak in TaskMemoryManager

2017-12-06 Thread Lijie Xu (JIRA)
Lijie Xu created SPARK-22713: Summary: OOM caused by the memory contention and memory leak in TaskMemoryManager Key: SPARK-22713 URL: https://issues.apache.org/jira/browse/SPARK-22713 Project: Spark

[jira] [Updated] (SPARK-22713) OOM caused by the memory contention and memory leak in TaskMemoryManager

2017-12-06 Thread Lijie Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Xu updated SPARK-22713: - Description: The pdf version of this issue is at

[jira] [Commented] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279818#comment-16279818 ] Apache Spark commented on SPARK-22516: -- User 'smurakozi' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22516: Assignee: Apache Spark > CSV Read breaks: When "multiLine" = "true", if "comment" option

[jira] [Assigned] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22516: Assignee: (was: Apache Spark) > CSV Read breaks: When "multiLine" = "true", if

[jira] [Updated] (SPARK-22713) OOM caused by the memory contention and memory leak in TaskMemoryManager

2017-12-06 Thread Lijie Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Xu updated SPARK-22713: - Description: The pdf version of this issue with high-quality figures is available at

[jira] [Commented] (SPARK-22678) Schema Exception of SaveAsTable with SparkML Pipeline.transform Result DataFrame

2017-12-06 Thread geosmart (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279814#comment-16279814 ] geosmart commented on SPARK-22678: -- so nobody answered! but I resolve it by split the onehot pipeline

[jira] [Assigned] (SPARK-22712) Use `buildReaderWithPartitionValues` in native OrcFileFormat

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22712: Assignee: Apache Spark > Use `buildReaderWithPartitionValues` in native OrcFileFormat >

[jira] [Assigned] (SPARK-22712) Use `buildReaderWithPartitionValues` in native OrcFileFormat

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22712: Assignee: (was: Apache Spark) > Use `buildReaderWithPartitionValues` in native

[jira] [Commented] (SPARK-22712) Use `buildReaderWithPartitionValues` in native OrcFileFormat

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279861#comment-16279861 ] Apache Spark commented on SPARK-22712: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Updated] (SPARK-22660) se position() and limit() to fix ambiguity issue in scala-2.12

2017-12-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22660: -- Summary: se position() and limit() to fix ambiguity issue in scala-2.12 (was: Compile with scala-2.12

[jira] [Updated] (SPARK-22718) Expose mechanism for testing timeout based testcases in StreamTest

2017-12-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-22718: -- Priority: Minor (was: Major) Can you be more specific? there aren't details in the PR about what this

[jira] [Resolved] (SPARK-22704) Reduce # of mutable variables in Least and greatest

2017-12-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22704. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19899

[jira] [Assigned] (SPARK-22704) Reduce # of mutable variables in Least and greatest

2017-12-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22704: --- Assignee: Kazuaki Ishizaki > Reduce # of mutable variables in Least and greatest >

[jira] [Assigned] (SPARK-22719) refactor ConstantPropagation

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22719: Assignee: (was: Apache Spark) > refactor ConstantPropagation >

[jira] [Assigned] (SPARK-22719) refactor ConstantPropagation

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22719: Assignee: Apache Spark > refactor ConstantPropagation > > >

[jira] [Commented] (SPARK-22719) refactor ConstantPropagation

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280501#comment-16280501 ] Apache Spark commented on SPARK-22719: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-22695) Avoid the generation of useless mutable states by scalaUDF

2017-12-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22695: --- Assignee: Marco Gaido > Avoid the generation of useless mutable states by scalaUDF >

[jira] [Resolved] (SPARK-22695) Avoid the generation of useless mutable states by scalaUDF

2017-12-06 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22695. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19900

[jira] [Comment Edited] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-06 Thread Julien Cuquemelle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280548#comment-16280548 ] Julien Cuquemelle edited comment on SPARK-22683 at 12/6/17 5:32 PM: I

[jira] [Created] (SPARK-22720) Make EventTimeWatermark UnaryNode

2017-12-06 Thread Xiao Li (JIRA)
Xiao Li created SPARK-22720: --- Summary: Make EventTimeWatermark UnaryNode Key: SPARK-22720 URL: https://issues.apache.org/jira/browse/SPARK-22720 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-06 Thread Julien Cuquemelle (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280548#comment-16280548 ] Julien Cuquemelle commented on SPARK-22683: --- I don't understand your statement about delaying

[jira] [Commented] (SPARK-22683) Allow tuning the number of dynamically allocated executors wrt task number

2017-12-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280555#comment-16280555 ] Sean Owen commented on SPARK-22683: --- When there are more tasks than executor slots, you're saying you'd

[jira] [Assigned] (SPARK-22720) Make EventTimeWatermark UnaryNode

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22720: Assignee: Xiao Li (was: Apache Spark) > Make EventTimeWatermark UnaryNode >

[jira] [Assigned] (SPARK-22720) Make EventTimeWatermark UnaryNode

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22720: Assignee: Apache Spark (was: Xiao Li) > Make EventTimeWatermark UnaryNode >

[jira] [Commented] (SPARK-22720) Make EventTimeWatermark UnaryNode

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280600#comment-16280600 ] Apache Spark commented on SPARK-22720: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Updated] (SPARK-22720) Make EventTimeWatermark Extend UnaryNode

2017-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22720: Summary: Make EventTimeWatermark Extend UnaryNode (was: Make EventTimeWatermark UnaryNode) > Make

[jira] [Created] (SPARK-22721) BytesToBytesMap peak memory usage not accurate after reset()

2017-12-06 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-22721: - Summary: BytesToBytesMap peak memory usage not accurate after reset() Key: SPARK-22721 URL: https://issues.apache.org/jira/browse/SPARK-22721 Project:

[jira] [Commented] (SPARK-22717) Postgres is assumed to cascade truncate table by default which is not true

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280895#comment-16280895 ] Dongjoon Hyun commented on SPARK-22717: --- In the PR, we are discussing the side effects of

[jira] [Comment Edited] (SPARK-13493) json to DataFrame to parquet does not respect case sensitiveness

2017-12-06 Thread Ratnakar Choudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280909#comment-16280909 ] Ratnakar Choudhary edited comment on SPARK-13493 at 12/6/17 9:06 PM: -

[jira] [Assigned] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-12-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22516: -- Assignee: Sandor Murakozi > CSV Read breaks: When "multiLine" = "true", if "comment"

[jira] [Commented] (SPARK-18005) optional binary Dataframe Column throws (UTF8) is not a group while loading a Dataframe

2017-12-06 Thread Duo Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280820#comment-16280820 ] Duo Xu commented on SPARK-18005: [~buntha] [~emaynard] Can you provide a sample parquet file which can

[jira] [Resolved] (SPARK-22516) CSV Read breaks: When "multiLine" = "true", if "comment" option is set as last line's first character

2017-12-06 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22516. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19906

[jira] [Commented] (SPARK-22721) BytesToBytesMap peak memory usage not accurate after reset()

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280851#comment-16280851 ] Apache Spark commented on SPARK-22721: -- User 'juliuszsompolski' has created a pull request for this

[jira] [Assigned] (SPARK-22721) BytesToBytesMap peak memory usage not accurate after reset()

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22721: Assignee: (was: Apache Spark) > BytesToBytesMap peak memory usage not accurate after

[jira] [Assigned] (SPARK-22721) BytesToBytesMap peak memory usage not accurate after reset()

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22721: Assignee: Apache Spark > BytesToBytesMap peak memory usage not accurate after reset() >

[jira] [Commented] (SPARK-13493) json to DataFrame to parquet does not respect case sensitiveness

2017-12-06 Thread Ratnakar Choudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280909#comment-16280909 ] Ratnakar Choudhary commented on SPARK-13493: Hi, I am on Spark 2.1 and still seeing this

[jira] [Resolved] (SPARK-22720) Make EventTimeWatermark Extend UnaryNode

2017-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22720. - Resolution: Fixed Target Version/s: 2.3.0 > Make EventTimeWatermark Extend UnaryNode >

[jira] [Commented] (SPARK-22697) Avoid the generation of useless mutable states by GenerateMutableProjection

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280071#comment-16280071 ] Apache Spark commented on SPARK-22697: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22697) Avoid the generation of useless mutable states by GenerateMutableProjection

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22697: Assignee: (was: Apache Spark) > Avoid the generation of useless mutable states by

[jira] [Assigned] (SPARK-22697) Avoid the generation of useless mutable states by GenerateMutableProjection

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22697: Assignee: Apache Spark > Avoid the generation of useless mutable states by

[jira] [Assigned] (SPARK-22696) Avoid the generation of useless mutable states by objects functions

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22696: Assignee: (was: Apache Spark) > Avoid the generation of useless mutable states by

[jira] [Assigned] (SPARK-22696) Avoid the generation of useless mutable states by objects functions

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22696: Assignee: Apache Spark > Avoid the generation of useless mutable states by objects

[jira] [Commented] (SPARK-22696) Avoid the generation of useless mutable states by objects functions

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279965#comment-16279965 ] Apache Spark commented on SPARK-22696: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Updated] (SPARK-22708) spark on yarn error but Final app status: SUCCEEDED, exitCode: 0

2017-12-06 Thread jinchen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jinchen updated SPARK-22708: Description: I got log : {code:java} 17/12/06 18:14:59 INFO state.StateStoreCoordinatorRef: Registered

[jira] [Resolved] (SPARK-22693) Avoid the generation of useless mutable states in complexTypeCreator and predicates

2017-12-06 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22693. - Resolution: Fixed Assignee: Marco Gaido Fix Version/s: 2.3.0 > Avoid the generation of

[jira] [Commented] (SPARK-22716) revisit ctx.addReferenceObj

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281050#comment-16281050 ] Apache Spark commented on SPARK-22716: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22716) revisit ctx.addReferenceObj

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22716: Assignee: Apache Spark > revisit ctx.addReferenceObj > --- > >

[jira] [Assigned] (SPARK-22716) revisit ctx.addReferenceObj

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22716: Assignee: (was: Apache Spark) > revisit ctx.addReferenceObj >

[jira] [Created] (SPARK-22722) Test Coverage for Type Coercion Compatibility

2017-12-06 Thread Xiao Li (JIRA)
Xiao Li created SPARK-22722: --- Summary: Test Coverage for Type Coercion Compatibility Key: SPARK-22722 URL: https://issues.apache.org/jira/browse/SPARK-22722 Project: Spark Issue Type: Umbrella

[jira] [Created] (SPARK-22723) Add support for other data types and add mode info to ImageSchema

2017-12-06 Thread Tomas Nykodym (JIRA)
Tomas Nykodym created SPARK-22723: - Summary: Add support for other data types and add mode info to ImageSchema Key: SPARK-22723 URL: https://issues.apache.org/jira/browse/SPARK-22723 Project: Spark

[jira] [Created] (SPARK-22724) TakeOrderedAndProjectExec operator has poor performance when sorting on low cardinality keys

2017-12-06 Thread Jonny Serencsa (JIRA)
Jonny Serencsa created SPARK-22724: -- Summary: TakeOrderedAndProjectExec operator has poor performance when sorting on low cardinality keys Key: SPARK-22724 URL: https://issues.apache.org/jira/browse/SPARK-22724

[jira] [Updated] (SPARK-22714) Spark API Not responding when Fatal exception occurred in event loop

2017-12-06 Thread Ariya Mizutani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariya Mizutani updated SPARK-22714: --- Description: To reproduce, let Spark to throw an OOM Exception in event loop: {noformat}

[jira] [Resolved] (SPARK-22694) Avoid the generation of useless mutable states by regexp functions

2017-12-06 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-22694. - Resolution: Invalid > Avoid the generation of useless mutable states by regexp functions >

[jira] [Resolved] (SPARK-22684) Avoid the generation of useless mutable states by datetime functions

2017-12-06 Thread Marco Gaido (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-22684. - Resolution: Invalid > Avoid the generation of useless mutable states by datetime functions >

[jira] [Created] (SPARK-22715) Reuse array in CreateNamedStruct

2017-12-06 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-22715: --- Summary: Reuse array in CreateNamedStruct Key: SPARK-22715 URL: https://issues.apache.org/jira/browse/SPARK-22715 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22715) Reuse array in CreateNamedStruct

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16280168#comment-16280168 ] Apache Spark commented on SPARK-22715: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-22715) Reuse array in CreateNamedStruct

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22715: Assignee: (was: Apache Spark) > Reuse array in CreateNamedStruct >

[jira] [Assigned] (SPARK-22715) Reuse array in CreateNamedStruct

2017-12-06 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22715: Assignee: Apache Spark > Reuse array in CreateNamedStruct >

[jira] [Created] (SPARK-22716) revisit ctx.addReferenceObj

2017-12-06 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-22716: --- Summary: revisit ctx.addReferenceObj Key: SPARK-22716 URL: https://issues.apache.org/jira/browse/SPARK-22716 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-22714) Spark API Not responding when Fatal exception occurred in event loop

2017-12-06 Thread Ariya Mizutani (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariya Mizutani updated SPARK-22714: --- Description: To reproduce, let Spark to throw an OOM Exception in event loop: {norormat}

[jira] [Created] (SPARK-22714) Spark API Not responding when Fatal exception occurred in event loop

2017-12-06 Thread Ariya Mizutani (JIRA)
Ariya Mizutani created SPARK-22714: -- Summary: Spark API Not responding when Fatal exception occurred in event loop Key: SPARK-22714 URL: https://issues.apache.org/jira/browse/SPARK-22714 Project:

[jira] [Resolved] (SPARK-22678) Schema Exception of SaveAsTable with SparkML Pipeline.transform Result DataFrame

2017-12-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22678. --- Resolution: Not A Problem Target Version/s: (was: 2.2.0) > Schema Exception of

[jira] [Resolved] (SPARK-22708) spark on yarn error but Final app status: SUCCEEDED, exitCode: 0

2017-12-06 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22708. --- Resolution: Not A Problem You can see this is an Elasticsearch error. Depending on how your app is

[jira] [Commented] (SPARK-13493) json to DataFrame to parquet does not respect case sensitiveness

2017-12-06 Thread Michel Lemay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281112#comment-16281112 ] Michel Lemay commented on SPARK-13493: -- Here we decided to snake_case everything and leave case

[jira] [Commented] (SPARK-15705) Spark won't read ORC schema from metastore for partitioned tables

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281170#comment-16281170 ] Dongjoon Hyun commented on SPARK-15705: --- Since 2.2.1 is released, I'll update the result from

[jira] [Updated] (SPARK-15757) Error occurs when using Spark sql "select" statement on orc file after hive sql "insert overwrite tb1 select * from sourcTb" has been executed on this orc file

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-15757: -- Fix Version/s: 2.2.1 > Error occurs when using Spark sql "select" statement on orc file after

[jira] [Commented] (SPARK-15757) Error occurs when using Spark sql "select" statement on orc file after hive sql "insert overwrite tb1 select * from sourcTb" has been executed on this orc file

2017-12-06 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16281192#comment-16281192 ] Dongjoon Hyun commented on SPARK-15757: --- This is fixed by SPARK-14387 in 2.2.1. {code} hive> CREATE

  1   2   >