[jira] [Commented] (SPARK-22792) PySpark UDF registering issue

[ https://issues.apache.org/jira/browse/SPARK-22792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292163#comment-16292163 ] Hyukjin Kwon commented on SPARK-22792: -- Does this work in 2.1.x or lower version? Ju

[jira] [Commented] (SPARK-22792) PySpark UDF registering issue

[ https://issues.apache.org/jira/browse/SPARK-22792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292162#comment-16292162 ] Annamalai Venugopal commented on SPARK-22792: - Sorry am new to this.I'll chan

[jira] [Updated] (SPARK-22792) PySpark UDF registering issue

[ https://issues.apache.org/jira/browse/SPARK-22792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-22792: - Fix Version/s: (was: 2.2.1) > PySpark UDF registering issue > - >

[jira] [Updated] (SPARK-22792) PySpark UDF registering issue

[ https://issues.apache.org/jira/browse/SPARK-22792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-22792: - Target Version/s: (was: 2.2.1) > PySpark UDF registering issue > -

[jira] [Updated] (SPARK-22792) PySpark UDF registering issue

[ https://issues.apache.org/jira/browse/SPARK-22792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-22792: - Priority: Major (was: Blocker) > PySpark UDF registering issue > - >

[jira] [Commented] (SPARK-22792) PySpark UDF registering issue

[ https://issues.apache.org/jira/browse/SPARK-22792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292158#comment-16292158 ] Hyukjin Kwon commented on SPARK-22792: -- Please don't set the blocker and target vers

[jira] [Updated] (SPARK-22794) Spark Job failed, but the state is succeeded in Yarn Web

[ https://issues.apache.org/jira/browse/SPARK-22794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXinXIaoLei updated SPARK-22794: -- Attachment: task_is_succeeded_in_yarn_web.png > Spark Job failed, but the state is succeeded in

[jira] [Updated] (SPARK-22794) Spark Job failed, but the state is succeeded in Yarn Web

[ https://issues.apache.org/jira/browse/SPARK-22794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] KaiXinXIaoLei updated SPARK-22794: -- Description: I run a job in yarn mode, the job is failed: {noformat} 17/12/15 11:55:16 INFO Sh

[jira] [Created] (SPARK-22794) Spark Job failed, but the state is succeeded in Yarn Web

KaiXinXIaoLei created SPARK-22794: - Summary: Spark Job failed, but the state is succeeded in Yarn Web Key: SPARK-22794 URL: https://issues.apache.org/jira/browse/SPARK-22794 Project: Spark Is

[jira] [Updated] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

[ https://issues.apache.org/jira/browse/SPARK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Agarwal updated SPARK-22371: --- Attachment: ShuffleIssue.java Helper.scala > dag-scheduler-event-loop thread

[jira] [Updated] (SPARK-22793) Memory leak in Spark Thrift Server

[ https://issues.apache.org/jira/browse/SPARK-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zuotingbing updated SPARK-22793: Description: 1. Start HiveThriftServer2. 2. Connect to thriftserver through beeline. 3. Close the b

[jira] [Updated] (SPARK-22793) Memory leak in Spark Thrift Server

[ https://issues.apache.org/jira/browse/SPARK-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zuotingbing updated SPARK-22793: Description: 1. Start HiveThriftServer2 2. Connect to thriftserver through beeline 3. Close the bee

[jira] [Created] (SPARK-22793) Memory leak in Spark Thrift Server

zuotingbing created SPARK-22793: --- Summary: Memory leak in Spark Thrift Server Key: SPARK-22793 URL: https://issues.apache.org/jira/browse/SPARK-22793 Project: Spark Issue Type: Bug Co

[jira] [Created] (SPARK-22792) PySpark UDF registering issue

Annamalai Venugopal created SPARK-22792: --- Summary: PySpark UDF registering issue Key: SPARK-22792 URL: https://issues.apache.org/jira/browse/SPARK-22792 Project: Spark Issue Type: Quest

[jira] [Comment Edited] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

[ https://issues.apache.org/jira/browse/SPARK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292134#comment-16292134 ] Mayank Agarwal edited comment on SPARK-22371 at 12/15/17 7:14 AM: -

[jira] [Comment Edited] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

[ https://issues.apache.org/jira/browse/SPARK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292134#comment-16292134 ] Mayank Agarwal edited comment on SPARK-22371 at 12/15/17 7:13 AM: -

[jira] [Updated] (SPARK-22371) dag-scheduler-event-loop thread stopped with error Attempted to access garbage collected accumulator 5605982

[ https://issues.apache.org/jira/browse/SPARK-22371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Agarwal updated SPARK-22371: --- Attachment: ShuffleIssue.java Hi, Sorry for late reply. >From our analysis this error seems

[jira] [Resolved] (SPARK-22753) Get rid of dataSource.writeAndRead

[ https://issues.apache.org/jira/browse/SPARK-22753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22753. - Resolution: Fixed Assignee: Li Yuanjian Fix Version/s: 2.3.0 > Get rid of dataSource.writ

[jira] [Assigned] (SPARK-22791) Redact Output of Explain

[ https://issues.apache.org/jira/browse/SPARK-22791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22791: Assignee: Xiao Li (was: Apache Spark) > Redact Output of Explain > --

[jira] [Commented] (SPARK-22791) Redact Output of Explain

[ https://issues.apache.org/jira/browse/SPARK-22791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292131#comment-16292131 ] Apache Spark commented on SPARK-22791: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-22791) Redact Output of Explain

[ https://issues.apache.org/jira/browse/SPARK-22791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22791: Assignee: Apache Spark (was: Xiao Li) > Redact Output of Explain > --

[jira] [Created] (SPARK-22791) Redact Output of Explain

Xiao Li created SPARK-22791: --- Summary: Redact Output of Explain Key: SPARK-22791 URL: https://issues.apache.org/jira/browse/SPARK-22791 Project: Spark Issue Type: Bug Components: SQL

[jira] [Resolved] (SPARK-22787) Add a TPCH query suite

[ https://issues.apache.org/jira/browse/SPARK-22787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22787. - Resolution: Fixed Fix Version/s: 2.3.0 > Add a TPCH query suite > -- > >

[jira] [Commented] (SPARK-22647) Docker files for image creation

[ https://issues.apache.org/jira/browse/SPARK-22647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292113#comment-16292113 ] Anirudh Ramanathan commented on SPARK-22647: I think we haven't run that part

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16292048#comment-16292048 ] Xuefu Zhang commented on SPARK-22765: - [~tgraves], I think it would help if SPARK-216

[jira] [Created] (SPARK-22790) add a configurable factor to describe HadoopFsRelation's size

Nan Zhu created SPARK-22790: --- Summary: add a configurable factor to describe HadoopFsRelation's size Key: SPARK-22790 URL: https://issues.apache.org/jira/browse/SPARK-22790 Project: Spark Issue Ty

[jira] [Commented] (SPARK-22790) add a configurable factor to describe HadoopFsRelation's size

[ https://issues.apache.org/jira/browse/SPARK-22790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291985#comment-16291985 ] Nan Zhu commented on SPARK-22790: - created per discussion in https://github.com/apache/sp

[jira] [Commented] (SPARK-22781) Support creating streaming dataset with ORC files

[ https://issues.apache.org/jira/browse/SPARK-22781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291960#comment-16291960 ] Dongjoon Hyun commented on SPARK-22781: --- Hi, [~tdas] and [~zsxwing]. Could you give

[jira] [Assigned] (SPARK-22789) Add ContinuousExecution for continuous processing queries

[ https://issues.apache.org/jira/browse/SPARK-22789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22789: Assignee: (was: Apache Spark) > Add ContinuousExecution for continuous processing quer

[jira] [Commented] (SPARK-22789) Add ContinuousExecution for continuous processing queries

[ https://issues.apache.org/jira/browse/SPARK-22789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291900#comment-16291900 ] Apache Spark commented on SPARK-22789: -- User 'joseph-torres' has created a pull requ

[jira] [Assigned] (SPARK-22789) Add ContinuousExecution for continuous processing queries

[ https://issues.apache.org/jira/browse/SPARK-22789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22789: Assignee: Apache Spark > Add ContinuousExecution for continuous processing queries > -

[jira] [Resolved] (SPARK-22047) HiveExternalCatalogVersionsSuite is Flaky on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-22047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22047. - Resolution: Fixed Assignee: Wenchen Fan Fix Version/s: 2.3.0 > HiveExternalCatalo

[jira] [Commented] (SPARK-22036) BigDecimal multiplication sometimes returns null

[ https://issues.apache.org/jira/browse/SPARK-22036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291805#comment-16291805 ] Anvesh R commented on SPARK-22036: -- +1 Issue reproduced on spark-2.2.0 : Data at s3 lo

[jira] [Resolved] (SPARK-22496) beeline display operation log

[ https://issues.apache.org/jira/browse/SPARK-22496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-22496. - Resolution: Fixed > beeline display operation log > - > > Key

[jira] [Created] (SPARK-22789) Add ContinuousExecution for continuous processing queries

Jose Torres created SPARK-22789: --- Summary: Add ContinuousExecution for continuous processing queries Key: SPARK-22789 URL: https://issues.apache.org/jira/browse/SPARK-22789 Project: Spark Issue

[jira] [Resolved] (SPARK-22733) refactor StreamExecution for extensibility

[ https://issues.apache.org/jira/browse/SPARK-22733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-22733. -- Resolution: Fixed Assignee: Jose Torres Fix Version/s: 2.3.0 > refactor StreamE

[jira] [Assigned] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-22778: -- Assignee: Yinan Li > Kubernetes scheduler at master failing to run applications succes

[jira] [Resolved] (SPARK-22778) Kubernetes scheduler at master failing to run applications successfully

[ https://issues.apache.org/jira/browse/SPARK-22778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22778. Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19972 [https:/

[jira] [Resolved] (SPARK-3419) Scheduler shouldn't delay running a task when executors don't reside at any of its preferred locations

[ https://issues.apache.org/jira/browse/SPARK-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-3419. - Resolution: Fixed Fix Version/s: 1.3.0 This has been fixed for a long time, looks like as p

[jira] [Assigned] (SPARK-22788) HdfsUtils.getOutputStream uses non-existent Hadoop conf "hdfs.append.support"

[ https://issues.apache.org/jira/browse/SPARK-22788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22788: Assignee: (was: Apache Spark) > HdfsUtils.getOutputStream uses non-existent Hadoop con

[jira] [Commented] (SPARK-22788) HdfsUtils.getOutputStream uses non-existent Hadoop conf "hdfs.append.support"

[ https://issues.apache.org/jira/browse/SPARK-22788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291602#comment-16291602 ] Apache Spark commented on SPARK-22788: -- User 'vanzin' has created a pull request for

[jira] [Assigned] (SPARK-22788) HdfsUtils.getOutputStream uses non-existent Hadoop conf "hdfs.append.support"

[ https://issues.apache.org/jira/browse/SPARK-22788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22788: Assignee: Apache Spark > HdfsUtils.getOutputStream uses non-existent Hadoop conf "hdfs.app

[jira] [Created] (SPARK-22788) HdfsUtils.getOutputStream uses non-existent Hadoop conf "hdfs.append.support"

Marcelo Vanzin created SPARK-22788: -- Summary: HdfsUtils.getOutputStream uses non-existent Hadoop conf "hdfs.append.support" Key: SPARK-22788 URL: https://issues.apache.org/jira/browse/SPARK-22788 Pro

[jira] [Updated] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Cuquemelle updated SPARK-22683: -- Description: While migrating a series of jobs from MR to Spark using dynamicAllocation,

[jira] [Assigned] (SPARK-22787) Add a TPCH query suite

[ https://issues.apache.org/jira/browse/SPARK-22787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22787: Assignee: Apache Spark (was: Xiao Li) > Add a TPCH query suite > -- >

[jira] [Commented] (SPARK-22787) Add a TPCH query suite

[ https://issues.apache.org/jira/browse/SPARK-22787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291486#comment-16291486 ] Apache Spark commented on SPARK-22787: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-22787) Add a TPCH query suite

[ https://issues.apache.org/jira/browse/SPARK-22787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22787: Assignee: Xiao Li (was: Apache Spark) > Add a TPCH query suite > -- >

[jira] [Created] (SPARK-22787) Add a TPCH query suite

Xiao Li created SPARK-22787: --- Summary: Add a TPCH query suite Key: SPARK-22787 URL: https://issues.apache.org/jira/browse/SPARK-22787 Project: Spark Issue Type: Test Components: SQL A

[jira] [Commented] (SPARK-14822) Add lazy executor startup to Mesos Scheduler

[ https://issues.apache.org/jira/browse/SPARK-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291404#comment-16291404 ] Imran Rashid commented on SPARK-14822: -- [~mgummelt] is this still relevant? seems l

[jira] [Resolved] (SPARK-16496) Add wholetext as option for reading text in SQL.

[ https://issues.apache.org/jira/browse/SPARK-16496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-16496. - Resolution: Fixed Assignee: Prashant Sharma Fix Version/s: 2.3.0 > Add wholetext as optio

[jira] [Commented] (SPARK-22047) HiveExternalCatalogVersionsSuite is Flaky on Jenkins

[ https://issues.apache.org/jira/browse/SPARK-22047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291364#comment-16291364 ] Imran Rashid commented on SPARK-22047: -- [~cloud_fan] think we can close this now? I

[jira] [Assigned] (SPARK-22774) Add compilation check for generated code in TPCDSQuerySuite

[ https://issues.apache.org/jira/browse/SPARK-22774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-22774: --- Assignee: Kazuaki Ishizaki > Add compilation check for generated code in TPCDSQuerySuite > -

[jira] [Resolved] (SPARK-22774) Add compilation check for generated code in TPCDSQuerySuite

[ https://issues.apache.org/jira/browse/SPARK-22774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22774. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19971 [https://githu

[jira] [Commented] (SPARK-22786) only use AppStatusPlugin in history server

[ https://issues.apache.org/jira/browse/SPARK-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291238#comment-16291238 ] Marcelo Vanzin commented on SPARK-22786: As I mentioned in the PR, I don't see wh

[jira] [Assigned] (SPARK-22786) only use AppStatusPlugin in history server

[ https://issues.apache.org/jira/browse/SPARK-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22786: Assignee: Wenchen Fan (was: Apache Spark) > only use AppStatusPlugin in history server >

[jira] [Commented] (SPARK-22786) only use AppStatusPlugin in history server

[ https://issues.apache.org/jira/browse/SPARK-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291234#comment-16291234 ] Apache Spark commented on SPARK-22786: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-22786) only use AppStatusPlugin in history server

[ https://issues.apache.org/jira/browse/SPARK-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22786: Assignee: Apache Spark (was: Wenchen Fan) > only use AppStatusPlugin in history server >

[jira] [Created] (SPARK-22786) only use AppStatusPlugin in history server

Wenchen Fan created SPARK-22786: --- Summary: only use AppStatusPlugin in history server Key: SPARK-22786 URL: https://issues.apache.org/jira/browse/SPARK-22786 Project: Spark Issue Type: Improvem

[jira] [Comment Edited] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291141#comment-16291141 ] Julien Cuquemelle edited comment on SPARK-22683 at 12/14/17 5:09 PM: --

[jira] [Comment Edited] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291141#comment-16291141 ] Julien Cuquemelle edited comment on SPARK-22683 at 12/14/17 5:09 PM: --

[jira] [Assigned] (SPARK-22496) beeline display operation log

[ https://issues.apache.org/jira/browse/SPARK-22496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22496: Assignee: (was: Apache Spark) > beeline display operation log > --

[jira] [Assigned] (SPARK-22496) beeline display operation log

[ https://issues.apache.org/jira/browse/SPARK-22496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22496: Assignee: Apache Spark > beeline display operation log > - > >

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291156#comment-16291156 ] Julien Cuquemelle commented on SPARK-22683: --- I did see SPARK-16158 before openi

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291141#comment-16291141 ] Julien Cuquemelle commented on SPARK-22683: --- [~tgraves], thanks a lot for your

[jira] [Resolved] (SPARK-22785) remove ColumnVector.anyNullsSet

[ https://issues.apache.org/jira/browse/SPARK-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22785. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19980 [https://githu

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291037#comment-16291037 ] Thomas Graves commented on SPARK-22683: --- Another way to approach this is to have a

[jira] [Updated] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Julien Cuquemelle updated SPARK-22683: -- Description: While migrating a series of jobs from MR to Spark using dynamicAllocation,

[jira] [Comment Edited] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280548#comment-16280548 ] Julien Cuquemelle edited comment on SPARK-22683 at 12/14/17 3:25 PM: --

[jira] [Commented] (SPARK-22359) Improve the test coverage of window functions

[ https://issues.apache.org/jira/browse/SPARK-22359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290964#comment-16290964 ] Attila Zsolt Piros commented on SPARK-22359: I would like to join and take th

[jira] [Assigned] (SPARK-22785) remove ColumnVector.anyNullsSet

[ https://issues.apache.org/jira/browse/SPARK-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22785: Assignee: Wenchen Fan (was: Apache Spark) > remove ColumnVector.anyNullsSet > ---

[jira] [Assigned] (SPARK-22785) remove ColumnVector.anyNullsSet

[ https://issues.apache.org/jira/browse/SPARK-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22785: Assignee: Apache Spark (was: Wenchen Fan) > remove ColumnVector.anyNullsSet > ---

[jira] [Commented] (SPARK-22785) remove ColumnVector.anyNullsSet

[ https://issues.apache.org/jira/browse/SPARK-22785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290797#comment-16290797 ] Apache Spark commented on SPARK-22785: -- User 'cloud-fan' has created a pull request

[jira] [Created] (SPARK-22785) remove ColumnVector.anyNullsSet

Wenchen Fan created SPARK-22785: --- Summary: remove ColumnVector.anyNullsSet Key: SPARK-22785 URL: https://issues.apache.org/jira/browse/SPARK-22785 Project: Spark Issue Type: Sub-task

[jira] [Resolved] (SPARK-22775) move dictionary related APIs from ColumnVector to WritableColumnVector

[ https://issues.apache.org/jira/browse/SPARK-22775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-22775. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 19970 [https://githu

[jira] [Commented] (SPARK-22644) Make ML testsuite support StructuredStreaming test

[ https://issues.apache.org/jira/browse/SPARK-22644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290773#comment-16290773 ] Apache Spark commented on SPARK-22644: -- User 'WeichenXu123' has created a pull reque

[jira] [Assigned] (SPARK-22784) Configure reading buffer size in Spark History Server

[ https://issues.apache.org/jira/browse/SPARK-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22784: Assignee: Apache Spark > Configure reading buffer size in Spark History Server > -

[jira] [Resolved] (SPARK-22782) Boost speed, use kafka010 consumer kafka

[ https://issues.apache.org/jira/browse/SPARK-22782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-22782. --- Resolution: Invalid This sounds like some kind of comment for the mailing list. There's no specific

[jira] [Commented] (SPARK-22784) Configure reading buffer size in Spark History Server

[ https://issues.apache.org/jira/browse/SPARK-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290745#comment-16290745 ] Apache Spark commented on SPARK-22784: -- User 'MikhailErofeev' has created a pull req

[jira] [Assigned] (SPARK-22784) Configure reading buffer size in Spark History Server

[ https://issues.apache.org/jira/browse/SPARK-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22784: Assignee: (was: Apache Spark) > Configure reading buffer size in Spark History Server

[jira] [Updated] (SPARK-22784) Configure reading buffer size in Spark History Server

[ https://issues.apache.org/jira/browse/SPARK-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Erofeev updated SPARK-22784: Description: Motivation: Our Spark History Server spends most of the backfill time inside B

[jira] [Updated] (SPARK-22784) Configure reading buffer size in Spark History Server

[ https://issues.apache.org/jira/browse/SPARK-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Erofeev updated SPARK-22784: Attachment: replay-baseline.svg > Configure reading buffer size in Spark History Server > -

[jira] [Commented] (SPARK-22752) FileNotFoundException while reading from Kafka

[ https://issues.apache.org/jira/browse/SPARK-22752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290659#comment-16290659 ] Marco Gaido commented on SPARK-22752: - thanks [~zsxwing]. You are right. I am closing

[jira] [Updated] (SPARK-22784) Increase reading buffer size in Spark History Server

[ https://issues.apache.org/jira/browse/SPARK-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Erofeev updated SPARK-22784: Affects Version/s: (was: 2.0.0) 2.2.1 Target Version/s: (w

[jira] [Updated] (SPARK-22784) Configure reading buffer size in Spark History Server

[ https://issues.apache.org/jira/browse/SPARK-22784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Erofeev updated SPARK-22784: Summary: Configure reading buffer size in Spark History Server (was: Increase reading buff

[jira] [Resolved] (SPARK-22752) FileNotFoundException while reading from Kafka

[ https://issues.apache.org/jira/browse/SPARK-22752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido resolved SPARK-22752. - Resolution: Duplicate > FileNotFoundException while reading from Kafka >

[jira] [Created] (SPARK-22784) Increase reading buffer size in Spark History Server

Mikhail Erofeev created SPARK-22784: --- Summary: Increase reading buffer size in Spark History Server Key: SPARK-22784 URL: https://issues.apache.org/jira/browse/SPARK-22784 Project: Spark Is

[jira] [Comment Edited] (SPARK-22783) event log directory(spark-history) filled by large .inprogress files for spark streaming applications

[ https://issues.apache.org/jira/browse/SPARK-22783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290620#comment-16290620 ] omkar kankalapati edited comment on SPARK-22783 at 12/14/17 9:39 AM: --

[jira] [Commented] (SPARK-22783) event log directory(spark-history) filled by large .inprogress files for spark streaming applications

[ https://issues.apache.org/jira/browse/SPARK-22783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290620#comment-16290620 ] omkar kankalapati commented on SPARK-22783: --- EventLoggingListener (org.apache.s

[jira] [Created] (SPARK-22783) event log directory(spark-history) filled by large .inprogress files for spark streaming applications

omkar kankalapati created SPARK-22783: - Summary: event log directory(spark-history) filled by large .inprogress files for spark streaming applications Key: SPARK-22783 URL: https://issues.apache.org/jira/brows

[jira] [Updated] (SPARK-22782) Boost speed, use kafka010 consumer kafka

[ https://issues.apache.org/jira/browse/SPARK-22782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] licun updated SPARK-22782: -- Description: We use spark structured streaming to consumer kafka, but we find the consumer speed is too sl

[jira] [Updated] (SPARK-22782) Boost speed, use kafka010 consumer kafka

[ https://issues.apache.org/jira/browse/SPARK-22782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] licun updated SPARK-22782: -- Description: We use spark structured streaming to consumer kafka, but we find the consumer speed is too sl

[jira] [Created] (SPARK-22782) Boost speed, use kafka010 consumer kafka

licun created SPARK-22782: - Summary: Boost speed, use kafka010 consumer kafka Key: SPARK-22782 URL: https://issues.apache.org/jira/browse/SPARK-22782 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-22771) SQL concat for binary

[ https://issues.apache.org/jira/browse/SPARK-22771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22771: Assignee: Apache Spark > SQL concat for binary > -- > >

[jira] [Assigned] (SPARK-22771) SQL concat for binary

[ https://issues.apache.org/jira/browse/SPARK-22771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-22771: Assignee: (was: Apache Spark) > SQL concat for binary > -- > >

[jira] [Commented] (SPARK-22771) SQL concat for binary

[ https://issues.apache.org/jira/browse/SPARK-22771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290551#comment-16290551 ] Apache Spark commented on SPARK-22771: -- User 'maropu' has created a pull request for

[jira] [Commented] (SPARK-22660) Use position() and limit() to fix ambiguity issue in scala-2.12

[ https://issues.apache.org/jira/browse/SPARK-22660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290518#comment-16290518 ] liyunzhang commented on SPARK-22660: [~srowen]: there is another modification about

[jira] [Commented] (SPARK-22660) Use position() and limit() to fix ambiguity issue in scala-2.12

[ https://issues.apache.org/jira/browse/SPARK-22660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290514#comment-16290514 ] Apache Spark commented on SPARK-22660: -- User 'kellyzly' has created a pull request f