[jira] [Updated] (SPARK-24828) Incompatible parquet formats - java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary

2018-07-16 Thread Romeo Kienzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romeo Kienzer updated SPARK-24828: -- Description: As requested by [~hyukjin.kwon] here a new issue - related issue can be found

[jira] [Commented] (SPARK-17557) SQL query on parquet table java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary

2018-07-16 Thread Romeo Kienzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546059#comment-16546059 ] Romeo Kienzer commented on SPARK-17557: --- Dear [~hyukjin.kwon] - I've done so - new issue is

[jira] [Created] (SPARK-24828) Incompatible parquet formats - java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary

2018-07-16 Thread Romeo Kienzer (JIRA)
Romeo Kienzer created SPARK-24828: - Summary: Incompatible parquet formats - java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainLongDictionary Key: SPARK-24828

[jira] [Commented] (SPARK-24568) Code refactoring for DataType equalsXXX methods

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546009#comment-16546009 ] Apache Spark commented on SPARK-24568: -- User 'swapnilushinde' has created a pull request for this

[jira] [Assigned] (SPARK-24568) Code refactoring for DataType equalsXXX methods

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24568: Assignee: (was: Apache Spark) > Code refactoring for DataType equalsXXX methods >

[jira] [Assigned] (SPARK-24568) Code refactoring for DataType equalsXXX methods

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24568: Assignee: Apache Spark > Code refactoring for DataType equalsXXX methods >

[jira] [Updated] (SPARK-24402) Optimize `In` expression when only one element in the collection or collection is empty

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-24402: - Fix Version/s: (was: 2.4.0) > Optimize `In` expression when only one element in the

[jira] [Reopened] (SPARK-24402) Optimize `In` expression when only one element in the collection or collection is empty

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-24402: -- This was reverted. > Optimize `In` expression when only one element in the collection or >

[jira] [Resolved] (SPARK-24798) sortWithinPartitions(xx) will failed in java.lang.NullPointerException

2018-07-16 Thread shengyao piao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shengyao piao resolved SPARK-24798. --- Resolution: Not A Problem > sortWithinPartitions(xx) will failed in

[jira] [Commented] (SPARK-24798) sortWithinPartitions(xx) will failed in java.lang.NullPointerException

2018-07-16 Thread shengyao piao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545973#comment-16545973 ] shengyao piao commented on SPARK-24798: --- Hi [~mahmoudmahdi24] , [~dmateusp] Thank you! It's

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-16 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545962#comment-16545962 ] Saisai Shao commented on SPARK-24615: - Sorry [~tgraves] for the late response. Yes,  when requesting

[jira] [Commented] (SPARK-17557) SQL query on parquet table java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545961#comment-16545961 ] Hyukjin Kwon commented on SPARK-17557: -- Please go ahead but I would alternatively open a separate

[jira] [Commented] (SPARK-23255) Add user guide and examples for DataFrame image reading functions

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545960#comment-16545960 ] Hyukjin Kwon commented on SPARK-23255: -- Please go ahead. > Add user guide and examples for

[jira] [Resolved] (SPARK-24644) Pyarrow exception while running pandas_udf on pyspark 2.3.1

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24644. -- Resolution: Invalid Let me leave this resolved. Please reopen this if the same issue exists

[jira] [Resolved] (SPARK-20220) Add thrift scheduling pool config in scheduling docs

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-20220. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21778

[jira] [Assigned] (SPARK-20220) Add thrift scheduling pool config in scheduling docs

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-20220: Assignee: Miklos Christine > Add thrift scheduling pool config in scheduling docs >

[jira] [Resolved] (SPARK-23259) Clean up legacy code around hive external catalog

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23259. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21780

[jira] [Assigned] (SPARK-23259) Clean up legacy code around hive external catalog

2018-07-16 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23259: Assignee: Feng Liu > Clean up legacy code around hive external catalog >

[jira] [Updated] (SPARK-21481) Add indexOf method in ml.feature.HashingTF similar to mllib.feature.HashingTF

2018-07-16 Thread chenzhiming (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenzhiming updated SPARK-21481: Attachment: idea64.exe > Add indexOf method in ml.feature.HashingTF similar to

[jira] [Updated] (SPARK-21481) Add indexOf method in ml.feature.HashingTF similar to mllib.feature.HashingTF

2018-07-16 Thread chenzhiming (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chenzhiming updated SPARK-21481: Attachment: (was: idea64.exe) > Add indexOf method in ml.feature.HashingTF similar to

[jira] [Updated] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-16 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-24615: -- Summary: Accelerator-aware task scheduling for Spark (was: Accelerator aware task scheduling

[jira] [Updated] (SPARK-24826) Self-Join not working in Apache Spark 2.2.2

2018-07-16 Thread Michael Yannakopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Yannakopoulos updated SPARK-24826: -- Description: Running a self-join against a table derived from a parquet file

[jira] [Created] (SPARK-24827) Some memory waste in History Server by strings in AccumulableInfo objects

2018-07-16 Thread Misha Dmitriev (JIRA)
Misha Dmitriev created SPARK-24827: -- Summary: Some memory waste in History Server by strings in AccumulableInfo objects Key: SPARK-24827 URL: https://issues.apache.org/jira/browse/SPARK-24827

[jira] [Updated] (SPARK-24826) Self-Join not working in Apache Spark 2.2.2

2018-07-16 Thread Michael Yannakopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Yannakopoulos updated SPARK-24826: -- Description: Running a self-join against a table derived from a parquet file

[jira] [Updated] (SPARK-24826) Self-Join not working in Apache Spark 2.2.2

2018-07-16 Thread Michael Yannakopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Yannakopoulos updated SPARK-24826: -- Attachment: part-0-48210471-3088-4cee-8670-a332444bae66-c000.gz.parquet >

[jira] [Created] (SPARK-24826) Self-Join not working in Apache Spark 2.2.2

2018-07-16 Thread Michael Yannakopoulos (JIRA)
Michael Yannakopoulos created SPARK-24826: - Summary: Self-Join not working in Apache Spark 2.2.2 Key: SPARK-24826 URL: https://issues.apache.org/jira/browse/SPARK-24826 Project: Spark

[jira] [Commented] (SPARK-24801) Empty byte[] arrays in spark.network.sasl.SaslEncryption$EncryptedMessage can waste a lot of memory

2018-07-16 Thread Misha Dmitriev (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545820#comment-16545820 ] Misha Dmitriev commented on SPARK-24801: Correct, there are indeed 40583 instances of

[jira] [Resolved] (SPARK-24402) Optimize `In` expression when only one element in the collection or collection is empty

2018-07-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24402. - Resolution: Fixed Fix Version/s: 2.4.0 > Optimize `In` expression when only one element in the

[jira] [Updated] (SPARK-24825) [K8S][TEST] Kubernetes integration tests don't trace the maven project dependency structure

2018-07-16 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah updated SPARK-24825: --- Issue Type: Bug (was: Improvement) > [K8S][TEST] Kubernetes integration tests don't trace the

[jira] [Created] (SPARK-24825) [K8S][TEST] Kubernetes integration tests don't trace the maven project dependency structure

2018-07-16 Thread Matt Cheah (JIRA)
Matt Cheah created SPARK-24825: -- Summary: [K8S][TEST] Kubernetes integration tests don't trace the maven project dependency structure Key: SPARK-24825 URL: https://issues.apache.org/jira/browse/SPARK-24825

[jira] [Updated] (SPARK-24825) [K8S][TEST] Kubernetes integration tests don't trace the maven project dependency structure

2018-07-16 Thread Matt Cheah (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Cheah updated SPARK-24825: --- Priority: Critical (was: Major) > [K8S][TEST] Kubernetes integration tests don't trace the maven

[jira] [Resolved] (SPARK-24805) Don't ignore files without .avro extension by default

2018-07-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24805. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > Don't ignore files

[jira] [Updated] (SPARK-23901) Data Masking Functions

2018-07-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23901: Fix Version/s: (was: 2.4.0) > Data Masking Functions > -- > >

[jira] [Resolved] (SPARK-23901) Data Masking Functions

2018-07-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23901. - Resolution: Won't Fix > Data Masking Functions > -- > > Key:

[jira] [Reopened] (SPARK-23901) Data Masking Functions

2018-07-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-23901: - > Data Masking Functions > -- > > Key: SPARK-23901 >

[jira] [Commented] (SPARK-24801) Empty byte[] arrays in spark.network.sasl.SaslEncryption$EncryptedMessage can waste a lot of memory

2018-07-16 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545676#comment-16545676 ] Imran Rashid commented on SPARK-24801: -- I'm surprised there are so many {{EncryptedMessage}}

[jira] [Commented] (SPARK-16617) Upgrade to Avro 1.8.x

2018-07-16 Thread Thomas Omans (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545625#comment-16545625 ] Thomas Omans commented on SPARK-16617: -- My code is getting the Schema.getLogicalType bug using

[jira] [Commented] (SPARK-24644) Pyarrow exception while running pandas_udf on pyspark 2.3.1

2018-07-16 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545589#comment-16545589 ] Bryan Cutler commented on SPARK-24644: -- [~helkhalfi], the error in the stack trace is coming from

[jira] [Commented] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

2018-07-16 Thread Iqbal Singh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545577#comment-16545577 ] Iqbal Singh commented on SPARK-24295: - Hey [~XuanYuan], We are processing 3000 files every 5

[jira] [Commented] (SPARK-17557) SQL query on parquet table java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary

2018-07-16 Thread Romeo Kienzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545557#comment-16545557 ] Romeo Kienzer commented on SPARK-17557: --- [~jayadevan.m] [~hyukjin.kwon] can you please re-open?

[jira] [Updated] (SPARK-17557) SQL query on parquet table java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary

2018-07-16 Thread Romeo Kienzer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Romeo Kienzer updated SPARK-17557: -- Attachment: a2_m2.parquet.zip > SQL query on parquet table

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-07-16 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545519#comment-16545519 ] Bryan Cutler commented on SPARK-23874: -- [~smilegator], we are aiming to have the Arrow 0.10.0

[jira] [Commented] (SPARK-23901) Data Masking Functions

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545491#comment-16545491 ] Apache Spark commented on SPARK-23901: -- User 'mn-mikke' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-23901) Data Masking Functions

2018-07-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545446#comment-16545446 ] Reynold Xin edited comment on SPARK-23901 at 7/16/18 4:31 PM: -- I actually

[jira] [Commented] (SPARK-23901) Data Masking Functions

2018-07-16 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545446#comment-16545446 ] Reynold Xin commented on SPARK-23901: - I actually feel pretty strongly we should remove them.   >

[jira] [Commented] (SPARK-23901) Data Masking Functions

2018-07-16 Thread Marek Novotny (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545443#comment-16545443 ] Marek Novotny commented on SPARK-23901: --- Is there a consensus on getting the masking functions to

[jira] [Commented] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-07-16 Thread Sanket Reddy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545436#comment-16545436 ] Sanket Reddy commented on SPARK-24787: -- [~vanzin] do you have any suggestions regarding this issue?

[jira] [Assigned] (SPARK-24734) Fix containsNull of Concat for array type.

2018-07-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24734: --- Assignee: Takuya Ueshin > Fix containsNull of Concat for array type. >

[jira] [Resolved] (SPARK-24734) Fix containsNull of Concat for array type.

2018-07-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24734. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21704

[jira] [Commented] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2018-07-16 Thread Valery Khamenya (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545314#comment-16545314 ] Valery Khamenya commented on SPARK-20174: - Ok, I found a combo-workaround that seems to work:

[jira] [Updated] (SPARK-24813) HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive

2018-07-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-24813: -- Affects Version/s: (was: 2.1.3) > HiveExternalCatalogVersionsSuite still flaky; fall back to

[jira] [Commented] (SPARK-24529) Add spotbugs into maven build process

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545284#comment-16545284 ] Apache Spark commented on SPARK-24529: -- User 'wangyum' has created a pull request for this issue:

[jira] [Resolved] (SPARK-18230) MatrixFactorizationModel.recommendProducts throws NoSuchElement exception when the user does not exist

2018-07-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-18230. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21740

[jira] [Commented] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2018-07-16 Thread Valery Khamenya (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545282#comment-16545282 ] Valery Khamenya commented on SPARK-20174: - Guys, I am tracking this issue for quite some time

[jira] [Assigned] (SPARK-18230) MatrixFactorizationModel.recommendProducts throws NoSuchElement exception when the user does not exist

2018-07-16 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-18230: - Assignee: shahid > MatrixFactorizationModel.recommendProducts throws NoSuchElement exception

[jira] [Commented] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-07-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545253#comment-16545253 ] Thomas Graves commented on SPARK-24615: --- [~jerryshao] ^ > Accelerator aware task scheduling for

[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data

2018-07-16 Thread Brad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545240#comment-16545240 ] Brad commented on SPARK-21097: -- Hi [~menelaus] The processing time delay is just a way to simulate

[jira] [Commented] (SPARK-15343) NoClassDefFoundError when initializing Spark with YARN

2018-07-16 Thread Antony (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545225#comment-16545225 ] Antony commented on SPARK-15343: {{--conf spark.hadoop.yarn.timeline-service.enabled=false  is work for

[jira] [Commented] (SPARK-24182) Improve error message for client mode when AM fails

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545167#comment-16545167 ] Apache Spark commented on SPARK-24182: -- User 'wangyum' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20050) Kafka 0.10 DirectStream doesn't commit last processed batch's offset when graceful shutdown

2018-07-16 Thread Sasaki Toru (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sasaki Toru resolved SPARK-20050. - Resolution: Not A Problem > Kafka 0.10 DirectStream doesn't commit last processed batch's

[jira] [Commented] (SPARK-24799) A solution of dealing with data skew in left,right,inner join

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545032#comment-16545032 ] Apache Spark commented on SPARK-24799: -- User 'marymwu' has created a pull request for this issue:

[jira] [Updated] (SPARK-24812) Last Access Time in the table description is not valid

2018-07-16 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith updated SPARK-24812: --- Description: Last Access Time in the table description is not valid,  Test steps: Step 1 -  create a

[jira] [Updated] (SPARK-24812) Last Access Time in the table description is not valid

2018-07-16 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith updated SPARK-24812: --- Attachment: image-2018-07-16-15-38-26-717.png > Last Access Time in the table description is not valid >

[jira] [Updated] (SPARK-24812) Last Access Time in the table description is not valid

2018-07-16 Thread Sujith (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith updated SPARK-24812: --- Attachment: image-2018-07-16-15-37-28-896.png > Last Access Time in the table description is not valid >

[jira] [Issue Comment Deleted] (SPARK-24816) SQL interface support repartitionByRange

2018-07-16 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24816: Comment: was deleted (was: I'm working on.) > SQL interface support repartitionByRange >

[jira] [Assigned] (SPARK-24816) SQL interface support repartitionByRange

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24816: Assignee: Apache Spark > SQL interface support repartitionByRange >

[jira] [Assigned] (SPARK-24816) SQL interface support repartitionByRange

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24816: Assignee: (was: Apache Spark) > SQL interface support repartitionByRange >

[jira] [Commented] (SPARK-24816) SQL interface support repartitionByRange

2018-07-16 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544980#comment-16544980 ] Apache Spark commented on SPARK-24816: -- User 'wangyum' has created a pull request for this issue:

[jira] [Commented] (SPARK-24794) DriverWrapper should have both master addresses in -Dspark.master

2018-07-16 Thread Ecaterina (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544973#comment-16544973 ] Ecaterina commented on SPARK-24794: --- Yes, I also face this problem. Would be nice if somebody could

[jira] [Updated] (SPARK-24816) SQL interface support repartitionByRange

2018-07-16 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24816: Description: SQL interface support {{repartitionByRange}} to improvement data pushdown. I have

[jira] [Created] (SPARK-24824) Make Spark task speculation a per-stage config

2018-07-16 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24824: Summary: Make Spark task speculation a per-stage config Key: SPARK-24824 URL: https://issues.apache.org/jira/browse/SPARK-24824 Project: Spark Issue Type:

[jira] [Created] (SPARK-24823) Cancel a job that contains barrier stage(s) if the barrier tasks don't get launched within a configured time

2018-07-16 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24823: Summary: Cancel a job that contains barrier stage(s) if the barrier tasks don't get launched within a configured time Key: SPARK-24823 URL:

[jira] [Created] (SPARK-24822) Python support for barrier execution mode

2018-07-16 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24822: Summary: Python support for barrier execution mode Key: SPARK-24822 URL: https://issues.apache.org/jira/browse/SPARK-24822 Project: Spark Issue Type: New

[jira] [Created] (SPARK-24821) Fail fast when submitted job compute on a subset of all the partitions for a barrier stage

2018-07-16 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24821: Summary: Fail fast when submitted job compute on a subset of all the partitions for a barrier stage Key: SPARK-24821 URL: https://issues.apache.org/jira/browse/SPARK-24821

[jira] [Created] (SPARK-24820) Fail fast when submitted job contains PartitionPruningRDD in a barrier stage

2018-07-16 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24820: Summary: Fail fast when submitted job contains PartitionPruningRDD in a barrier stage Key: SPARK-24820 URL: https://issues.apache.org/jira/browse/SPARK-24820

[jira] [Created] (SPARK-24819) Fail fast when no enough slots to launch the barrier stage on job submitted

2018-07-16 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24819: Summary: Fail fast when no enough slots to launch the barrier stage on job submitted Key: SPARK-24819 URL: https://issues.apache.org/jira/browse/SPARK-24819 Project:

[jira] [Created] (SPARK-24818) Ensure all the barrier tasks in the same stage are launched together

2018-07-16 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24818: Summary: Ensure all the barrier tasks in the same stage are launched together Key: SPARK-24818 URL: https://issues.apache.org/jira/browse/SPARK-24818 Project: Spark

[jira] [Created] (SPARK-24817) Implement BarrierTaskContext.barrier()

2018-07-16 Thread Jiang Xingbo (JIRA)
Jiang Xingbo created SPARK-24817: Summary: Implement BarrierTaskContext.barrier() Key: SPARK-24817 URL: https://issues.apache.org/jira/browse/SPARK-24817 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-24538) ByteArrayDecimalType support push down to parquet data sources

2018-07-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-24538: Summary: ByteArrayDecimalType support push down to parquet data sources (was:

[jira] [Resolved] (SPARK-24538) ByteArrayDecimalType support push down to the data sources

2018-07-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24538. - Resolution: Fixed Assignee: Yuming Wang Fix Version/s: 2.4.0 Target

[jira] [Assigned] (SPARK-24549) Support DecimalType push down to the parquet data sources

2018-07-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24549: --- Assignee: Yuming Wang > Support DecimalType push down to the parquet data sources >

[jira] [Commented] (SPARK-21791) ORC should support column names with dot

2018-07-16 Thread Furcy Pin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544899#comment-16544899 ] Furcy Pin commented on SPARK-21791: --- Indeed, it works like this. Awesome! thanks!      > ORC

[jira] [Updated] (SPARK-24816) SQL interface support repartitionByRange

2018-07-16 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24816: Attachment: DISTRIBUTE_BY_SORT_BY.png RANGE_DISTRIBUTE_BY_SORT_BY.png > SQL

[jira] [Updated] (SPARK-24558) Driver prints the wrong info in the log when the executor which holds cacheBlock is IDLE.Time-out value displayed is not as per configuration value.

2018-07-16 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sandeep katta updated SPARK-24558: -- Affects Version/s: 2.2.1 2.2.2 > Driver prints the wrong info in the

[jira] [Assigned] (SPARK-24558) Driver prints the wrong info in the log when the executor which holds cacheBlock is IDLE.Time-out value displayed is not as per configuration value.

2018-07-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-24558: --- Assignee: sandeep katta > Driver prints the wrong info in the log when the executor which

[jira] [Resolved] (SPARK-24558) Driver prints the wrong info in the log when the executor which holds cacheBlock is IDLE.Time-out value displayed is not as per configuration value.

2018-07-16 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24558. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21565

[jira] [Updated] (SPARK-24816) SQL interface support repartitionByRange

2018-07-16 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-24816: Description: SQL interface support {{repartitionByRange}} to improvement data pushdown. I have

[jira] [Commented] (SPARK-23874) Upgrade apache/arrow to 0.10.0

2018-07-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16544841#comment-16544841 ] Xiao Li commented on SPARK-23874: - [~bryanc] [~icexelloss] I saw you are working on the JIRA

[jira] [Resolved] (SPARK-24810) Fix paths to resource files in AvroSuite

2018-07-16 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24810. - Resolution: Fixed Assignee: Maxim Gekk Fix Version/s: 2.4.0 > Fix paths to resource