[jira] [Created] (SPARK-21635) ACOS(2) and ASIN(2) should be null

2017-08-03 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-21635: --- Summary: ACOS(2) and ASIN(2) should be null Key: SPARK-21635 URL: https://issues.apache.org/jira/browse/SPARK-21635 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2017-08-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113974#comment-16113974 ] Felix Cheung commented on SPARK-15799: -- we submitted 2.2.0 release to CRAN and got some comment that

[jira] [Created] (SPARK-21634) Change OneRowRelation from a case object to case class

2017-08-03 Thread Reynold Xin (JIRA)
Reynold Xin created SPARK-21634: --- Summary: Change OneRowRelation from a case object to case class Key: SPARK-21634 URL: https://issues.apache.org/jira/browse/SPARK-21634 Project: Spark Issue

[jira] [Updated] (SPARK-21626) The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.

2017-08-03 Thread Gu Chao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gu Chao updated SPARK-21626: Summary: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. (was:

[jira] [Updated] (SPARK-21626) "WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"

2017-08-03 Thread Gu Chao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gu Chao updated SPARK-21626: Description: After starting spark-shell, It outputs: {code:none} Setting default log level to "WARN". To

[jira] [Updated] (SPARK-21626) "WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"

2017-08-03 Thread Gu Chao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gu Chao updated SPARK-21626: Description: After starting spark-shell, It outputs: Setting default log level to "WARN". To adjust

[jira] [Created] (SPARK-21633) Unary Transformer in Python

2017-08-03 Thread Ajay Saini (JIRA)
Ajay Saini created SPARK-21633: -- Summary: Unary Transformer in Python Key: SPARK-21633 URL: https://issues.apache.org/jira/browse/SPARK-21633 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

2017-08-03 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113806#comment-16113806 ] Saisai Shao commented on SPARK-21618: - [~benmayne] If you try the master branch of Spark with

[jira] [Commented] (SPARK-21624) Optimize communication cost of RF/GBT/DT

2017-08-03 Thread Peng Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113793#comment-16113793 ] Peng Meng commented on SPARK-21624: --- Thanks [~mlnick], use Vector and compress is reasonable. I will

[jira] [Created] (SPARK-21632) There is no need to make attempts for createDirectory if the dir had existed

2017-08-03 Thread liuzhaokun (JIRA)
liuzhaokun created SPARK-21632: -- Summary: There is no need to make attempts for createDirectory if the dir had existed Key: SPARK-21632 URL: https://issues.apache.org/jira/browse/SPARK-21632 Project:

[jira] [Commented] (SPARK-21627) analyze hive table compute stats for columns with mixed case exception

2017-08-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113778#comment-16113778 ] Liang-Chi Hsieh commented on SPARK-21627: - I think it is just solved by SPARK-21599. > analyze

[jira] [Commented] (SPARK-21630) Pmod should not throw a divide by zero exception

2017-08-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113777#comment-16113777 ] Liang-Chi Hsieh commented on SPARK-21630: - Maybe duplicate to SPARK-21205? > Pmod should not

[jira] [Commented] (SPARK-21631) Building Spark with SBT unsuccessful when source code in Mllib is modified, But with MVN is ok

2017-08-03 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113775#comment-16113775 ] Liang-Chi Hsieh commented on SPARK-21631: - Looks like just it is not compliant with Spark code

[jira] [Commented] (SPARK-20870) Update the output of spark-sql -H

2017-08-03 Thread Bravo Zhang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113773#comment-16113773 ] Bravo Zhang commented on SPARK-20870: - Hi [~smilegator], I can't find the code handling the help

[jira] [Comment Edited] (SPARK-21629) OR nullability is incorrect

2017-08-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113752#comment-16113752 ] Takeshi Yamamuro edited comment on SPARK-21629 at 8/4/17 12:58 AM: ---

[jira] [Commented] (SPARK-21629) OR nullability is incorrect

2017-08-03 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113752#comment-16113752 ] Takeshi Yamamuro commented on SPARK-21629: -- What's a concrete query and result example? It is

[jira] [Created] (SPARK-21631) Building Spark with SBT unsuccessful when source code in Mllib is modified, But with MVN is ok

2017-08-03 Thread Sean Wong (JIRA)
Sean Wong created SPARK-21631: - Summary: Building Spark with SBT unsuccessful when source code in Mllib is modified, But with MVN is ok Key: SPARK-21631 URL: https://issues.apache.org/jira/browse/SPARK-21631

[jira] [Commented] (SPARK-20812) Add Mesos Secrets support to the spark dispatcher

2017-08-03 Thread Arthur Rand (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113731#comment-16113731 ] Arthur Rand commented on SPARK-20812: - https://github.com/apache/spark/pull/18837 > Add Mesos

[jira] [Commented] (SPARK-19870) Repeatable deadlock on BlockInfoManager and TorrentBroadcast

2017-08-03 Thread David Lewis (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113727#comment-16113727 ] David Lewis commented on SPARK-19870: - I think I'm hitting a similar bug, here are two stack traces

[jira] [Commented] (SPARK-18406) Race between end-of-task and completion iterator read lock release

2017-08-03 Thread Taichi Sano (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113702#comment-16113702 ] Taichi Sano commented on SPARK-18406: - Hello, I am experiencing an issue very similar to this. I am

[jira] [Commented] (SPARK-20853) spark.ui.reverseProxy=true leads to hanging communication to master

2017-08-03 Thread Josh Bacon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113670#comment-16113670 ] Josh Bacon commented on SPARK-20853: For the record, I'm experiencing the exact same behavior as

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-08-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113658#comment-16113658 ] Marcelo Vanzin commented on SPARK-19112: Yes, we can't merge the PR until Facebook re-licenses

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-08-03 Thread Adam Kennedy (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113653#comment-16113653 ] Adam Kennedy commented on SPARK-19112: -- Will this be impacted by LEGAL-303? zstd-jni embeds zstd

[jira] [Commented] (SPARK-21478) Unpersist a DF also unpersists related DFs

2017-08-03 Thread Roberto Mirizzi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113635#comment-16113635 ] Roberto Mirizzi commented on SPARK-21478: - Hi [~smilegator] Is that documented somewhere? >

[jira] [Resolved] (SPARK-21618) http(s) not accepted in spark-submit jar uri

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21618. --- Resolution: Duplicate > http(s) not accepted in spark-submit jar uri >

[jira] [Commented] (SPARK-21595) introduction of spark.sql.windowExec.buffer.spill.threshold in spark 2.2 breaks existing workflow

2017-08-03 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113622#comment-16113622 ] Tejas Patil commented on SPARK-21595: - [~hvanhovell] : I am fine with either options you mentioned.

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113605#comment-16113605 ] Mark Hamstra commented on SPARK-21619: -- But part of the point of the split in my half-baked example

[jira] [Updated] (SPARK-21617) ALTER TABLE...ADD COLUMNS broken in Hive 2.1 for DS tables

2017-08-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-21617: --- Summary: ALTER TABLE...ADD COLUMNS broken in Hive 2.1 for DS tables (was: ALTER TABLE...ADD

[jira] [Commented] (SPARK-21617) ALTER TABLE...ADD COLUMNS creates invalid metadata in Hive metastore for DS tables

2017-08-03 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113602#comment-16113602 ] Marcelo Vanzin commented on SPARK-21617: Here's the full test error from our internal build

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113598#comment-16113598 ] Reynold Xin commented on SPARK-21619: - Just look at structured streaming. That eould be one example.

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113593#comment-16113593 ] Reynold Xin commented on SPARK-21619: - Just generate different physical plan? > Fail the

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113586#comment-16113586 ] Mark Hamstra commented on SPARK-21619: -- Or you can just enlighten me on how one should design a

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113579#comment-16113579 ] Reynold Xin commented on SPARK-21619: - Ok so we are good with this one. Sorry I don't see why this

[jira] [Comment Edited] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113555#comment-16113555 ] Mark Hamstra edited comment on SPARK-21619 at 8/3/17 10:01 PM: --- Yes, I

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113571#comment-16113571 ] Mark Hamstra commented on SPARK-21619: -- _"Why would you want to execute multiple semantically

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113555#comment-16113555 ] Mark Hamstra commented on SPARK-21619: -- Yes, I absolutely understand that this issue and PR are

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113538#comment-16113538 ] Reynold Xin commented on SPARK-21619: - Also self-joins are very difficult to handle. They have

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113536#comment-16113536 ] Reynold Xin commented on SPARK-21619: - Mark that's a great point but you are going into the

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113526#comment-16113526 ] Mark Hamstra commented on SPARK-21619: -- Two reason, mostly: 1) To provide better guarantees that

[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE

2017-08-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113519#comment-16113519 ] Burak Yavuz commented on SPARK-21588: - that's what I was proposing. `null` seemed more familiar than

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113516#comment-16113516 ] Reynold Xin commented on SPARK-21619: - Sorry I don't understand your question or point at all. Why

[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE

2017-08-03 Thread Anton Okolnychyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113515#comment-16113515 ] Anton Okolnychyi commented on SPARK-21588: -- Sure, but the converter will not be called if the

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113510#comment-16113510 ] Mark Hamstra commented on SPARK-21619: -- Ok, but my point is that if plans are to be canonicalized

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113498#comment-16113498 ] Reynold Xin commented on SPARK-21619: - Canonicalized plan is used for semantic comparison. This has

[jira] [Commented] (SPARK-21619) Fail the execution of canonicalized plans explicitly

2017-08-03 Thread Mark Hamstra (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113494#comment-16113494 ] Mark Hamstra commented on SPARK-21619: -- Can you provide a little more context, Reynold, since on its

[jira] [Created] (SPARK-21630) Pmod should not throw a divide by zero exception

2017-08-03 Thread Herman van Hovell (JIRA)
Herman van Hovell created SPARK-21630: - Summary: Pmod should not throw a divide by zero exception Key: SPARK-21630 URL: https://issues.apache.org/jira/browse/SPARK-21630 Project: Spark

[jira] [Created] (SPARK-21629) OR nullability is incorrect

2017-08-03 Thread Herman van Hovell (JIRA)
Herman van Hovell created SPARK-21629: - Summary: OR nullability is incorrect Key: SPARK-21629 URL: https://issues.apache.org/jira/browse/SPARK-21629 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE

2017-08-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113422#comment-16113422 ] Burak Yavuz commented on SPARK-21588: - [~vinodkc] [~aokolnychyi] It happens when the config has a

[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE

2017-08-03 Thread Anton Okolnychyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113412#comment-16113412 ] Anton Okolnychyi commented on SPARK-21588: -- I did not manage to reproduce this. I tried: {code}

[jira] [Commented] (SPARK-15799) Release SparkR on CRAN

2017-08-03 Thread Brendan Dwyer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113404#comment-16113404 ] Brendan Dwyer commented on SPARK-15799: --- Is there any update on this? > Release SparkR on CRAN >

[jira] [Commented] (SPARK-21595) introduction of spark.sql.windowExec.buffer.spill.threshold in spark 2.2 breaks existing workflow

2017-08-03 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113388#comment-16113388 ] Herman van Hovell commented on SPARK-21595: --- The old and the new code are not exactly the same.

[jira] [Commented] (SPARK-21097) Dynamic allocation will preserve cached data

2017-08-03 Thread Brad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113264#comment-16113264 ] Brad commented on SPARK-21097: -- I'm still working on thoroughly benchmarking and testing this change. If

[jira] [Updated] (SPARK-21453) Cached Kafka consumer may be closed too early

2017-08-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21453: - Summary: Cached Kafka consumer may be closed too early (was: Streaming kafka source (structured

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

2017-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: (was: Nuochen Lyu) > Speculative task that got CommitDenied exception

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

2017-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: Nuochen Lyu > Speculative task that got CommitDenied exception shows up as

[jira] [Resolved] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

2017-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20713. --- Resolution: Fixed Assignee: Nuochen Lyu Fix Version/s: 2.3.0 > Speculative

[jira] [Commented] (SPARK-21453) Streaming kafka source (structured spark)

2017-08-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113224#comment-16113224 ] Shixiong Zhu commented on SPARK-21453: -- Reopened this one. There might be some bug in caching Kafka

[jira] [Reopened] (SPARK-21453) Streaming kafka source (structured spark)

2017-08-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-21453: -- > Streaming kafka source (structured spark) > - > >

[jira] [Commented] (SPARK-21453) Streaming kafka source (structured spark)

2017-08-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113220#comment-16113220 ] Shixiong Zhu commented on SPARK-21453: -- [~ppanero] could you create a new ticket for the Kafka

[jira] [Commented] (SPARK-21453) Streaming kafka source (structured spark)

2017-08-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113218#comment-16113218 ] Shixiong Zhu commented on SPARK-21453: -- I'm aware of the Kafka producer issue. Right now a

[jira] [Commented] (SPARK-21367) R older version of Roxygen2 on Jenkins

2017-08-03 Thread Felix Cheung (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113145#comment-16113145 ] Felix Cheung commented on SPARK-21367: -- still seeing it Warning messages: 1: In

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

2017-08-03 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113133#comment-16113133 ] John Zhuge commented on SPARK-21618: We have not backported HADOOP-14383 to CDH5. > http(s) not

[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

2017-08-03 Thread Miles Crawford (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113067#comment-16113067 ] Miles Crawford edited comment on SPARK-18838 at 8/3/17 5:13 PM: We have

[jira] [Commented] (SPARK-17669) Strange behavior using Datasets

2017-08-03 Thread Miles Crawford (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113083#comment-16113083 ] Miles Crawford commented on SPARK-17669: This UI behavior is caused by SPARK-18838 - events are

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

2017-08-03 Thread Miles Crawford (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113067#comment-16113067 ] Miles Crawford commented on SPARK-18838: We have disabled our eventlog listener, which is

[jira] [Comment Edited] (SPARK-18838) High latency of event processing for large jobs

2017-08-03 Thread Miles Crawford (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113067#comment-16113067 ] Miles Crawford edited comment on SPARK-18838 at 8/3/17 4:51 PM: We have

[jira] [Resolved] (SPARK-21599) Collecting column statistics for datasource tables may fail with java.util.NoSuchElementException

2017-08-03 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-21599. - Resolution: Fixed Fix Version/s: 2.3.0 > Collecting column statistics for datasource tables may

[jira] [Commented] (SPARK-21598) Collect usability/events information from Spark History Server

2017-08-03 Thread Eric Vandenberg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113040#comment-16113040 ] Eric Vandenberg commented on SPARK-21598: - [~steve_l] Do you have any input / thoughts here?

[jira] [Commented] (SPARK-21143) Fail to fetch blocks >1MB in size in presence of conflicting Netty version

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112974#comment-16112974 ] Sean Owen commented on SPARK-21143: --- [~baz33] please see the JIRA and make the change if you want to

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112973#comment-16112973 ] Sean Owen commented on SPARK-19552: --- You will still have to make Spark work with 4.1.x even if it's

[jira] [Resolved] (SPARK-21628) Explicitly specify Java version in maven compiler plugin so IntelliJ imports project correctly

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21628. --- Resolution: Duplicate > Explicitly specify Java version in maven compiler plugin so IntelliJ imports

[jira] [Updated] (SPARK-21627) analyze hive table compute stats for columns with mixed case exception

2017-08-03 Thread Bogdan Raducanu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21627: Summary: analyze hive table compute stats for columns with mixed case exception (was:

[jira] [Created] (SPARK-21628) Explicitly specify Java version in maven compiler plugin so IntelliJ imports project correctly

2017-08-03 Thread Andrew Ray (JIRA)
Andrew Ray created SPARK-21628: -- Summary: Explicitly specify Java version in maven compiler plugin so IntelliJ imports project correctly Key: SPARK-21628 URL: https://issues.apache.org/jira/browse/SPARK-21628

[jira] [Commented] (SPARK-19552) Upgrade Netty version to 4.1.8 final

2017-08-03 Thread BDeus (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112879#comment-16112879 ] BDeus commented on SPARK-19552: --- I have the same problem with gRPC too, if we don't want upgrade to 4.1.x,

[jira] [Commented] (SPARK-21143) Fail to fetch blocks >1MB in size in presence of conflicting Netty version

2017-08-03 Thread Basile Deustua (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112873#comment-16112873 ] Basile Deustua commented on SPARK-21143: I have the exact same issue with io.grpc which heavily

[jira] [Commented] (SPARK-21570) File __spark_libs__XXX.zip does not exist on networked file system w/ yarn

2017-08-03 Thread Albert Chu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112868#comment-16112868 ] Albert Chu commented on SPARK-21570: There's no scheme. Just using "file://" to treat like a local

[jira] [Updated] (SPARK-21627) hive compute stats for columns exception with column name camel case

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-21627: -- Affects Version/s: (was: 3.0.0) 2.3.0 master = 2.3.0 right now > hive

[jira] [Commented] (SPARK-21627) hive compute stats for columns exception with column name camel case

2017-08-03 Thread Bogdan Raducanu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112851#comment-16112851 ] Bogdan Raducanu commented on SPARK-21627: - I expect it fails only in master branch. That's why

[jira] [Commented] (SPARK-21627) hive compute stats for columns exception with column name camel case

2017-08-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112821#comment-16112821 ] Hyukjin Kwon commented on SPARK-21627: -- Would you mind fixing {{Affects Version/s:}}? I guess we

[jira] [Assigned] (SPARK-21602) Add map_keys and map_values functions to R

2017-08-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-21602: Assignee: Hyukjin Kwon > Add map_keys and map_values functions to R >

[jira] [Resolved] (SPARK-21602) Add map_keys and map_values functions to R

2017-08-03 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21602. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18809

[jira] [Updated] (SPARK-21627) hive compute stats for columns exception with column name camel case

2017-08-03 Thread Bogdan Raducanu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21627: Description: {code} sql("create table tabel1(b int) partitioned by (partColumn int)")

[jira] [Updated] (SPARK-21627) hive compute stats for columns exception with column name camel case

2017-08-03 Thread Bogdan Raducanu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bogdan Raducanu updated SPARK-21627: Description: {code} sql("create table tabel1(b int) partitioned by (partColumn int)")

[jira] [Created] (SPARK-21627) hive compute stats for columns exception with column name camel case

2017-08-03 Thread Bogdan Raducanu (JIRA)
Bogdan Raducanu created SPARK-21627: --- Summary: hive compute stats for columns exception with column name camel case Key: SPARK-21627 URL: https://issues.apache.org/jira/browse/SPARK-21627 Project:

[jira] [Commented] (SPARK-21086) CrossValidator, TrainValidationSplit should preserve all models after fitting

2017-08-03 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112623#comment-16112623 ] Nick Pentreath commented on SPARK-21086: I just want to understand _why_ folks want to keep all

[jira] [Commented] (SPARK-20922) Unsafe deserialization in Spark LauncherConnection

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112602#comment-16112602 ] Sean Owen commented on SPARK-20922: --- If you'd email a suggested CVE description to

[jira] [Commented] (SPARK-20922) Unsafe deserialization in Spark LauncherConnection

2017-08-03 Thread Aditya Sharad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112600#comment-16112600 ] Aditya Sharad commented on SPARK-20922: --- Apologies for the delay in getting back to you. I believe

[jira] [Issue Comment Deleted] (SPARK-21625) sqrt(negative number) should be null

2017-08-03 Thread panbingkun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] panbingkun updated SPARK-21625: --- Comment: was deleted (was: case class Sqrt(child: Expression) extends UnaryMathExpression(math.sqrt,

[jira] [Commented] (SPARK-21625) sqrt(negative number) should be null

2017-08-03 Thread panbingkun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112571#comment-16112571 ] panbingkun commented on SPARK-21625: case class Sqrt(child: Expression) extends

[jira] [Assigned] (SPARK-21605) Let IntelliJ IDEA correctly detect Language level and Target byte code version

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21605: - Assignee: Chang chen > Let IntelliJ IDEA correctly detect Language level and Target byte code

[jira] [Resolved] (SPARK-21605) Let IntelliJ IDEA correctly detect Language level and Target byte code version

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21605. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18808

[jira] [Commented] (SPARK-21626) "WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"

2017-08-03 Thread Gu Chao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112564#comment-16112564 ] Gu Chao commented on SPARK-21626: - [~srowen] I can solve this problem, but I do not know why.

[jira] [Comment Edited] (SPARK-21626) "WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"

2017-08-03 Thread Gu Chao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112564#comment-16112564 ] Gu Chao edited comment on SPARK-21626 at 8/3/17 10:57 AM: -- [~srowen] I can solve

[jira] [Resolved] (SPARK-21626) "WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21626. --- Resolution: Not A Problem Not a problem, not even specific to Spark. It means what it says, and is

[jira] [Created] (SPARK-21626) "WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"

2017-08-03 Thread Gu Chao (JIRA)
Gu Chao created SPARK-21626: --- Summary: "WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable" Key: SPARK-21626 URL:

[jira] [Comment Edited] (SPARK-21618) http(s) not accepted in spark-submit jar uri

2017-08-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112529#comment-16112529 ] Steve Loughran edited comment on SPARK-21618 at 8/3/17 10:09 AM: - If

[jira] [Resolved] (SPARK-21611) Error class name for log in several classes.

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21611. --- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18816

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

2017-08-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112529#comment-16112529 ] Steve Loughran commented on SPARK-21618: If you're relying on hadoop-common to provide the FS

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

2017-08-03 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112524#comment-16112524 ] Sean Owen commented on SPARK-21618: --- I see, so this may really not work in general. At least we'd

[jira] [Commented] (SPARK-21618) http(s) not accepted in spark-submit jar uri

2017-08-03 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112520#comment-16112520 ] Steve Loughran commented on SPARK-21618: BTW, we haven't backported HADOOP-14383 into HDP; don't

  1   2   >