[jira] [Created] (SPARK-15328) Word2Vec import for original binary format

2016-05-14 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-15328: --- Summary: Word2Vec import for original binary format Key: SPARK-15328 URL: https://issues.apache.org/jira/browse/SPARK-15328 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16625) Oracle JDBC table creation fails with ORA-00902: invalid datatype

2016-07-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395307#comment-15395307 ] Yuming Wang commented on SPARK-16625: - {code:java} val jdbcUrl =

[jira] [Commented] (SPARK-16846) read.csv() option: "inferSchema" don't work

2016-08-02 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403592#comment-15403592 ] Yuming Wang commented on SPARK-16846: - You may need to be remove -schema-. The following code works:

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

2017-02-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853784#comment-15853784 ] Yuming Wang commented on SPARK-16441: - set {{spark.dynamicAllocation.maxExecutors}} to a reasonable

[jira] [Updated] (SPARK-16441) Spark application hang when dynamic allocation is enabled

2017-02-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-16441: Attachment: SPARK-16441-yarn-metrics.jpg SPARK-16441-threadDump.jpg

[jira] [Updated] (SPARK-16441) Spark application hang when dynamic allocation is enabled

2017-02-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-16441: Affects Version/s: 2.1.0 > Spark application hang when dynamic allocation is enabled >

[jira] [Updated] (SPARK-16441) Spark application hang when dynamic allocation is enabled

2017-02-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-16441: Attachment: SPARK-16441-compare-apply-PR-16819.zip > Spark application hang when dynamic

[jira] [Commented] (SPARK-16441) Spark application hang when dynamic allocation is enabled

2017-02-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855414#comment-15855414 ] Yuming Wang commented on SPARK-16441: - [~cenyuhai],

[jira] [Updated] (SPARK-19300) Executor is waiting for lock

2017-01-21 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19300: Attachment: WAITING.jpg stderr.jpg [~zsxwing], There are 4 blocks to fetch, but

[jira] [Comment Edited] (SPARK-19300) Executor is waiting for lock

2017-01-21 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833224#comment-15833224 ] Yuming Wang edited comment on SPARK-19300 at 1/22/17 2:04 AM: -- [~zsxwing],

[jira] [Comment Edited] (SPARK-19300) Executor is waiting for lock

2017-01-21 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833224#comment-15833224 ] Yuming Wang edited comment on SPARK-19300 at 1/22/17 2:02 AM: -- [~zsxwing],

[jira] [Comment Edited] (SPARK-19300) Executor is waiting for lock

2017-01-21 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833224#comment-15833224 ] Yuming Wang edited comment on SPARK-19300 at 1/22/17 2:04 AM: -- !WAITING.jpg!

[jira] [Commented] (SPARK-19693) SET mapreduce.job.reduces automatically converted to spark.sql.shuffle.partitions

2017-02-21 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877382#comment-15877382 ] Yuming Wang commented on SPARK-19693: - I'm working on. > SET mapreduce.job.reduces automatically

[jira] [Issue Comment Deleted] (SPARK-19693) SET mapreduce.job.reduces automatically converted to spark.sql.shuffle.partitions

2017-02-21 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19693: Comment: was deleted (was: I'm working on.) > SET mapreduce.job.reduces automatically converted

[jira] [Created] (SPARK-19693) SET mapreduce.job.reduces automatically converted to spark.sql.shuffle.partitions

2017-02-21 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-19693: --- Summary: SET mapreduce.job.reduces automatically converted to spark.sql.shuffle.partitions Key: SPARK-19693 URL: https://issues.apache.org/jira/browse/SPARK-19693

[jira] [Created] (SPARK-19660) Replace the configuration property names that are deprecated in the version of Hadoop 2.6

2017-02-19 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-19660: --- Summary: Replace the configuration property names that are deprecated in the version of Hadoop 2.6 Key: SPARK-19660 URL: https://issues.apache.org/jira/browse/SPARK-19660

[jira] [Commented] (SPARK-19226) Report failure reason from Reporter Thread

2017-02-11 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862684#comment-15862684 ] Yuming Wang commented on SPARK-19226: - Try to increase ApplicationMaster's Java heap {{-- conf

[jira] [Comment Edited] (SPARK-19226) Report failure reason from Reporter Thread

2017-02-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862684#comment-15862684 ] Yuming Wang edited comment on SPARK-19226 at 2/13/17 2:38 AM: -- Try to

[jira] [Commented] (SPARK-19764) Executors hang with supposedly running task that are really finished.

2017-02-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888255#comment-15888255 ] Yuming Wang commented on SPARK-19764: - Could you provide the full thread dump? May be Netty issue,

[jira] [Commented] (SPARK-17364) Can not query hive table starting with number

2016-09-04 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15462472#comment-15462472 ] Yuming Wang commented on SPARK-17364: - {code} SELECT * from `temp`.`20160826_ip_list` limit 100

[jira] [Created] (SPARK-17685) WholeStageCodegenExec throws IndexOutOfBoundsException

2016-09-27 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-17685: --- Summary: WholeStageCodegenExec throws IndexOutOfBoundsException Key: SPARK-17685 URL: https://issues.apache.org/jira/browse/SPARK-17685 Project: Spark Issue

[jira] [Resolved] (SPARK-17891) SQL-based three column join loses first column

2016-10-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-17891. - Resolution: Fixed > SQL-based three column join loses first column >

[jira] [Issue Comment Deleted] (SPARK-17891) SQL-based three column join loses first column

2016-10-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-17891: Comment: was deleted (was: *Workaround:* # Disable BroadcastHashJoin by setting

[jira] [Commented] (SPARK-17891) SQL-based three column join loses first column

2016-10-20 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15593685#comment-15593685 ] Yuming Wang commented on SPARK-17891: - *Workaround:* # Disable BroadcastHashJoin by setting

[jira] [Commented] (SPARK-18827) Cann't cache broadcast to disk

2016-12-11 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15741009#comment-15741009 ] Yuming Wang commented on SPARK-18827: - I will create a PR later. > Cann't cache broadcast to disk >

[jira] [Created] (SPARK-18827) Cann't cache broadcast to disk

2016-12-11 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-18827: --- Summary: Cann't cache broadcast to disk Key: SPARK-18827 URL: https://issues.apache.org/jira/browse/SPARK-18827 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-18837) It will not hidden if job or stage description too long

2016-12-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18837: Description: *previous*: !ui-2.0.0.gif! *current*: !ui-2.1.0.gif! was:!attached-image.gif!

[jira] [Updated] (SPARK-18837) It will not hidden if job or stage description too long

2016-12-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18837: Description: !attached-image.gif! > It will not hidden if job or stage description too long >

[jira] [Updated] (SPARK-18837) It will not hidden if job or stage description too long

2016-12-12 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18837: Attachment: ui-2.0.0.gif ui-2.1.0.gif > It will not hidden if job or stage

[jira] [Created] (SPARK-18837) It will not hidden if job or stage description too long

2016-12-12 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-18837: --- Summary: It will not hidden if job or stage description too long Key: SPARK-18837 URL: https://issues.apache.org/jira/browse/SPARK-18837 Project: Spark Issue

[jira] [Updated] (SPARK-18827) Cann't read broadcast if broadcast blocks are stored on-disk

2016-12-14 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18827: Description: How to reproduce it: {code:java} test("Cache broadcast to disk") { val conf =

[jira] [Updated] (SPARK-18827) Cann't read broadcast if broadcast blocks are stored on-disk

2016-12-13 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18827: Summary: Cann't read broadcast if broadcast blocks are stored on-disk (was: Cann't read broadcast

[jira] [Updated] (SPARK-18827) Cann't read broadcast if broadcast blocks are stored on-disk,

2016-12-13 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18827: Summary: Cann't read broadcast if broadcast blocks are stored on-disk, (was: Cann't cache

[jira] [Updated] (SPARK-18681) Throw Filtering is supported only on partition keys of type string exception

2016-12-07 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18681: Description: Cloudera put

[jira] [Commented] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

2017-01-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816920#comment-15816920 ] Yuming Wang commented on SPARK-19090: - Try this, it works for me: {code} sbin/start-thriftserver.sh

[jira] [Issue Comment Deleted] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

2017-01-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19090: Comment: was deleted (was: Try this, it works for me, my spark version is 2.1.0: {code} spark-sql

[jira] [Comment Edited] (SPARK-19090) Dynamic Resource Allocation not respecting spark.executor.cores

2017-01-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816920#comment-15816920 ] Yuming Wang edited comment on SPARK-19090 at 1/11/17 2:42 AM: -- Try this, it

[jira] [Created] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-09 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-19146: --- Summary: Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop Key: SPARK-19146 URL:

[jira] [Commented] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-09 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814152#comment-15814152 ] Yuming Wang commented on SPARK-19146: - I will create a PR later > Drop more elements when

[jira] [Updated] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-09 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19146: Attachment: can-not-consume-taskEnd-events.jpg > Drop more elements when stageData.taskData.size >

[jira] [Commented] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-09 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814207#comment-15814207 ] Yuming Wang commented on SPARK-19146: - The activated tasks more and more and then

[jira] [Commented] (SPARK-19175) columns changed orc table encouter 'IndexOutOfBoundsException' when read the old schema files

2017-01-11 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817750#comment-15817750 ] Yuming Wang commented on SPARK-19175: - Looks like you create seven Jiras (SPARK-19175, SPARK-19174,

[jira] [Closed] (SPARK-18680) Throw Filtering is supported only on partition keys of type string exception

2016-12-01 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang closed SPARK-18680. --- Resolution: Duplicate > Throw Filtering is supported only on partition keys of type string exception

[jira] [Created] (SPARK-18681) Throw Filtering is supported only on partition keys of type string exception

2016-12-01 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-18681: --- Summary: Throw Filtering is supported only on partition keys of type string exception Key: SPARK-18681 URL: https://issues.apache.org/jira/browse/SPARK-18681 Project:

[jira] [Created] (SPARK-18680) Throw Filtering is supported only on partition keys of type string exception

2016-12-01 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-18680: --- Summary: Throw Filtering is supported only on partition keys of type string exception Key: SPARK-18680 URL: https://issues.apache.org/jira/browse/SPARK-18680 Project:

[jira] [Commented] (SPARK-18681) Throw Filtering is supported only on partition keys of type string exception

2016-12-01 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713624#comment-15713624 ] Yuming Wang commented on SPARK-18681: - I will pull request for this issue later. > Throw Filtering

[jira] [Updated] (SPARK-18681) Throw Filtering is supported only on partition keys of type string exception

2016-12-03 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18681: Description: Cloudera put

[jira] [Commented] (SPARK-18645) spark-daemon.sh arguments error lead to throws Unrecognized option

2016-11-30 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708003#comment-15708003 ] Yuming Wang commented on SPARK-18645: - I will pull request for this issue later. > spark-daemon.sh

[jira] [Created] (SPARK-18645) spark-daemon.sh arguments error lead to throws Unrecognized option

2016-11-30 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-18645: --- Summary: spark-daemon.sh arguments error lead to throws Unrecognized option Key: SPARK-18645 URL: https://issues.apache.org/jira/browse/SPARK-18645 Project: Spark

[jira] [Updated] (SPARK-18681) Throw Filtering is supported only on partition keys of type string exception

2016-12-02 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18681: Description: I'm using MySQL for the Hive Metastore. {{hive.metastore.try.direct.sql=true}} and

[jira] [Issue Comment Deleted] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19146: Comment: was deleted (was: I will create a PR later) > Drop more elements when

[jira] [Updated] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19146: Description: The performance of the

[jira] [Updated] (SPARK-19146) Drop more elements when stageData.taskData.size > retainedTasks to reduce the number of times on call drop

2017-01-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19146: Description: The performance of the

[jira] [Commented] (SPARK-18910) Can't use UDF that jar file in hdfs

2016-12-20 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15764350#comment-15764350 ] Yuming Wang commented on SPARK-18910: - This should be a duplicate of SPARK-12868. > Can't use UDF

[jira] [Commented] (SPARK-15044) spark-sql will throw "input path does not exist" exception if it handles a partition which exists in hive table, but the path is removed manually

2016-12-26 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15779766#comment-15779766 ] Yuming Wang commented on SPARK-15044: - I've tested on v2.1.0-rc5, it works fine if

[jira] [Updated] (SPARK-18827) Cann't read broadcast if broadcast blocks are stored on-disk

2016-12-16 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-18827: Attachment: NoSuchElementException4722.gif > Cann't read broadcast if broadcast blocks are stored

[jira] [Commented] (SPARK-19927) SparkThriftServer2 can not get ''--hivevar" variables in spark 2.1

2017-03-22 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15936601#comment-15936601 ] Yuming Wang commented on SPARK-19927: - Is this duplicated by

[jira] [Updated] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Summary: Speed up HadoopMapReduceCommitProtocol#commitJob for many output files (was: Speed up

[jira] [Commented] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945379#comment-15945379 ] Yuming Wang commented on SPARK-20107: - OK, I will add

[jira] [Comment Edited] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15945379#comment-15945379 ] Yuming Wang edited comment on SPARK-20107 at 3/28/17 3:37 PM: -- OK, I will

[jira] [Resolved] (SPARK-19811) sparksql 2.1 can not prune hive partition

2017-03-23 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-19811. - Resolution: Duplicate > sparksql 2.1 can not prune hive partition >

[jira] [Updated] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Component/s: Documentation > Speed up HadoopMapReduceCommitProtocol#commitJob for many output

[jira] [Updated] (SPARK-20107) Speed up HadoopMapReduceCommitProtocol#commitJob for many output files

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Component/s: (was: SQL) > Speed up HadoopMapReduceCommitProtocol#commitJob for many output

[jira] [Updated] (SPARK-20107) Add spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version option to configuration.md

2017-03-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Description: Set {{spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2}} can speed up

[jira] [Updated] (SPARK-20107) Speed up FileOutputCommitter#commitJob for many output files

2017-03-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20107: Description: Set {{mapreduce.fileoutputcommitter.algorithm.version=2}} to speed up

[jira] [Commented] (SPARK-20107) Speed up FileOutputCommitter#commitJob for many output files

2017-03-27 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943220#comment-15943220 ] Yuming Wang commented on SPARK-20107: - I will create a PR later > Speed up

[jira] [Created] (SPARK-20107) Speed up FileOutputCommitter#commitJob for many output files

2017-03-27 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-20107: --- Summary: Speed up FileOutputCommitter#commitJob for many output files Key: SPARK-20107 URL: https://issues.apache.org/jira/browse/SPARK-20107 Project: Spark

[jira] [Created] (SPARK-20120) spark-sql CLI support silent mode

2017-03-27 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-20120: --- Summary: spark-sql CLI support silent mode Key: SPARK-20120 URL: https://issues.apache.org/jira/browse/SPARK-20120 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-19927) SparkThriftServer2 can not get ''--hivevar" variables in spark 2.1

2017-03-23 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15937780#comment-15937780 ] Yuming Wang commented on SPARK-19927: - [~xwc3504] You can use my own released

[jira] [Updated] (SPARK-20187) Replace loadTable with moveFile to speed up load table for many output files

2017-04-01 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20187: Attachment: spark.moveFile.log.tar.gz spark.loadTable.log.tar.gz > Replace

[jira] [Updated] (SPARK-20187) Replace loadTable with moveFile to speed up load table for many output files

2017-04-01 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20187: Description:

[jira] [Created] (SPARK-20187) Replace loadTable with moveFile can speed up load table for many output files

2017-04-01 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-20187: --- Summary: Replace loadTable with moveFile can speed up load table for many output files Key: SPARK-20187 URL: https://issues.apache.org/jira/browse/SPARK-20187 Project:

[jira] [Commented] (SPARK-20187) Replace loadTable with moveFile can speed up load table for many output files

2017-04-01 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15952161#comment-15952161 ] Yuming Wang commented on SPARK-20187: - I'm working on. > Replace loadTable with moveFile can speed

[jira] [Updated] (SPARK-20187) Replace loadTable with moveFile to speed up load table for many output files

2017-04-01 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20187: Summary: Replace loadTable with moveFile to speed up load table for many output files (was:

[jira] [Issue Comment Deleted] (SPARK-20187) Replace loadTable with moveFile to speed up load table for many output files

2017-04-01 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20187: Comment: was deleted (was: I'm working on.) > Replace loadTable with moveFile to speed up load

[jira] [Resolved] (SPARK-20187) Replace loadTable with moveFile to speed up load table for many output files

2017-04-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-20187. - Resolution: Duplicate > Replace loadTable with moveFile to speed up load table for many output

[jira] [Closed] (SPARK-20337) Support upgrade a jar dependency and don't restart SparkContext

2017-04-14 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang closed SPARK-20337. --- Resolution: Won't Fix > Support upgrade a jar dependency and don't restart SparkContext >

[jira] [Created] (SPARK-20337) Support upgrade a jar dependency and don't restart SparkContext

2017-04-14 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-20337: --- Summary: Support upgrade a jar dependency and don't restart SparkContext Key: SPARK-20337 URL: https://issues.apache.org/jira/browse/SPARK-20337 Project: Spark

[jira] [Updated] (SPARK-19764) Executors hang with supposedly running task that are really finished.

2017-02-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-19764: Attachment: netty-6153.jpg something like this: !netty-6153.jpg! > Executors hang with supposedly

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-02-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889373#comment-15889373 ] Yuming Wang commented on SPARK-18769: - How about this approach:

[jira] [Created] (SPARK-20247) Add jar but this jar is missing later shouldn't affect jobs that doesn't use this jar

2017-04-06 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-20247: --- Summary: Add jar but this jar is missing later shouldn't affect jobs that doesn't use this jar Key: SPARK-20247 URL: https://issues.apache.org/jira/browse/SPARK-20247

[jira] [Commented] (SPARK-20247) Add jar but this jar is missing later shouldn't affect jobs that doesn't use this jar

2017-04-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960265#comment-15960265 ] Yuming Wang commented on SPARK-20247: - I will create a PR later. > Add jar but this jar is missing

[jira] [Issue Comment Deleted] (SPARK-20247) Add jar but this jar is missing later shouldn't affect jobs that doesn't use this jar

2017-04-06 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-20247: Comment: was deleted (was: I will create a PR later.) > Add jar but this jar is missing later

[jira] [Created] (SPARK-21574) set hive.exec.max.dynamic.partitions lose effect

2017-07-29 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-21574: --- Summary: set hive.exec.max.dynamic.partitions lose effect Key: SPARK-21574 URL: https://issues.apache.org/jira/browse/SPARK-21574 Project: Spark Issue Type:

[jira] [Created] (SPARK-21625) sqrt(negative number) should be null

2017-08-03 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-21625: --- Summary: sqrt(negative number) should be null Key: SPARK-21625 URL: https://issues.apache.org/jira/browse/SPARK-21625 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-21625) sqrt(negative number) should be null

2017-08-03 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16112454#comment-16112454 ] Yuming Wang commented on SPARK-21625: - [~panbingkun] Yes, This is Hive's logic: {code:java}

[jira] [Updated] (SPARK-21625) sqrt(negative number) should be null

2017-08-03 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21625: Description: Both Hive and MySQL are null: {code:sql} hive> select SQRT(-10.0); OK NULL Time

[jira] [Created] (SPARK-21635) ACOS(2) and ASIN(2) should be null

2017-08-03 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-21635: --- Summary: ACOS(2) and ASIN(2) should be null Key: SPARK-21635 URL: https://issues.apache.org/jira/browse/SPARK-21635 Project: Spark Issue Type: Sub-task

[jira] [Commented] (SPARK-21246) Unexpected Data Type conversion from LONG to BIGINT

2017-06-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067620#comment-16067620 ] Yuming Wang commented on SPARK-21246: - {{Seq(3)}} should be {{Seq(3L)}}, This works for me:

[jira] [Commented] (SPARK-21253) Cannot fetch big blocks to disk

2017-06-29 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069109#comment-16069109 ] Yuming Wang commented on SPARK-21253: - I checked it, all jars are latest 2.2.0-rcX. > Cannot fetch

[jira] [Updated] (SPARK-21253) Cannot fetch big blocks to disk

2017-06-30 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21253: Description: Spark *cluster* can reproduce, *local* can't: 1. Start a spark context with

[jira] [Created] (SPARK-21269) MetadataFetchFailedException: Missing an output location for shuffle 0

2017-06-30 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-21269: --- Summary: MetadataFetchFailedException: Missing an output location for shuffle 0 Key: SPARK-21269 URL: https://issues.apache.org/jira/browse/SPARK-21269 Project: Spark

[jira] [Updated] (SPARK-21269) MetadataFetchFailedException: Missing an output location for shuffle 0

2017-06-30 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21269: Description: Spark *cluster* can reproduce, *local* can't: 1. Start a spark context with

[jira] [Created] (SPARK-21253) Cannot fetch big blocks to disk

2017-06-29 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-21253: --- Summary: Cannot fetch big blocks to disk Key: SPARK-21253 URL: https://issues.apache.org/jira/browse/SPARK-21253 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21253) Cannot fetch big blocks to disk

2017-06-29 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068232#comment-16068232 ] Yuming Wang commented on SPARK-21253: - It may be hang for a {{spark-sql}} application also:

[jira] [Updated] (SPARK-21253) Cannot fetch big blocks to disk

2017-06-29 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21253: Description: Spark *cluster* can reproduce, *local* can't: 1. Start a spark context with

[jira] [Updated] (SPARK-21253) Cannot fetch big blocks to disk

2017-06-29 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21253: Attachment: ui-thread-dump-jqhadoop221-154.gif > Cannot fetch big blocks to disk >

[jira] [Updated] (SPARK-21253) Cannot fetch big blocks to disk

2017-06-29 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21253: Description: Spark *cluster* can reproduce, *local* can't: 1. Start a spark context with

[jira] [Created] (SPARK-21646) BinaryComparison shouldn't auto cast string to int/long

2017-08-05 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-21646: --- Summary: BinaryComparison shouldn't auto cast string to int/long Key: SPARK-21646 URL: https://issues.apache.org/jira/browse/SPARK-21646 Project: Spark Issue

[jira] [Updated] (SPARK-21646) BinaryComparison shouldn't auto cast string to int/long

2017-08-05 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-21646: Description: How to reproduce: hive: {code:sql} $ hive -S hive> create table spark_21646(c1

  1   2   3   4   5   6   7   8   9   10   >