spark git commit: [SPARK-18838][CORE] Add separate listener queues to LiveListenerBus.

2017-09-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 718bbc939 -> c6ff59a23 [SPARK-18838][CORE] Add separate listener queues to LiveListenerBus. This change modifies the live listener bus so that all listeners are added to queues; each queue has its own thread to dispatch events, making it

spark git commit: [SPARK-22067][SQL] ArrowWriter should use position when setting UTF8String ByteBuffer

2017-09-19 Thread ueshin
Repository: spark Updated Branches: refs/heads/master ee13f3e3d -> 718bbc939 [SPARK-22067][SQL] ArrowWriter should use position when setting UTF8String ByteBuffer ## What changes were proposed in this pull request? The ArrowWriter StringWriter was setting Arrow data using a position of 0

spark git commit: [SPARK-19318][SPARK-22041][SPARK-16625][BACKPORT-2.1][SQL] Docker test case failure: `: General data types to be mapped to Oracle`

2017-09-19 Thread lixiao
Repository: spark Updated Branches: refs/heads/branch-2.1 30ce056d8 -> 56865a1e9 [SPARK-19318][SPARK-22041][SPARK-16625][BACKPORT-2.1][SQL] Docker test case failure: `: General data types to be mapped to Oracle` ## What changes were proposed in this pull request? This PR is backport of

spark git commit: [SPARK-21969][SQL] CommandUtils.updateTableStats should call refreshTable

2017-09-19 Thread lixiao
Repository: spark Updated Branches: refs/heads/master d5aefa83a -> ee13f3e3d [SPARK-21969][SQL] CommandUtils.updateTableStats should call refreshTable ## What changes were proposed in this pull request? Tables in the catalog cache are not invalidated once their statistics are updated. As a

spark git commit: [SPARK-21338][SQL] implement isCascadingTruncateTable() method in AggregatedDialect

2017-09-19 Thread lixiao
Repository: spark Updated Branches: refs/heads/master 2f962422a -> d5aefa83a [SPARK-21338][SQL] implement isCascadingTruncateTable() method in AggregatedDialect ## What changes were proposed in this pull request? org.apache.spark.sql.jdbc.JdbcDialect's method: def

spark git commit: [MINOR][ML] Remove unnecessary default value setting for evaluators.

2017-09-19 Thread yliang
Repository: spark Updated Branches: refs/heads/master 8319432af -> 2f962422a [MINOR][ML] Remove unnecessary default value setting for evaluators. ## What changes were proposed in this pull request? Remove unnecessary default value setting for all evaluators, as we have set them in

spark git commit: [SPARK-21917][CORE][YARN] Supporting adding http(s) resources in yarn mode

2017-09-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 581200af7 -> 8319432af [SPARK-21917][CORE][YARN] Supporting adding http(s) resources in yarn mode ## What changes were proposed in this pull request? In the current Spark, when submitting application on YARN with remote resources

spark git commit: [SPARK-21428][SQL][FOLLOWUP] CliSessionState should point to the actual metastore not a dummy one

2017-09-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 1bc17a6b8 -> 581200af7 [SPARK-21428][SQL][FOLLOWUP] CliSessionState should point to the actual metastore not a dummy one ## What changes were proposed in this pull request? While running bin/spark-sql, we will reuse cliSessionState, but

spark git commit: [SPARK-22052] Incorrect Metric assigned in MetricsReporter.scala

2017-09-19 Thread srowen
Repository: spark Updated Branches: refs/heads/branch-2.2 d0234ebcf -> 6764408f6 [SPARK-22052] Incorrect Metric assigned in MetricsReporter.scala Current implementation for processingRate-total uses wrong metric: mistakenly uses inputRowsPerSecond instead of processedRowsPerSecond ## What

spark git commit: [SPARK-22052] Incorrect Metric assigned in MetricsReporter.scala

2017-09-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master 7c92351f4 -> 1bc17a6b8 [SPARK-22052] Incorrect Metric assigned in MetricsReporter.scala Current implementation for processingRate-total uses wrong metric: mistakenly uses inputRowsPerSecond instead of processedRowsPerSecond ## What

spark git commit: [MINOR][CORE] Cleanup dead code and duplication in Mem. Management

2017-09-19 Thread srowen
Repository: spark Updated Branches: refs/heads/master a11db942a -> 7c92351f4 [MINOR][CORE] Cleanup dead code and duplication in Mem. Management ## What changes were proposed in this pull request? * Removed the method

spark git commit: [SPARK-21923][CORE] Avoid calling reserveUnrollMemoryForThisTask for every record

2017-09-19 Thread wenchen
Repository: spark Updated Branches: refs/heads/master 10f45b3c8 -> a11db942a [SPARK-21923][CORE] Avoid calling reserveUnrollMemoryForThisTask for every record ## What changes were proposed in this pull request? When Spark persist data to Unsafe memory, we call the method