git commit: [SPARK-3006] Failed to execute spark-shell in Windows OS

2014-08-13 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 0c7b45290 - 9497b12d4 [SPARK-3006] Failed to execute spark-shell in Windows OS Modified the order of the options and arguments in spark-shell.cmd Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Closes #1918 from

git commit: SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic...

2014-08-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.0 9e783a270 - 6c6409e8b SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic... ...ationInfo is initialized properly after deserialization Author: Jacek Lewandowski lewandowski.ja...@gmail.com Closes #1922 from

git commit: SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic...

2014-08-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 c39a3f337 - f5d9176fb SPARK-3009: Reverted readObject method in ApplicationInfo so that Applic... ...ationInfo is initialized properly after deserialization Author: Jacek Lewandowski lewandowski.ja...@gmail.com Closes #1947 from

git commit: [SPARK-2169] Don't copy appName / basePath everywhere.

2014-08-18 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 3abd0c1cd - 66ade00f9 [SPARK-2169] Don't copy appName / basePath everywhere. Instead of keeping copies in all pages, just reference the values kept in the base SparkUI instance (by making them available via getters). Author: Marcelo

git commit: [SPARK-2718] [yarn] Handle quotes and other characters in user args.

2014-08-18 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master d1d0ee41c - 6201b2764 [SPARK-2718] [yarn] Handle quotes and other characters in user args. Due to the way Yarn runs things through bash, normal quoting doesn't work as expected. This change applies the necessary voodoo to the user args to

git commit: [SPARK-2718] [yarn] Handle quotes and other characters in user args.

2014-08-18 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 e08333463 - 25cabd7ee [SPARK-2718] [yarn] Handle quotes and other characters in user args. Due to the way Yarn runs things through bash, normal quoting doesn't work as expected. This change applies the necessary voodoo to the user args

git commit: SPARK_LOGFILE and SPARK_ROOT_LOGGER no longer need in spark-daemon.sh

2014-08-20 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 311831db7 - 5f72d7bcf SPARK_LOGFILE and SPARK_ROOT_LOGGER no longer need in spark-daemon.sh Author: wangfei wangfei_he...@126.com Closes #2057 from scwf/patch-7 and squashes the following commits: 1b7b9a5 [wangfei] SPARK_LOGFILE and

git commit: [SPARK-3140] Clarify confusing PySpark exception message

2014-08-20 Thread andrewor14
exception message that distinguishes between these cases will prevent similar headaches that I have gone through. Author: Andrew Or andrewo...@gmail.com Closes #2067 from andrewor14/python-exception and squashes the following commits: 742f823 [Andrew Or] Further clarify warning messages

git commit: [SPARK-3140] Clarify confusing PySpark exception message

2014-08-20 Thread andrewor14
exception message that distinguishes between these cases will prevent similar headaches that I have gone through. Author: Andrew Or andrewo...@gmail.com Closes #2067 from andrewor14/python-exception and squashes the following commits: 742f823 [Andrew Or] Further clarify warning messages e96a7a0

git commit: [SPARK-2886] Use more specific actor system name than spark

2014-08-26 Thread andrewor14
. Author: Andrew Or andrewo...@gmail.com Closes #1810 from andrewor14/service-name and squashes the following commits: 8c459ed [Andrew Or] Use a common variable for driver/executor actor system names 3a92843 [Andrew Or] Change actor name to sparkDriver and sparkExecutor 921363e [Andrew Or] Merge

git commit: [SPARK-2886] Use more specific actor system name than spark

2014-08-26 Thread andrewor14
. Author: Andrew Or andrewo...@gmail.com Closes #1810 from andrewor14/service-name and squashes the following commits: 8c459ed [Andrew Or] Use a common variable for driver/executor actor system names 3a92843 [Andrew Or] Change actor name to sparkDriver and sparkExecutor 921363e [Andrew Or] Merge branch

git commit: [SPARK-3170][CORE][BUG]:RDD info loss in StorageTab and ExecutorTab

2014-08-27 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master b92d823ad - d8298c46b [SPARK-3170][CORE][BUG]:RDD info loss in StorageTab and ExecutorTab compeleted stage only need to remove its own partitions that are no longer cached. However, StorageTab may lost some rdds which are cached actually.

git commit: [SPARK-3170][CORE][BUG]:RDD info loss in StorageTab and ExecutorTab

2014-08-27 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 1d468df33 - 8f8e2a4ee [SPARK-3170][CORE][BUG]:RDD info loss in StorageTab and ExecutorTab compeleted stage only need to remove its own partitions that are no longer cached. However, StorageTab may lost some rdds which are cached

git commit: SPARK-3082. yarn.Client.logClusterResourceDetails throws NPE if requeste...

2014-08-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 96df92906 - 92af2314f SPARK-3082. yarn.Client.logClusterResourceDetails throws NPE if requeste... ...d queue doesn't exist Author: Sandy Ryza sa...@cloudera.com Closes #1984 from sryza/sandy-spark-3082 and squashes the following commits:

git commit: SPARK-3082. yarn.Client.logClusterResourceDetails throws NPE if requeste...

2014-08-28 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 0b9718add - f4cbf5e3d SPARK-3082. yarn.Client.logClusterResourceDetails throws NPE if requeste... ...d queue doesn't exist Author: Sandy Ryza sa...@cloudera.com Closes #1984 from sryza/sandy-spark-3082 and squashes the following

git commit: [SPARK-3061] Fix Maven build under Windows

2014-09-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 32ec0a8cd - 378b2315b [SPARK-3061] Fix Maven build under Windows The Maven build was failing on Windows because it tried to call the unix `unzip` utility to extract the Py4J files into core's build directory. I've fixed this issue by

git commit: [SPARK-1919] Fix Windows spark-shell --jars

2014-09-02 Thread andrewor14
Closes #2211 from andrewor14/windows-shell-jars and squashes the following commits: 262c6a2 [Andrew Or] Oops... Add the new code to the correct place 0d5a0c1 [Andrew Or] Format jar path only for adding to shell classpath 42bd626 [Andrew Or] Remove unnecessary code 0049f1b [Andrew Or] Remove

git commit: [SPARK-3216] [SPARK-3232] Spark-shell is broken in branch-1.0 / Backport SPARK-3006 into branch-1.0

2014-09-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.0 d47581638 - 8dd7690e2 [SPARK-3216] [SPARK-3232] Spark-shell is broken in branch-1.0 / Backport SPARK-3006 into branch-1.0 Author: Kousuke Saruta saru...@oss.nttdata.co.jp Author: Andrew Or andrewo...@gmail.com Closes #2136 from

git commit: [SPARK-2845] Add timestamps to block manager events.

2014-09-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master e5d376801 - ccc69e26e [SPARK-2845] Add timestamps to block manager events. These are not used by the UI but are useful when analysing the logs from a spark job. Author: Marcelo Vanzin van...@cloudera.com Closes #654 from

git commit: [SPARK-3388] Expose aplication ID in ApplicationStart event, use it in history server.

2014-09-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master ccc69e26e - f2b5b619a [SPARK-3388] Expose aplication ID in ApplicationStart event, use it in history server. This change exposes the application ID generated by the Spark Master, Mesos or Yarn via the SparkListenerApplicationStart event.

git commit: [Minor] Fix outdated Spark version

2014-09-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master f2b5b619a - 2784822e4 [Minor] Fix outdated Spark version This is causing the event logs to include a file called SPARK_VERSION_1.0.0, which is not accurate. Author: Andrew Or andrewo...@gmail.com Author: andrewor14 andrewo...@gmail.com

git commit: [SPARK-3303][core] fix SparkContextSchedulerCreationSuite test error

2014-09-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master a52240792 - e08ea7393 [SPARK-3303][core] fix SparkContextSchedulerCreationSuite test error run test with the master branch with this command when mesos native lib is set sbt/sbt -Phive test-only

git commit: [SPARK-3233] Executor never stop its SparnEnv, BlockManager, ConnectionManager etc.

2014-09-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master e08ea7393 - 4bba10c41 [SPARK-3233] Executor never stop its SparnEnv, BlockManager, ConnectionManager etc. Author: Kousuke Saruta saru...@oss.nttdata.co.jp Closes #2138 from sarutak/SPARK-3233 and squashes the following commits: c0205b7

git commit: SPARK-3337 Paranoid quoting in shell to allow install dirs with spaces within.

2014-09-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 711356b42 - e16a8e7db SPARK-3337 Paranoid quoting in shell to allow install dirs with spaces within. ... Tested ! TBH, it isn't a great idea to have directory with spaces within. Because emacs doesn't like it then hadoop doesn't like it.

git commit: SPARK-2425 Don't kill a still-running Application because of some misbehaving Executors

2014-09-08 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 2b7ab814f - 092e2f152 SPARK-2425 Don't kill a still-running Application because of some misbehaving Executors Introduces a LOADING - RUNNING ApplicationState transition and prevents Master from removing an Application with RUNNING

git commit: SPARK-2425 Don't kill a still-running Application because of some misbehaving Executors

2014-09-09 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 7a236dcf8 - e884805ce SPARK-2425 Don't kill a still-running Application because of some misbehaving Executors Introduces a LOADING - RUNNING ApplicationState transition and prevents Master from removing an Application with RUNNING

git commit: [SPARK-3193]output errer info when Process exit code is not zero in test suite

2014-09-09 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master f0f1ba09b - 26862337c [SPARK-3193]output errer info when Process exit code is not zero in test suite https://issues.apache.org/jira/browse/SPARK-3193 I noticed that sometimes pr tests failed due to the Process exitcode != 0,refer to

git commit: [SPARK-3345] Do correct parameters for ShuffleFileGroup

2014-09-09 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 24262684e - e5f77ae97 [SPARK-3345] Do correct parameters for ShuffleFileGroup In the method `newFileGroup` of class `FileShuffleBlockManager`, the parameters for creating new `ShuffleFileGroup` object is in wrong order. Because in

git commit: [SPARK-1919] Fix Windows spark-shell --jars

2014-09-09 Thread andrewor14
...@gmail.com Closes #2211 from andrewor14/windows-shell-jars and squashes the following commits: 262c6a2 [Andrew Or] Oops... Add the new code to the correct place 0d5a0c1 [Andrew Or] Format jar path only for adding to shell classpath 42bd626 [Andrew Or] Remove unnecessary code 0049f1b [Andrew Or] Remove

git commit: [SPARK-3411] Improve load-balancing of concurrently-submitted drivers across workers

2014-09-10 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master e4f4886d7 - 558962a83 [SPARK-3411] Improve load-balancing of concurrently-submitted drivers across workers If the waiting driver array is too big, the drivers in it will be dispatched to the first worker we get(if it has enough

git commit: [Spark-3490] Disable SparkUI for tests

2014-09-11 Thread andrewor14
...@gmail.com Closes #2363 from andrewor14/disable-ui-for-tests and squashes the following commits: 332a7d5 [Andrew Or] No need to set spark.ui.port to 0 anymore 30c93a2 [Andrew Or] Simplify streaming UISuite a431b84 [Andrew Or] Fix streaming test failures 8f5ae53 [Andrew Or] Fix no new line at the end

git commit: [SPARK-3429] Don't include the empty string as a defaultAclUser

2014-09-11 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 6324eb7b5 - ce59725b8 [SPARK-3429] Don't include the empty string as a defaultAclUser Changes logging from ``` 14/09/05 02:01:08 INFO SecurityManager: Changing view acls to: aash, 14/09/05 02:01:08 INFO SecurityManager: Changing modify

git commit: [SPARK-3429] Don't include the empty string as a defaultAclUser

2014-09-11 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 2ffc7980c - 4245404e8 [SPARK-3429] Don't include the empty string as a defaultAclUser Changes logging from ``` 14/09/05 02:01:08 INFO SecurityManager: Changing view acls to: aash, 14/09/05 02:01:08 INFO SecurityManager: Changing

git commit: SPARK-2482: Resolve sbt warnings during build

2014-09-11 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master f858f4668 - 33c7a738a SPARK-2482: Resolve sbt warnings during build At the same time, import the `scala.language.postfixOps` and ` org.scalatest.time.SpanSugar._` cause `scala.language.postfixOps` doesn't work Author: witgo wi...@qq.com

git commit: [SPARK-3465] fix task metrics aggregation in local mode

2014-09-11 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 4245404e8 - e69deb818 [SPARK-3465] fix task metrics aggregation in local mode Before overwrite t.taskMetrics, take a deepcopy of it. Author: Davies Liu davies@gmail.com Closes #2338 from davies/fix_metric and squashes the

git commit: Revert [Spark-3490] Disable SparkUI for tests

2014-09-12 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 e69deb818 - f17b7957a Revert [Spark-3490] Disable SparkUI for tests This reverts commit 2ffc7980c6818eec05e32141c52e335bc71daed9. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit:

git commit: [SPARK-2714] DAGScheduler logs jobid when runJob finishes

2014-09-15 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master e59fac1f9 - 37d925280 [SPARK-2714] DAGScheduler logs jobid when runJob finishes DAGScheduler logs jobid when runJob finishes Author: yantangzhai tyz0...@163.com Closes #1617 from YanTangZhai/SPARK-2714 and squashes the following commits:

git commit: [SPARK-3490] Disable SparkUI for tests (backport into 1.1)

2014-09-16 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 856156b40 - 937de93e8 [SPARK-3490] Disable SparkUI for tests (backport into 1.1) Original PR: #2363 Author: Andrew Or andrewo...@gmail.com Closes #2415 from andrewor14/disable-ui-for-tests-1.1 and squashes the following commits

git commit: Docs: move HA subsections to a deeper indentation level

2014-09-17 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 85e7c52ba - 0690410e4 Docs: move HA subsections to a deeper indentation level Makes the table of contents read better Author: Andrew Ash and...@andrewash.com Closes #2402 from ash211/docs/better-indentation and squashes the following

git commit: [SPARK-3571] Spark standalone cluster mode doesn't work.

2014-09-17 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 7fc3bb7c8 - cbc065039 [SPARK-3571] Spark standalone cluster mode doesn't work. I think, this issue is caused by #1106 Author: Kousuke Saruta saru...@oss.nttdata.co.jp Closes #2436 from sarutak/SPARK-3571 and squashes the following

git commit: [SPARK-3564][WebUI] Display App ID on HistoryPage

2014-09-17 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master cbc065039 - 6688a266f [SPARK-3564][WebUI] Display App ID on HistoryPage Author: Kousuke Saruta saru...@oss.nttdata.co.jp Closes #2424 from sarutak/display-appid-on-webui and squashes the following commits: 417fe90 [Kousuke Saruta] Added

git commit: [SPARK-3564][WebUI] Display App ID on HistoryPage

2014-09-17 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 0690410e4 - 3f1f9744b [SPARK-3564][WebUI] Display App ID on HistoryPage Author: Kousuke Saruta saru...@oss.nttdata.co.jp Closes #2424 from sarutak/display-appid-on-webui and squashes the following commits: 417fe90 [Kousuke Saruta]

git commit: [SPARK-3567] appId field in SparkDeploySchedulerBackend should be volatile

2014-09-17 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 6688a266f - 1147973f1 [SPARK-3567] appId field in SparkDeploySchedulerBackend should be volatile Author: Kousuke Saruta saru...@oss.nttdata.co.jp Closes #2428 from sarutak/appid-volatile-modification and squashes the following commits:

git commit: [SPARK-3560] Fixed setting spark.jars system property in yarn-cluster mode

2014-09-18 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 471e6a3a4 - b3ed37e5b [SPARK-3560] Fixed setting spark.jars system property in yarn-cluster mode Author: Victsm victor@gmail.com Author: Min Shen ms...@linkedin.com Closes #2449 from Victsm/SPARK-3560 and squashes the following

git commit: [SPARK-3560] Fixed setting spark.jars system property in yarn-cluster mode

2014-09-18 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 2b286926b - 832dff64d [SPARK-3560] Fixed setting spark.jars system property in yarn-cluster mode Author: Victsm victor@gmail.com Author: Min Shen ms...@linkedin.com Closes #2449 from Victsm/SPARK-3560 and squashes the following

git commit: [Minor Hot Fix] Move a line in SparkSubmit to the right place

2014-09-18 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master b3ed37e5b - 9306297d1 [Minor Hot Fix] Move a line in SparkSubmit to the right place This was introduced in #2449 Author: Andrew Or andrewo...@gmail.com Closes #2452 from andrewor14/standalone-hot-fix and squashes the following commits

git commit: [Docs] Fix outdated docs for standalone cluster

2014-09-19 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 99b06b6fd - 8af237061 [Docs] Fix outdated docs for standalone cluster This is now supported! Author: andrewor14 andrewo...@gmail.com Author: Andrew Or andrewo...@gmail.com Closes #2461 from andrewor14/document-standalone-cluster

git commit: [Docs] Fix outdated docs for standalone cluster

2014-09-19 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 1687d6ba9 - fd8835323 [Docs] Fix outdated docs for standalone cluster This is now supported! Author: andrewor14 andrewo...@gmail.com Author: Andrew Or andrewo...@gmail.com Closes #2461 from andrewor14/document-standalone-cluster

git commit: [SPARK-3653] Respect SPARK_*_MEMORY for cluster mode

2014-09-23 Thread andrewor14
, this property is simply not propagated to the worker nodes. `SPARK_EXECUTOR_MEMORY` is picked up from `SparkContext`, but in cluster mode the driver runs on one of the worker machines, where this environment variable may not be set. Author: Andrew Or andrewo...@gmail.com Closes #2500 from andrewor14

git commit: [SPARK-3653] Respect SPARK_*_MEMORY for cluster mode

2014-09-23 Thread andrewor14
, this property is simply not propagated to the worker nodes. `SPARK_EXECUTOR_MEMORY` is picked up from `SparkContext`, but in cluster mode the driver runs on one of the worker machines, where this environment variable may not be set. Author: Andrew Or andrewo...@gmail.com Closes #2500 from andrewor14

git commit: [SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI

2014-09-23 Thread andrewor14
andrewor14, mubarak Author: Mubarak Seyed mubarak.se...@gmail.com Author: Tathagata Das tathagata.das1...@gmail.com Closes #2464 from tdas/streaming-callsite and squashes the following commits: dc54c71 [Tathagata Das] Made changes based on PR comments. 390b45d [Tathagata Das] Fixed minor bugs

git commit: [SPARK-1853] Show Streaming application code context (file, line number) in Spark Stages UI

2014-09-23 Thread andrewor14
a look andrewor14, mubarak Author: Mubarak Seyed mubarak.se...@gmail.com Author: Tathagata Das tathagata.das1...@gmail.com Closes #2464 from tdas/streaming-callsite and squashes the following commits: dc54c71 [Tathagata Das] Made changes based on PR comments. 390b45d [Tathagata Das] Fixed minor bugs

git commit: [SPARK-3476] Remove outdated memory checks in Yarn

2014-09-26 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 30461c6ac - 8da10bf14 [SPARK-3476] Remove outdated memory checks in Yarn See description in [JIRA](https://issues.apache.org/jira/browse/SPARK-3476). Author: Andrew Or andrewo...@gmail.com Closes #2528 from andrewor14/yarn-memory-checks

git commit: Docs : use --total-executor-cores rather than --cores after spark-shell

2014-09-27 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 d6ed5abf1 - d9d94e0b0 Docs : use --total-executor-cores rather than --cores after spark-shell Author: CrazyJvm crazy...@gmail.com Closes #2540 from CrazyJvm/standalone-core and squashes the following commits: 66d9fc6 [CrazyJvm] use

git commit: [SPARK-3755][Core] Do not bind port 1 - 1024 to server in spark

2014-10-01 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master dcb2f73f1 - 6390aae4e [SPARK-3755][Core] Do not bind port 1 - 1024 to server in spark Non-root user use port 1- 1024 to start jetty server will get the exception java.net.SocketException: Permission denied, so not use these ports

git commit: [SPARK-3755][Core] Do not bind port 1 - 1024 to server in spark

2014-10-01 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 a7d2df42b - c8c3b49f2 [SPARK-3755][Core] Do not bind port 1 - 1024 to server in spark Non-root user use port 1- 1024 to start jetty server will get the exception java.net.SocketException: Permission denied, so not use these ports

git commit: [SPARK-3755][Core] avoid trying privileged port when request a non-privileged port

2014-10-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 127e97bee - 8081ce8bd [SPARK-3755][Core] avoid trying privileged port when request a non-privileged port pwendell, ```tryPort``` is not compatible with old code in last PR, this is to fix it. And after discuss with srowen renamed the

git commit: [SPARK-3755][Core] avoid trying privileged port when request a non-privileged port

2014-10-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 68693519e - 16789f62a [SPARK-3755][Core] avoid trying privileged port when request a non-privileged port pwendell, ```tryPort``` is not compatible with old code in last PR, this is to fix it. And after discuss with srowen renamed the

git commit: [DEPLOY] SPARK-3759: Return the exit code of the driver process

2014-10-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 8081ce8bd - 42d5077fd [DEPLOY] SPARK-3759: Return the exit code of the driver process SparkSubmitDriverBootstrapper.scala now returns the exit code of the driver process, instead of always returning 0. Author: Eric Eijkelenboom

git commit: [DEPLOY] SPARK-3759: Return the exit code of the driver process

2014-10-02 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 16789f62a - 699af62dc [DEPLOY] SPARK-3759: Return the exit code of the driver process SparkSubmitDriverBootstrapper.scala now returns the exit code of the driver process, instead of always returning 0. Author: Eric Eijkelenboom

git commit: SPARK-2058: Overriding SPARK_HOME/conf with SPARK_CONF_DIR

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 699af62dc - 5d991db8a SPARK-2058: Overriding SPARK_HOME/conf with SPARK_CONF_DIR Update of PR #997. With this PR, setting SPARK_CONF_DIR overrides SPARK_HOME/conf (not only spark-defaults.conf and spark-env). Author: EugenCepoi

git commit: [SPARK-3696]Do not override the user-difined conf_dir

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 5d991db8a - d5af9e160 [SPARK-3696]Do not override the user-difined conf_dir https://issues.apache.org/jira/browse/SPARK-3696 We see if SPARK_CONF_DIR is already defined before assignment. Author: WangTaoTheTonic

git commit: [SPARK-3535][Mesos] Fix resource handling.

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 6a1d48f4f - a8c52d534 [SPARK-3535][Mesos] Fix resource handling. Author: Brenden Matthews bren...@diddyinc.com Closes #2401 from brndnmtthws/master and squashes the following commits: 4abaa5d [Brenden Matthews] [SPARK-3535][Mesos] Fix

git commit: [SPARK-3535][Mesos] Fix resource handling.

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 d5af9e160 - 6f1509784 [SPARK-3535][Mesos] Fix resource handling. Author: Brenden Matthews bren...@diddyinc.com Closes #2401 from brndnmtthws/master and squashes the following commits: 4abaa5d [Brenden Matthews] [SPARK-3535][Mesos]

git commit: [SPARK-3775] Not suitable error message in spark-shell.cmd

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 6f1509784 - f13025669 [SPARK-3775] Not suitable error message in spark-shell.cmd Modified some sentence of error message in bin\*.cmd. Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Closes #2640 from tsudukim/feature/SPARK-3775

git commit: [SPARK-3774] typo comment in bin/utils.sh

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 358d7ffd0 - e5566e05b [SPARK-3774] typo comment in bin/utils.sh Modified the comment of bin/utils.sh. Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Closes #2639 from tsudukim/feature/SPARK-3774 and squashes the following commits:

git commit: [SPARK-3774] typo comment in bin/utils.sh

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 f13025669 - e4ddedee6 [SPARK-3774] typo comment in bin/utils.sh Modified the comment of bin/utils.sh. Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Closes #2639 from tsudukim/feature/SPARK-3774 and squashes the following

git commit: [SPARK-3606] [yarn] Correctly configure AmIpFilter for Yarn HA.

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master e5566e05b - 30abef154 [SPARK-3606] [yarn] Correctly configure AmIpFilter for Yarn HA. The existing code only considered one of the RMs when running in Yarn HA mode, so it was possible to get errors if the active RM was not registered in

git commit: [SPARK-3763] The example of building with sbt should be sbt assembly instead of sbt compile

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 30abef154 - 1eb8389cb [SPARK-3763] The example of building with sbt should be sbt assembly instead of sbt compile In building-spark.md, there are some examples for making assembled package with maven but the example for building with sbt

git commit: [SPARK-3377] [SPARK-3610] Metrics can be accidentally aggregated / History server log name should not be based on user input

2014-10-03 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 1eb8389cb - 79e45c932 [SPARK-3377] [SPARK-3610] Metrics can be accidentally aggregated / History server log name should not be based on user input This PR is another solution for #2250 I'm using codahale base MetricsSystem of Spark with

git commit: [SPARK-3597][Mesos] Implement `killTask`.

2014-10-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master cf1d32e3e - 32fad4233 [SPARK-3597][Mesos] Implement `killTask`. The MesosSchedulerBackend did not previously implement `killTask`, resulting in an exception. Author: Brenden Matthews bren...@diddyinc.com Closes #2453 from

git commit: [SPARK-3597][Mesos] Implement `killTask`.

2014-10-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 e4ddedee6 - d9cf4d08a [SPARK-3597][Mesos] Implement `killTask`. The MesosSchedulerBackend did not previously implement `killTask`, resulting in an exception. Author: Brenden Matthews bren...@diddyinc.com Closes #2453 from

git commit: SPARK-1656: Fix potential resource leaks

2014-10-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 32fad4233 - a7c73130f SPARK-1656: Fix potential resource leaks JIRA: https://issues.apache.org/jira/browse/SPARK-1656 Author: zsxwing zsxw...@gmail.com Closes #577 from zsxwing/SPARK-1656 and squashes the following commits: c431095

git commit: SPARK-1656: Fix potential resource leaks

2014-10-05 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 d9cf4d08a - c068d9084 SPARK-1656: Fix potential resource leaks JIRA: https://issues.apache.org/jira/browse/SPARK-1656 Author: zsxwing zsxw...@gmail.com Closes #577 from zsxwing/SPARK-1656 and squashes the following commits: c431095

git commit: [SPARK-3808] PySpark fails to start in Windows

2014-10-07 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 82ab4a796 - 3a7875d9d [SPARK-3808] PySpark fails to start in Windows Modified syntax error of *.cmd script. Author: Masayoshi TSUZUKI tsudu...@oss.nttdata.co.jp Closes #2669 from tsudukim/feature/SPARK-3808 and squashes the following

git commit: [SPARK-3825] Log more detail when unrolling a block fails

2014-10-07 Thread andrewor14
Closes #2688 from andrewor14/cache-log-message and squashes the following commits: 28e33d6 [Andrew Or] Shy away from unrolling 5638c49 [Andrew Or] Grammar 39a0c28 [Andrew Or] Log more detail when unrolling a block fails Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http

git commit: [SPARK-3777] Display Executor ID for Tasks in Stage page

2014-10-07 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 553737c6e - 446063eca [SPARK-3777] Display Executor ID for Tasks in Stage page Now the Stage page only displays Executor(host) for tasks. However, there may be more than one Executors running in the same host. Currently, when some task

git commit: [SPARK-3829] Make Spark logo image on the header of HistoryPage as a link to HistoryPage's page #1

2014-10-07 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 e8afb733a - a1f833f75 [SPARK-3829] Make Spark logo image on the header of HistoryPage as a link to HistoryPage's page #1 There is a Spark logo on the header of HistoryPage. We can have too many HistoryPages if we run 20+ applications.

git commit: [SPARK-3844][UI] Truncate appName in WebUI if it is too long

2014-10-09 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 a44af7302 - 09d6a81ad [SPARK-3844][UI] Truncate appName in WebUI if it is too long Truncate appName in WebUI if it is too long. Author: Xiangrui Meng m...@databricks.com Closes #2707 from mengxr/truncate-app-name and squashes the

git commit: [SPARK-3844][UI] Truncate appName in WebUI if it is too long

2014-10-09 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master b9df8af62 - 86b392942 [SPARK-3844][UI] Truncate appName in WebUI if it is too long Truncate appName in WebUI if it is too long. Author: Xiangrui Meng m...@databricks.com Closes #2707 from mengxr/truncate-app-name and squashes the

git commit: add spark.driver.memory to config docs

2014-10-09 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 86b392942 - 13cab5ba4 add spark.driver.memory to config docs It took me a minute to track this down, so I thought it could be useful to have it in the docs. I'm unsure if 512mb is the default for spark.driver.memory? Also - there could

git commit: [SPARK-3921] Fix CoarseGrainedExecutorBackend's arguments for Standalone mode

2014-10-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 4d26aca77 - 186b497c9 [SPARK-3921] Fix CoarseGrainedExecutorBackend's arguments for Standalone mode The goal of this patch is to fix the swapped arguments in standalone mode, which was caused by

git commit: [SPARK-3946] gitignore in /python includes wrong directory

2014-10-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 9b6de6fbc - 7ced88b0d [SPARK-3946] gitignore in /python includes wrong directory Modified to ignore not the docs/ directory, but only the docs/_build/ which is the output directory of sphinx build. Author: Masayoshi TSUZUKI

git commit: [SPARK-3944][Core] Using Option[String] where value of String can be null

2014-10-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 7ced88b0d - 24b818b97 [SPARK-3944][Core] Using Option[String] where value of String can be null Author: shitis ssaxena@gmail.com Closes #2795 from Shiti/master and squashes the following commits: 46897d7 [shitis] Using Option Wrapper

git commit: [SPARK-3869] ./bin/spark-class miss Java version with _JAVA_OPTIONS set

2014-10-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 56096dbaa - 7b4f39f64 [SPARK-3869] ./bin/spark-class miss Java version with _JAVA_OPTIONS set When _JAVA_OPTIONS environment variable is set, a command java -version outputs a message like Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8.

git commit: [SPARK-3943] Some scripts bin\*.cmd pollutes environment variables in Windows

2014-10-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 7b4f39f64 - 66af8e250 [SPARK-3943] Some scripts bin\*.cmd pollutes environment variables in Windows Modified not to pollute environment variables. Just moved the main logic into `XXX2.cmd` from `XXX.cmd`, and call `XXX2.cmd` with cmd

git commit: [SPARK-2098] All Spark processes should support spark-defaults.conf, config file

2014-10-14 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 18ab6bd70 - 293a0b5db [SPARK-2098] All Spark processes should support spark-defaults.conf, config file This is another implementation about #1256 cc andrewor14 vanzin Author: GuoQiang Li wi...@qq.com Closes #2379 from witgo/SPARK-2098

git commit: [SPARK-3923] Increase Akka heartbeat pause above heartbeat interval

2014-10-16 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 2fe0ba956 - 7f7b50ed9 [SPARK-3923] Increase Akka heartbeat pause above heartbeat interval Something about the 2.3.4 upgrade seems to have made the issue manifest where all the services disconnect from each other after exactly 1000 seconds

git commit: [SQL]typo in HiveFromSpark

2014-10-16 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 7f7b50ed9 - be2ec4a91 [SQL]typo in HiveFromSpark Author: Kun Li jacky.li...@gmail.com Closes #2809 from jackylk/patch-1 and squashes the following commits: 46c926b [Kun Li] typo in HiveFromSpark Project:

git commit: [SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo

2014-10-16 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master be2ec4a91 - 642b246be [SPARK-3941][CORE] _remainingmem should not increase twice when updateBlockInfo In BlockManagermasterActor, _remainingMem would increase memSize for twice when updateBlockInfo if new storageLevel is invalid and old

git commit: [SPARK-3890][Docs]remove redundant spark.executor.memory in doc

2014-10-16 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 61e590354 - 2c4117083 [SPARK-3890][Docs]remove redundant spark.executor.memory in doc Introduced in https://github.com/pwendell/spark/commit/f7e79bc42c1635686c3af01eef147dae92de2529, I'm not sure why we need two spark.executor.memory

git commit: [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes

2014-10-16 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 56fd34af5 - dedace83f [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes JobProgressPage could not show Fair Scheduler Pools section sometimes. SparkContext starts webui and then postEnvironmentUpdate.

git commit: [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes

2014-10-16 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 2c4117083 - 35875e9ec [SPARK-3067] JobProgressPage could not show Fair Scheduler Pools section sometimes JobProgressPage could not show Fair Scheduler Pools section sometimes. SparkContext starts webui and then postEnvironmentUpdate.

git commit: [SPARK-3606] [yarn] Correctly configure AmIpFilter for Yarn HA (1.1 vers...

2014-10-17 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 35875e9ec - 0d958f163 [SPARK-3606] [yarn] Correctly configure AmIpFilter for Yarn HA (1.1 vers... ...ion). This is a backport of SPARK-3606 to branch-1.1. Some of the code had to be duplicated since branch-1.1 doesn't have the cleanup

git commit: [SPARK-3935][Core] log the number of records that has been written

2014-10-17 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master e678b9f02 - c35186206 [SPARK-3935][Core] log the number of records that has been written There is a unused variable(count) in saveAsHadoopDataset in PairRDDFunctions.scala. The initial idea of this variable seems to count the number of

git commit: [SPARK-3426] Fix sort-based shuffle error when spark.shuffle.compress and spark.shuffle.spill.compress settings are different

2014-10-22 Thread andrewor14
| |---|--|| | Read | spark.shuffle.spill.compress | spark.shuffle.compress | | Write | spark.shuffle.spill.compress | spark.shuffle.compress | Thanks to andrewor14 for debugging this with me! Author: Josh Rosen joshro...@databricks.com Closes #2890 from

git commit: [SPARK-3877][YARN] Throw an exception when application is not successful so that the exit code wil be set to 1

2014-10-22 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 813effc70 - 137d94235 [SPARK-3877][YARN] Throw an exception when application is not successful so that the exit code wil be set to 1 When an yarn application fails (yarn-cluster mode), the exit code of spark-submit is still 0. It's hard

git commit: [SPARK-3877][YARN] Throw an exception when application is not successful so that the exit code wil be set to 1 (for branch-1.1)

2014-10-22 Thread andrewor14
Repository: spark Updated Branches: refs/heads/branch-1.1 457ef5955 - eb62094e8 [SPARK-3877][YARN] Throw an exception when application is not successful so that the exit code wil be set to 1 (for branch-1.1) This is a patch to fix SPARK-3877 in branch-1.1. See also #2732 Author: zsxwing

git commit: [SPARK-2713] Executors of same application in same host should only download files jars once

2014-10-24 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 6a40a7684 - 7aacb7bfa [SPARK-2713] Executors of same application in same host should only download files jars once If Spark lunched multiple executors in one host for one application, every executor would download it dependent files and

git commit: [SPARK-4076] Parameter expansion in spark-config is wrong

2014-10-24 Thread andrewor14
Repository: spark Updated Branches: refs/heads/master 7aacb7bfa - 30ea2868e [SPARK-4076] Parameter expansion in spark-config is wrong In sbin/spark-config.sh, parameter expansion is used to extract source root as follows. this=${BASH_SOURCE-$0} I think, the parameter expansion should

  1   2   3   4   5   6   7   8   9   10   >