git commit: SPARK-1890 and SPARK-1891- add admin and modify acls

2014-08-05 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.1 6c0c65fc8 -> e3fe6571d SPARK-1890 and SPARK-1891- add admin and modify acls It was easier to combine these 2 jira since they touch many of the same places. This pr adds the following: - adds modify acls - adds admin acls (list of adm

git commit: SPARK-1890 and SPARK-1891- add admin and modify acls

2014-08-05 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 2c0f705e2 -> 1ca23 SPARK-1890 and SPARK-1891- add admin and modify acls It was easier to combine these 2 jira since they touch many of the same places. This pr adds the following: - adds modify acls - adds admin acls (list of admins/

git commit: SPARK-1528 - spark on yarn, add support for accessing remote HDFS

2014-08-05 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.1 b92a45058 -> 6c0c65fc8 SPARK-1528 - spark on yarn, add support for accessing remote HDFS Add a config (spark.yarn.access.namenodes) to allow applications running on yarn to access other secure HDFS cluster. User just specifies the nam

git commit: SPARK-1528 - spark on yarn, add support for accessing remote HDFS

2014-08-05 Thread tgraves
Repository: spark Updated Branches: refs/heads/master e87075df9 -> 2c0f705e2 SPARK-1528 - spark on yarn, add support for accessing remote HDFS Add a config (spark.yarn.access.namenodes) to allow applications running on yarn to access other secure HDFS cluster. User just specifies the namenod

git commit: [SPARK-2037]: yarn client mode doesn't support spark.yarn.max.executor.failures

2014-07-24 Thread tgraves
Repository: spark Updated Branches: refs/heads/master c960b5051 -> 323a83c52 [SPARK-2037]: yarn client mode doesn't support spark.yarn.max.executor.failures Author: GuoQiang Li Closes #1180 from witgo/SPARK-2037 and squashes the following commits: 3d52411 [GuoQiang Li] review commit 7058f4d

git commit: SPARK-2150: Provide direct link to finished application UI in yarn resou...

2014-07-24 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 42dfab7d3 -> 46e224aaa SPARK-2150: Provide direct link to finished application UI in yarn resou... ...rce manager UI Use the event logger directory to provide a direct link to finished application UI in yarn resourcemanager UI. Author: Ra

git commit: [YARN] SPARK-2577: File upload to viewfs is broken due to mount point re...

2014-07-22 Thread tgraves
Repository: spark Updated Branches: refs/heads/master ddadf1b00 -> 02e457294 [YARN] SPARK-2577: File upload to viewfs is broken due to mount point re... Opting to the option 2 defined in SPARK-2577, i.e., retrieve and pass the correct file system object to addResource. Author: Gera Shegalov

git commit: [YARN][SPARK-2606]:In some cases, the spark UI pages display incorrect

2014-07-22 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 5f7b99168 -> ddadf1b00 [YARN][SPARK-2606]:In some cases,the spark UI pages display incorrect The issue is caused by #1112 . Author: GuoQiang Li Closes #1501 from witgo/webui_style and squashes the following commits: 4b34998 [GuoQiang Li

git commit: SPARK-1707. Remove unnecessary 3 second sleep in YarnClusterScheduler

2014-07-21 Thread tgraves
Repository: spark Updated Branches: refs/heads/master cd273a238 -> f89cf65d7 SPARK-1707. Remove unnecessary 3 second sleep in YarnClusterScheduler Author: Sandy Ryza Closes #634 from sryza/sandy-spark-1707 and squashes the following commits: 2f6e358 [Sandy Ryza] Default min registered execu

git commit: SPARK-1291: Link the spark UI to RM ui in yarn-client mode

2014-07-15 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 9dd635eb5 -> 72ea56da8 SPARK-1291: Link the spark UI to RM ui in yarn-client mode Author: witgo Closes #1112 from witgo/SPARK-1291 and squashes the following commits: 6022bcd [witgo] review commit 1fbb925 [witgo] add addAmIpFilter to yar

git commit: [SPARK-1946] Submit tasks after (configured ratio) executors have been registered

2014-07-14 Thread tgraves
Repository: spark Updated Branches: refs/heads/master d60b09bb6 -> 3dd8af7a6 [SPARK-1946] Submit tasks after (configured ratio) executors have been registered Because submitting tasks and registering executors are asynchronous, in most situation, early stages' tasks run without preferred loc

git commit: SPARK-2400 : fix spark.yarn.max.executor.failures explaination

2014-07-08 Thread tgraves
Repository: spark Updated Branches: refs/heads/master c8a2313cd -> b520b6453 SPARK-2400 : fix spark.yarn.max.executor.failures explaination According to ```scala private val maxNumExecutorFailures = sparkConf.getInt("spark.yarn.max.executor.failures", sparkConf.getInt("spark.yarn.max.wo

svn commit: r1606915 - in /hadoop/common/site/main: author/src/documentation/content/xdocs/ publish/ publish/docs/r0.23.11/ publish/docs/r0.23.11/api/ publish/docs/r0.23.11/api/org/ publish/docs/r0.23

2014-06-30 Thread tgraves
Author: tgraves Date: Mon Jun 30 20:14:13 2014 New Revision: 1606915 URL: http://svn.apache.org/r1606915 Log: Updated site for release 0.23.11 [This commit notification would consist of 606 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r1606091 - in /hadoop/common/tags: release-0.23.11-rc0/ release-0.23.11/

2014-06-27 Thread tgraves
Author: tgraves Date: Fri Jun 27 13:25:29 2014 New Revision: 1606091 URL: http://svn.apache.org/r1606091 Log: Hadoop 0.23.11 release. Added: hadoop/common/tags/release-0.23.11/ (props changed) - copied from r1606090, hadoop/common/tags/release-0.23.11-rc0/ Removed: hadoop/common

git commit: Remove use of spark.worker.instances

2014-06-26 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 47f8829e0 -> 2d3080855 Remove use of spark.worker.instances spark.worker.instances was added as part of this commit: https://github.com/apache/spark/commit/1617816090e7b20124a512a43860a21232ebf511 My understanding is that SPARK_WORKER

git commit: Remove use of spark.worker.instances

2014-06-26 Thread tgraves
Repository: spark Updated Branches: refs/heads/master e4899a253 -> 48a82a827 Remove use of spark.worker.instances spark.worker.instances was added as part of this commit: https://github.com/apache/spark/commit/1617816090e7b20124a512a43860a21232ebf511 My understanding is that SPARK_WORKER_INS

git commit: Fixed small running on YARN docs typo

2014-06-23 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 dedd70903 -> d51eba56c Fixed small running on YARN docs typo The backslash is needed for multiline command Author: Vlad Closes #1158 from frol/patch-1 and squashes the following commits: e258044 [Vlad] Fixed small running on YARN do

git commit: Fixed small running on YARN docs typo

2014-06-23 Thread tgraves
Repository: spark Updated Branches: refs/heads/master e380767de -> b88238fae Fixed small running on YARN docs typo The backslash is needed for multiline command Author: Vlad Closes #1158 from frol/patch-1 and squashes the following commits: e258044 [Vlad] Fixed small running on YARN docs t

git commit: [SPARK-1395] Fix "local:" URI support in Yarn mode (again).

2014-06-23 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 9cb64b2c5 -> e380767de [SPARK-1395] Fix "local:" URI support in Yarn mode (again). Recent changes ignored the fact that path may be defined with "local:" URIs, which means they need to be explicitly added to the classpath everywhere a remot

git commit: [SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work

2014-06-19 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 67fca189c -> bce0897bc [SPARK-2051]In yarn.ClientBase spark.yarn.dist.* do not work Author: witgo Closes #969 from witgo/yarn_ClientBase and squashes the following commits: 8117765 [witgo] review commit 3bdbc52 [witgo] Merge branch 'mast

svn commit: r1603696 - /hadoop/common/tags/release-0.23.11-rc0/

2014-06-18 Thread tgraves
Author: tgraves Date: Thu Jun 19 01:32:25 2014 New Revision: 1603696 URL: http://svn.apache.org/r1603696 Log: Hadoop 0.23.11-rc0 release. Added: hadoop/common/tags/release-0.23.11-rc0/ (props changed) - copied from r1603695, hadoop/common/branches/branch-0.23.11/ Propchange: hadoop

svn commit: r1603694 - in /hadoop/common/branches/branch-0.23.11/hadoop-mapreduce-project: ./ hadoop-mapreduce-client/ hadoop-mapreduce-client/hadoop-mapreduce-client-app/ hadoop-mapreduce-client/hado

2014-06-18 Thread tgraves
Author: tgraves Date: Thu Jun 19 01:17:43 2014 New Revision: 1603694 URL: http://svn.apache.org/r1603694 Log: Preparing for release 0.23.11 Modified: hadoop/common/branches/branch-0.23.11/hadoop-mapreduce-project/CHANGES.txt hadoop/common/branches/branch-0.23.11/hadoop-mapreduce-project

svn commit: r1603694 - in /hadoop/common/branches/branch-0.23.11/hadoop-yarn-project: ./ hadoop-yarn/ hadoop-yarn/hadoop-yarn-api/ hadoop-yarn/hadoop-yarn-applications/ hadoop-yarn/hadoop-yarn-applica

2014-06-18 Thread tgraves
Author: tgraves Date: Thu Jun 19 01:17:43 2014 New Revision: 1603694 URL: http://svn.apache.org/r1603694 Log: Preparing for release 0.23.11 Modified: hadoop/common/branches/branch-0.23.11/hadoop-yarn-project/CHANGES.txt hadoop/common/branches/branch-0.23.11/hadoop-yarn-project/hadoop

svn commit: r1603694 [1/3] - in /hadoop/common/branches/branch-0.23.11/hadoop-common-project: ./ hadoop-annotations/ hadoop-auth-examples/ hadoop-auth/ hadoop-common/ hadoop-common/src/main/docs/

2014-06-18 Thread tgraves
Author: tgraves Date: Thu Jun 19 01:17:43 2014 New Revision: 1603694 URL: http://svn.apache.org/r1603694 Log: Preparing for release 0.23.11 Modified: hadoop/common/branches/branch-0.23.11/hadoop-common-project/hadoop-annotations/pom.xml hadoop/common/branches/branch-0.23.11/hadoop

svn commit: r1603694 - in /hadoop/common/branches/branch-0.23.11/hadoop-hdfs-project: hadoop-hdfs-httpfs/pom.xml hadoop-hdfs/CHANGES.txt hadoop-hdfs/pom.xml hadoop-hdfs/src/test/aop/build/aop.xml pom.

2014-06-18 Thread tgraves
Author: tgraves Date: Thu Jun 19 01:17:43 2014 New Revision: 1603694 URL: http://svn.apache.org/r1603694 Log: Preparing for release 0.23.11 Modified: hadoop/common/branches/branch-0.23.11/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop/common/branches/branch-0.23.11/hadoop-hdfs

svn commit: r1603694 - in /hadoop/common/branches/branch-0.23.11: ./ hadoop-assemblies/ hadoop-client/ hadoop-dist/ hadoop-minicluster/ hadoop-project-dist/ hadoop-project/ hadoop-tools/ hadoop-tools/

2014-06-18 Thread tgraves
Author: tgraves Date: Thu Jun 19 01:17:43 2014 New Revision: 1603694 URL: http://svn.apache.org/r1603694 Log: Preparing for release 0.23.11 Modified: hadoop/common/branches/branch-0.23.11/hadoop-assemblies/pom.xml hadoop/common/branches/branch-0.23.11/hadoop-client/pom.xml hadoop

svn commit: r1603694 [3/3] - in /hadoop/common/branches/branch-0.23.11/hadoop-common-project: ./ hadoop-annotations/ hadoop-auth-examples/ hadoop-auth/ hadoop-common/ hadoop-common/src/main/docs/

2014-06-18 Thread tgraves
Modified: hadoop/common/branches/branch-0.23.11/hadoop-common-project/pom.xml URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23.11/hadoop-common-project/pom.xml?rev=1603694&r1=1603693&r2=1603694&view=diff ==

svn commit: r1603642 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ hadoop-mapreduce-client/ hadoop-mapreduce-client/hadoop-mapreduce-client-app/ hadoop-mapreduce-client/hadoop-

2014-06-18 Thread tgraves
Author: tgraves Date: Wed Jun 18 21:47:05 2014 New Revision: 1603642 URL: http://svn.apache.org/r1603642 Log: Preparing for 0.23.12 development Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt hadoop/common/branches/branch-0.23/hadoop-mapreduce-project

svn commit: r1603642 - in /hadoop/common/branches/branch-0.23/hadoop-common-project: hadoop-annotations/pom.xml hadoop-auth-examples/pom.xml hadoop-auth/pom.xml hadoop-common/CHANGES.txt hadoop-common

2014-06-18 Thread tgraves
Author: tgraves Date: Wed Jun 18 21:47:05 2014 New Revision: 1603642 URL: http://svn.apache.org/r1603642 Log: Preparing for 0.23.12 development Modified: hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-annotations/pom.xml hadoop/common/branches/branch-0.23/hadoop-common

svn commit: r1603642 - in /hadoop/common/branches/branch-0.23/hadoop-yarn-project: ./ hadoop-yarn/ hadoop-yarn/hadoop-yarn-api/ hadoop-yarn/hadoop-yarn-applications/ hadoop-yarn/hadoop-yarn-applicatio

2014-06-18 Thread tgraves
Author: tgraves Date: Wed Jun 18 21:47:05 2014 New Revision: 1603642 URL: http://svn.apache.org/r1603642 Log: Preparing for 0.23.12 development Modified: hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn

svn commit: r1603642 - in /hadoop/common/branches/branch-0.23/hadoop-hdfs-project: hadoop-hdfs-httpfs/pom.xml hadoop-hdfs/CHANGES.txt hadoop-hdfs/pom.xml hadoop-hdfs/src/test/aop/build/aop.xml pom.xml

2014-06-18 Thread tgraves
Author: tgraves Date: Wed Jun 18 21:47:05 2014 New Revision: 1603642 URL: http://svn.apache.org/r1603642 Log: Preparing for 0.23.12 development Modified: hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop/common/branches/branch-0.23/hadoop-hdfs

svn commit: r1603642 - in /hadoop/common/branches/branch-0.23: ./ hadoop-assemblies/ hadoop-client/ hadoop-dist/ hadoop-minicluster/ hadoop-project-dist/ hadoop-project/ hadoop-tools/ hadoop-tools/had

2014-06-18 Thread tgraves
Author: tgraves Date: Wed Jun 18 21:47:05 2014 New Revision: 1603642 URL: http://svn.apache.org/r1603642 Log: Preparing for 0.23.12 development Modified: hadoop/common/branches/branch-0.23/hadoop-assemblies/pom.xml hadoop/common/branches/branch-0.23/hadoop-client/pom.xml hadoop

svn commit: r1603641 - /hadoop/common/branches/branch-0.23.11/

2014-06-18 Thread tgraves
Author: tgraves Date: Wed Jun 18 21:43:10 2014 New Revision: 1603641 URL: http://svn.apache.org/r1603641 Log: Branching for 0.23.11 releases Added: hadoop/common/branches/branch-0.23.11/ (props changed) - copied from r1603640, hadoop/common/branches/branch-0.23/ Propchange: hadoop

git commit: [SPARK-1930] The Container is running beyond physical memory limits, so as to be killed

2014-06-16 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 33db842b6 -> d7f94b934 [SPARK-1930] The Container is running beyond physical memory limits, so as to be killed Author: witgo Closes #894 from witgo/SPARK-1930 and squashes the following commits: 564307e [witgo] Update the running-on

git commit: [SPARK-1930] The Container is running beyond physical memory limits, so as to be killed

2014-06-16 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 4fdb49177 -> cdf2b0457 [SPARK-1930] The Container is running beyond physical memory limits, so as to be killed Author: witgo Closes #894 from witgo/SPARK-1930 and squashes the following commits: 564307e [witgo] Update the running-on-yar

git commit: [SPARK-2080] Yarn: report HS URL in client mode, correct user in cluster mode.

2014-06-12 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 358e7e51c -> 3962abaf9 [SPARK-2080] Yarn: report HS URL in client mode, correct user in cluster mode. Yarn client mode was not setting the app's tracking URL to the History Server's URL when configured by the user. Now client mode behav

git commit: [SPARK-2080] Yarn: report HS URL in client mode, correct user in cluster mode.

2014-06-12 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 83c226d45 -> ecde5b837 [SPARK-2080] Yarn: report HS URL in client mode, correct user in cluster mode. Yarn client mode was not setting the app's tracking URL to the History Server's URL when configured by the user. Now client mode behaves t

git commit: SPARK-1639. Tidy up some Spark on YARN code

2014-06-11 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 6e1193031 -> 2a4225dd9 SPARK-1639. Tidy up some Spark on YARN code This contains a bunch of small tidyings of the Spark on YARN code. I focused on the yarn stable code. @tgravescs, let me know if you'd like me to make these for the alpha

git commit: [SPARK-1978] In some cases, spark-yarn does not automatically restart the failed container

2014-06-10 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 89caa40e3 -> 35894af8c [SPARK-1978] In some cases, spark-yarn does not automatically restart the failed container Author: witgo Closes #921 from witgo/allocateExecutors and squashes the following commits: bc3aa66 [witgo] review comm

git commit: [SPARK-1978] In some cases, spark-yarn does not automatically restart the failed container

2014-06-10 Thread tgraves
Repository: spark Updated Branches: refs/heads/master a9a461c59 -> 884ca718b [SPARK-1978] In some cases, spark-yarn does not automatically restart the failed container Author: witgo Closes #921 from witgo/allocateExecutors and squashes the following commits: bc3aa66 [witgo] review commit 8

git commit: [SPARK-1522] : YARN ClientBase throws a NPE if there is no YARN Application CP

2014-06-09 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 6cf335d79 -> e27344768 [SPARK-1522] : YARN ClientBase throws a NPE if there is no YARN Application CP The current implementation of ClientBase.getDefaultYarnApplicationClasspath inspects the MRJobConfig class for the field DEFAULT_YARN_APP

git commit: SPARK-1557 Set permissions on event log files/directories

2014-04-29 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 c27ce2b4f -> bccd13ec1 SPARK-1557 Set permissions on event log files/directories This adds minimal setting of event log directory/files permissions. To have a secure environment the user must manually create the top level event log d

git commit: SPARK-1557 Set permissions on event log files/directories

2014-04-29 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 9a1184a8a -> 8db0f7e28 SPARK-1557 Set permissions on event log files/directories This adds minimal setting of event log directory/files permissions. To have a secure environment the user must manually create the top level event log direc

git commit: SPARK-1408 Modify Spark on Yarn to point to the history server when app ...

2014-04-17 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 a83a794f1 -> 6195fb8db SPARK-1408 Modify Spark on Yarn to point to the history server when app ... ...finishes Note this is dependent on https://github.com/apache/spark/pull/204 to have a working history server, but there are no code

git commit: SPARK-1408 Modify Spark on Yarn to point to the history server when app ...

2014-04-17 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 69047506b -> 0058b5d2c SPARK-1408 Modify Spark on Yarn to point to the history server when app ... ...finishes Note this is dependent on https://github.com/apache/spark/pull/204 to have a working history server, but there are no code depe

git commit: [SPARK-1395] Allow "local:" URIs to work on Yarn.

2014-04-17 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 b3ad707c4 -> a83a794f1 [SPARK-1395] Allow "local:" URIs to work on Yarn. This only works for the three paths defined in the environment (SPARK_JAR, SPARK_YARN_APP_JAR and SPARK_LOG4J_CONF). Tested by running SparkPi with local: and fil

svn commit: r1588287 - in /hadoop/common/branches/branch-2.4/hadoop-yarn-project: CHANGES.txt hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java

2014-04-17 Thread tgraves
Author: tgraves Date: Thu Apr 17 15:31:58 2014 New Revision: 1588287 URL: http://svn.apache.org/r1588287 Log: Merge 1588281 to branch-2.4. YARN-1931. Private API change in YARN-1824 in 2.4 broke compatibility with previous releases (Sandy Ryza via tgraves) Modified: hadoop/common/branches

git commit: [SPARK-1395] Allow "local:" URIs to work on Yarn.

2014-04-17 Thread tgraves
Repository: spark Updated Branches: refs/heads/master bb76eae1b -> 69047506b [SPARK-1395] Allow "local:" URIs to work on Yarn. This only works for the three paths defined in the environment (SPARK_JAR, SPARK_YARN_APP_JAR and SPARK_LOG4J_CONF). Tested by running SparkPi with local: and file: U

svn commit: r1588281 - in /hadoop/common/trunk/hadoop-yarn-project: CHANGES.txt hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java

2014-04-17 Thread tgraves
Author: tgraves Date: Thu Apr 17 15:23:22 2014 New Revision: 1588281 URL: http://svn.apache.org/r1588281 Log: YARN-1931. Private API change in YARN-1824 in 2.4 broke compatibility with previous releases (Sandy Ryza via tgraves) Modified: hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt

svn commit: r1588283 - in /hadoop/common/branches/branch-2/hadoop-yarn-project: CHANGES.txt hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java

2014-04-17 Thread tgraves
Author: tgraves Date: Thu Apr 17 15:25:27 2014 New Revision: 1588283 URL: http://svn.apache.org/r1588283 Log: Merge 1588281 to branch-2. YARN-1931. Private API change in YARN-1824 in 2.4 broke compatibility with previous releases (Sandy Ryza via tgraves) Modified: hadoop/common/branches

git commit: SPARK-1465: Spark compilation is broken with the latest hadoop-2.4.0 release

2014-04-16 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 b75301f1f -> d8fc4a4b6 SPARK-1465: Spark compilation is broken with the latest hadoop-2.4.0 release YARN-1824 changes the APIs (addToEnvironment, setEnvFromInputString) in Apps, which causes the spark build to break if built against a

git commit: SPARK-1465: Spark compilation is broken with the latest hadoop-2.4.0 release

2014-04-16 Thread tgraves
Repository: spark Updated Branches: refs/heads/master e269c24db -> 725925cf2 SPARK-1465: Spark compilation is broken with the latest hadoop-2.4.0 release YARN-1824 changes the APIs (addToEnvironment, setEnvFromInputString) in Apps, which causes the spark build to break if built against a vers

git commit: SPARK-1417: Spark on Yarn - spark UI link from resourcemanager is broken

2014-04-11 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-1.0 a1bb4c695 -> 9e90c464d SPARK-1417: Spark on Yarn - spark UI link from resourcemanager is broken Author: Thomas Graves Closes #344 from tgravescs/SPARK-1417 and squashes the following commits: c450b5f [Thomas Graves] fix test e1c1d7e

git commit: SPARK-1252. On YARN, use container-log4j.properties for executors

2014-04-07 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 83f2a2f14 -> 9dd8b9166 SPARK-1252. On YARN, use container-log4j.properties for executors container-log4j.properties is a file that YARN provides so that containers can have log4j.properties distinct from that of the NodeManagers. Logs now

git commit: SPARK-1350. Always use JAVA_HOME to run executor container JVMs.

2014-04-04 Thread tgraves
Repository: spark Updated Branches: refs/heads/master ee6e9e7d8 -> 7f32fd42a SPARK-1350. Always use JAVA_HOME to run executor container JVMs. Author: Sandy Ryza Closes #313 from sryza/sandy-spark-1350 and squashes the following commits: bb6d187 [Sandy Ryza] SPARK-1350. Always use JAVA_HOME

git commit: SPARK-1330 removed extra echo from comput_classpath.sh

2014-03-27 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 5b2d863e3 -> 426042ad2 SPARK-1330 removed extra echo from comput_classpath.sh remove the extra echo which prevents spark-class from working. Note that I did not update the comment above it, which is also wrong because I'm not sure what i

git commit: SPARK-1032. If Yarn app fails before registering, app master stays aroun...

2014-03-20 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-0.9 748f002b3 -> c6630d363 SPARK-1032. If Yarn app fails before registering, app master stays aroun... ...d long after This reopens https://github.com/apache/incubator-spark/pull/648 against the new repo. Author: Sandy Ryza Closes #28

git commit: SPARK-1051. On YARN, executors don't doAs submitting user

2014-03-20 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-0.9 1e366904c -> 748f002b3 SPARK-1051. On YARN, executors don't doAs submitting user This reopens https://github.com/apache/incubator-spark/pull/538 against the new repo Author: Sandy Ryza Closes #29 from sryza/sandy-spark-1051 and squa

git commit: Update the yarn alpha version to 0.9.1-incubating-SNAPSHOT

2014-03-19 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-0.9 250ec271c -> 72875b29b Update the yarn alpha version to 0.9.1-incubating-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/72875b29 Tree: http://git-wip-us.a

git commit: SPARK-1203 fix saving to hdfs from yarn

2014-03-19 Thread tgraves
Repository: spark Updated Branches: refs/heads/branch-0.9 d385b5a19 -> 250ec271c SPARK-1203 fix saving to hdfs from yarn Author: Thomas Graves Closes #173 from tgravescs/SPARK-1203 and squashes the following commits: 4fd5ded [Thomas Graves] adding import 964e3f7 [Thomas Graves] SPARK-1203 f

git commit: SPARK-1203 fix saving to hdfs from yarn

2014-03-19 Thread tgraves
Repository: spark Updated Branches: refs/heads/master d55ec86de -> 6112270c9 SPARK-1203 fix saving to hdfs from yarn Author: Thomas Graves Closes #173 from tgravescs/SPARK-1203 and squashes the following commits: 4fd5ded [Thomas Graves] adding import 964e3f7 [Thomas Graves] SPARK-1203 fix s

[1/3] SPARK-1189: Add Security to Spark - Akka, Http, ConnectionManager, UI use servlets

2014-03-06 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 40566e10a -> 7edbea41b http://git-wip-us.apache.org/repos/asf/spark/blob/7edbea41/core/src/test/scala/org/apache/spark/metrics/MetricsSystemSuite.scala -- diff --git a/core

git commit: SPARK-1051. On YARN, executors don't doAs submitting user

2014-02-28 Thread tgraves
Repository: spark Updated Branches: refs/heads/master 5f419bf9f -> 46dff3445 SPARK-1051. On YARN, executors don't doAs submitting user This reopens https://github.com/apache/incubator-spark/pull/538 against the new repo Author: Sandy Ryza Closes #29 from sryza/sandy-spark-1051 and squashes

git commit: SPARK-1032. If Yarn app fails before registering, app master stays aroun...

2014-02-28 Thread tgraves
Repository: spark Updated Branches: refs/heads/master edf8a56ab -> 5f419bf9f SPARK-1032. If Yarn app fails before registering, app master stays aroun... ...d long after This reopens https://github.com/apache/incubator-spark/pull/648 against the new repo. Author: Sandy Ryza Closes #28 from

git commit: SPARK-1053. Don't require SPARK_YARN_APP_JAR

2014-02-26 Thread tgraves
Repository: spark Updated Branches: refs/heads/master c852201ce -> b8a187195 SPARK-1053. Don't require SPARK_YARN_APP_JAR It looks this just requires taking out the checks. I verified that, with the patch, I was able to run spark-shell through yarn without setting the environment variable.

[2/4] git commit: Yarn workerRunnable refactor

2014-01-15 Thread tgraves
Yarn workerRunnable refactor Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/161ab939 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/161ab939 Diff: http://git-wip-us.apache.org/repos/asf/

[4/4] git commit: Merge pull request #366 from colorant/yarn-dev

2014-01-15 Thread tgraves
Merge pull request #366 from colorant/yarn-dev More yarn code refactor Try to retrive common code in yarn alpha/stable for client and workerRunnable to reduce duplicated codes. By put them into a trait in common dir and extends with them. Same works could be done for the remaining files in al

[3/4] git commit: Address comments to fix code formats

2014-01-15 Thread tgraves
Address comments to fix code formats Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/4c22c55a Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/4c22c55a Diff: http://git-wip-us.apache.org/re

[3/4] git commit: Update based on review comments

2014-01-10 Thread tgraves
Update based on review comments Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/9bdfbc04 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/9bdfbc04 Diff: http://git-wip-us.apache.org/repos/a

[2/4] git commit: add application type

2014-01-10 Thread tgraves
add application type Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/5fccd705 Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/5fccd705 Diff: http://git-wip-us.apache.org/repos/asf/incubato

[1/4] git commit: yarn-client addJar fix and misc other

2014-01-10 Thread tgraves
Updated Branches: refs/heads/master 7b58f116e -> 7cef8435d yarn-client addJar fix and misc other Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/c617083e Tree: http://git-wip-us.apache.org/repos/asf/inc

[4/4] git commit: Merge pull request #371 from tgravescs/yarn_client_addjar_misc_fixes

2014-01-10 Thread tgraves
Merge pull request #371 from tgravescs/yarn_client_addjar_misc_fixes Yarn client addjar and misc fixes Fix the addJar functionality in yarn-client mode, add support for the other options supported in yarn-standalone mode, set the application type on yarn in hadoop 2.X, add documentation, change

[3/3] git commit: Merge pull request #345 from colorant/yarn

2014-01-08 Thread tgraves
Merge pull request #345 from colorant/yarn support distributing extra files to worker for yarn client mode So that user doesn't need to package all dependency into one assemble jar as spark app jar Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.a

[2/3] git commit: Export --file for YarnClient mode to support sending extra files to worker on yarn cluster

2014-01-08 Thread tgraves
Export --file for YarnClient mode to support sending extra files to worker on yarn cluster Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/67af8031 Tree: http://git-wip-us.apache.org/repos/asf/incubator-sp

[1/3] git commit: Minor typo fix for yarn client

2014-01-08 Thread tgraves
Updated Branches: refs/heads/master bb6a39a68 -> 6eef78d76 Minor typo fix for yarn client Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/da4694a0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-

svn commit: r1550240 - in /hadoop/common/site/main: author/src/documentation/content/xdocs/ publish/ publish/docs/r0.23.10/ publish/docs/r0.23.10/api/ publish/docs/r0.23.10/api/org/ publish/docs/r0.23

2013-12-11 Thread tgraves
Author: tgraves Date: Wed Dec 11 19:38:50 2013 New Revision: 1550240 URL: http://svn.apache.org/r1550240 Log: Updated site for release 0.23.10 [This commit notification would consist of 606 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.]

svn commit: r1549854 - in /hadoop/common/tags: release-0.23.10-rc0/ release-0.23.10/

2013-12-10 Thread tgraves
Author: tgraves Date: Tue Dec 10 14:48:45 2013 New Revision: 1549854 URL: http://svn.apache.org/r1549854 Log: Hadoop 0.23.10 release Added: hadoop/common/tags/release-0.23.10/ (props changed) - copied from r1549853, hadoop/common/tags/release-0.23.10-rc0/ Removed: hadoop/common

Git Push Summary

2013-12-03 Thread tgraves
Updated Branches: refs/heads/testcommit [deleted] 58d9bbcfe

Git Push Summary

2013-12-03 Thread tgraves
Updated Branches: refs/heads/testcommit [created] 58d9bbcfe

svn commit: r1547277 - /hadoop/common/tags/release-0.23.10-rc0/

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:13:20 2013 New Revision: 1547277 URL: http://svn.apache.org/r1547277 Log: Hadoop 0.23.10-rc0 release Added: hadoop/common/tags/release-0.23.10-rc0/ (props changed) - copied from r1547276, hadoop/common/branches/branch-0.23.10/ Propchange: hadoop

svn commit: r1547276 [1/3] - in /hadoop/common/branches/branch-0.23.10/hadoop-common-project: hadoop-annotations/pom.xml hadoop-auth-examples/pom.xml hadoop-auth/pom.xml hadoop-common/pom.xml hadoop-c

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:11:20 2013 New Revision: 1547276 URL: http://svn.apache.org/r1547276 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23.10/hadoop-common-project/hadoop-annotations/pom.xml hadoop/common/branches/branch-0.23.10/hadoop

svn commit: r1547276 - in /hadoop/common/branches/branch-0.23.10: ./ hadoop-assemblies/ hadoop-client/ hadoop-dist/ hadoop-minicluster/ hadoop-project-dist/ hadoop-project/ hadoop-tools/ hadoop-tools/

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:11:20 2013 New Revision: 1547276 URL: http://svn.apache.org/r1547276 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23.10/hadoop-assemblies/pom.xml hadoop/common/branches/branch-0.23.10/hadoop-client/pom.xml hadoop

svn commit: r1547276 - in /hadoop/common/branches/branch-0.23.10/hadoop-hdfs-project: hadoop-hdfs-httpfs/pom.xml hadoop-hdfs/pom.xml hadoop-hdfs/src/test/aop/build/aop.xml pom.xml

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:11:20 2013 New Revision: 1547276 URL: http://svn.apache.org/r1547276 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23.10/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop/common/branches/branch-0.23.10/hadoop-hdfs

svn commit: r1547276 - in /hadoop/common/branches/branch-0.23.10/hadoop-yarn-project: ./ hadoop-yarn/ hadoop-yarn/hadoop-yarn-api/ hadoop-yarn/hadoop-yarn-applications/ hadoop-yarn/hadoop-yarn-applica

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:11:20 2013 New Revision: 1547276 URL: http://svn.apache.org/r1547276 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23.10/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/pom.xml hadoop/common/branches/branch-0.23.10

svn commit: r1547276 [3/3] - in /hadoop/common/branches/branch-0.23.10/hadoop-common-project: hadoop-annotations/pom.xml hadoop-auth-examples/pom.xml hadoop-auth/pom.xml hadoop-common/pom.xml hadoop-c

2013-12-02 Thread tgraves
Modified: hadoop/common/branches/branch-0.23.10/hadoop-common-project/pom.xml URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23.10/hadoop-common-project/pom.xml?rev=1547276&r1=1547275&r2=1547276&view=diff ==

svn commit: r1547276 - in /hadoop/common/branches/branch-0.23.10/hadoop-mapreduce-project: ./ hadoop-mapreduce-client/ hadoop-mapreduce-client/hadoop-mapreduce-client-app/ hadoop-mapreduce-client/hado

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:11:20 2013 New Revision: 1547276 URL: http://svn.apache.org/r1547276 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23.10/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/pom.xml hadoop/common

svn commit: r1547275 - in /hadoop/common/branches/branch-0.23/hadoop-yarn-project: ./ hadoop-yarn/ hadoop-yarn/hadoop-yarn-api/ hadoop-yarn/hadoop-yarn-applications/ hadoop-yarn/hadoop-yarn-applicatio

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:06:37 2013 New Revision: 1547275 URL: http://svn.apache.org/r1547275 Log: Preparing for 0.23.11 development Modified: hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn

svn commit: r1547275 - in /hadoop/common/branches/branch-0.23: ./ hadoop-assemblies/ hadoop-client/ hadoop-dist/ hadoop-minicluster/ hadoop-project-dist/ hadoop-project/ hadoop-tools/ hadoop-tools/had

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:06:37 2013 New Revision: 1547275 URL: http://svn.apache.org/r1547275 Log: Preparing for 0.23.11 development Modified: hadoop/common/branches/branch-0.23/hadoop-assemblies/pom.xml hadoop/common/branches/branch-0.23/hadoop-client/pom.xml hadoop

svn commit: r1547275 - in /hadoop/common/branches/branch-0.23/hadoop-common-project: hadoop-annotations/pom.xml hadoop-auth-examples/pom.xml hadoop-auth/pom.xml hadoop-common/CHANGES.txt hadoop-common

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:06:37 2013 New Revision: 1547275 URL: http://svn.apache.org/r1547275 Log: Preparing for 0.23.11 development Modified: hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-annotations/pom.xml hadoop/common/branches/branch-0.23/hadoop-common

svn commit: r1547275 - in /hadoop/common/branches/branch-0.23/hadoop-hdfs-project: hadoop-hdfs-httpfs/pom.xml hadoop-hdfs/CHANGES.txt hadoop-hdfs/pom.xml hadoop-hdfs/src/test/aop/build/aop.xml pom.xml

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:06:37 2013 New Revision: 1547275 URL: http://svn.apache.org/r1547275 Log: Preparing for 0.23.11 development Modified: hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop/common/branches/branch-0.23/hadoop-hdfs

svn commit: r1547275 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: ./ hadoop-mapreduce-client/ hadoop-mapreduce-client/hadoop-mapreduce-client-app/ hadoop-mapreduce-client/hadoop-

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:06:37 2013 New Revision: 1547275 URL: http://svn.apache.org/r1547275 Log: Preparing for 0.23.11 development Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt hadoop/common/branches/branch-0.23/hadoop-mapreduce-project

svn commit: r1547274 - /hadoop/common/branches/branch-0.23.10/

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:02:08 2013 New Revision: 1547274 URL: http://svn.apache.org/r1547274 Log: Branching for 0.23.10 release Added: hadoop/common/branches/branch-0.23.10/ (props changed) - copied from r1547273, hadoop/common/branches/branch-0.23/ Propchange: hadoop

svn commit: r1547273 - /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:00:59 2013 New Revision: 1547273 URL: http://svn.apache.org/r1547273 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt Modified: hadoop/common/branches/branch-0.23/hadoop

svn commit: r1547273 - /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:00:59 2013 New Revision: 1547273 URL: http://svn.apache.org/r1547273 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt Modified: hadoop/common/branches/branch-0.23/hadoop-yarn-project

svn commit: r1547273 - /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:00:59 2013 New Revision: 1547273 URL: http://svn.apache.org/r1547273 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Modified: hadoop/common/branches/branch-0.23/hadoop-hdfs

svn commit: r1547273 - /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt

2013-12-02 Thread tgraves
Author: tgraves Date: Tue Dec 3 04:00:59 2013 New Revision: 1547273 URL: http://svn.apache.org/r1547273 Log: Preparing for release 0.23.10 Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce

svn commit: r1524841 - in /hadoop/common/trunk/hadoop-mapreduce-project: CHANGES.txt hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java

2013-09-19 Thread tgraves
Author: tgraves Date: Thu Sep 19 20:58:18 2013 New Revision: 1524841 URL: http://svn.apache.org/r1524841 Log: MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta via tgraves) Modified: hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt hadoop/common/trunk

svn commit: r1524842 - in /hadoop/common/branches/branch-2.1-beta/hadoop-mapreduce-project: CHANGES.txt hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce

2013-09-19 Thread tgraves
Author: tgraves Date: Thu Sep 19 20:59:47 2013 New Revision: 1524842 URL: http://svn.apache.org/r1524842 Log: MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta via tgraves) Modified: hadoop/common/branches/branch-2.1-beta/hadoop-mapreduce-project/CHANGES.txt

svn commit: r1524844 - in /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project: CHANGES.txt hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/Typ

2013-09-19 Thread tgraves
Author: tgraves Date: Thu Sep 19 21:00:52 2013 New Revision: 1524844 URL: http://svn.apache.org/r1524844 Log: MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta via tgraves) Modified: hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt hadoop

svn commit: r1524843 - in /hadoop/common/branches/branch-2/hadoop-mapreduce-project: CHANGES.txt hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeCo

2013-09-19 Thread tgraves
Author: tgraves Date: Thu Sep 19 21:00:40 2013 New Revision: 1524843 URL: http://svn.apache.org/r1524843 Log: MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta via tgraves) Modified: hadoop/common/branches/branch-2/hadoop-mapreduce-project/CHANGES.txt hadoop

<    1   2   3   4   5   6   7   8   9   10   >