[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47497190 Thanks. I'm merging this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1245 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2059][SQL] Add analysis checks
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1265#issuecomment-47497677 This change actually exposes two other problems ... ``` -- TOK_ALLCOLREF explain select * from (select a.key, a.* from (select * from src) a join (select * from src1) b on (a.key = b.key)) t; -- DOT explain select * from (select a.key, a.`[k].*` from (select * from src) a join (select * from src1) b on (a.key = b.key)) t; -- EXPRESSION explain select * from (select a.key, a.key from (select * from src) a join (select * from src1) b on (a.key = b.key)) t; [info] - ambiguous_col *** FAILED *** [info] Failed to execute query using catalyst: [info] Error: Unresolved attributes: *, tree: [info] Project [*] [info]Subquery t [info] Project [key#1302,'a.[k].*] [info] Join Inner, Some((key#1302 = key#1311)) [info] Subquery a [info]Project [key#1302,value#1303] [info] LowerCaseSchema [info] InMemoryRelation [key#1302,value#1303], false, (HiveTableScan [key#1302,value#1303], (MetastoreRelation default, src, None), None) [info] Subquery b [info]Project [key#1311,value#1312] [info] LowerCaseSchema [info] InMemoryRelation [key#1311,value#1312], false, (HiveTableScan [key#1311,value#1312], (MetastoreRelation default, src1, None), None) [info] [info] org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved attributes: *, tree: [info] Project [*] [info]Subquery t [info] Project [key#1302,'a.[k].*] [info] Join Inner, Some((key#1302 = key#1311)) [info] Subquery a [info]Project [key#1302,value#1303] [info] LowerCaseSchema [info] InMemoryRelation [key#1302,value#1303], false, (HiveTableScan [key#1302,value#1303], (MetastoreRelation default, src, None), None) [info] Subquery b [info]Project [key#1311,value#1312] [info] LowerCaseSchema [info] InMemoryRelation [key#1311,value#1312], false, (HiveTableScan [key#1311,value#1312], (MetastoreRelation default, src1, None), None) ``` ``` -- HIVE-2477 Use name of original expression for name of CAST output explain select key from (select cast(key as int) from src )t; --backward explain select key2 from (select cast(key as int) key2 from src )t; [info] - alias_casted_column *** FAILED *** [info] Failed to execute query using catalyst: [info] Error: Unresolved attributes: 'key, tree: [info] Project ['key] [info]Subquery t [info] Project [CAST(key#630, IntegerType) AS c_0#636] [info] LowerCaseSchema [info] InMemoryRelation [key#630,value#631], false, (HiveTableScan [key#630,value#631], (MetastoreRelation default, src, None), None) [info] [info] org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved attributes: 'key, tree: [info] Project ['key] [info]Subquery t [info] Project [CAST(key#630, IntegerType) AS c_0#636] [info] LowerCaseSchema [info] InMemoryRelation [key#630,value#631], false, (HiveTableScan [key#630,value#631], (MetastoreRelation default, src, None), None) [info] ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-47498135 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16259/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-47498134 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/143#issuecomment-47498509 Thanks! I've merged this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-897: preemptively serialize closures
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/143 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2077 Log serializer that actually ends u...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1017#issuecomment-47498587 Merging this in master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2077 Log serializer that actually ends u...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1017 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1264#discussion_r14337870 --- Diff: core/src/main/scala/org/apache/spark/SparkException.scala --- @@ -22,3 +22,6 @@ class SparkException(message: String, cause: Throwable) def this(message: String) = this(message, null) } + +private[spark] class SparkDriverExecutionException(cause: Throwable) --- End diff -- Perhaps add a comment to clarify what this exception was thrown from and why it caused a job failure (i.e., exception occurred while running task finalization logic on the driver or something) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47500365 Besides comment, LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47500575 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47500571 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-47502435 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-47502430 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47503574 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47503575 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16260/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2185] Emit warning when task size excee...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1149#issuecomment-47504358 BTW in CoarseGrainedSchedulerBackend.launchTasks we also check for the task size. Can you remove that one? (That one is very annoying because it peeks into the internals of TaskSchedulerImpl --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-47505705 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP] SPARK-2157 Ability to write tight firewa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1107#issuecomment-47505706 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16261/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user adamosloizou commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-47508367 @vanzin The shutdown hook was my 1st attempt (it would also take care of Ctrl-C), but I eventually went for the current solution because the hook wouldn't exit gracefully. It bombs with a big exception: ``` uncaught exception during compilation: java.lang.AssertionError Exception in thread shutdownHook1 java.lang.AssertionError: assertion failed: Tried to find '$line9' in '/var/folders/yz/cmjy6vj1787fq0kb202qbms4gp/T/spark-a0256dbe-b4b0-4fa4-b718-b47e8392d84c' but it is not a directory at scala.reflect.io.AbstractFile.subdirectoryNamed(AbstractFile.scala:254) at scala.tools.nsc.backend.jvm.BytecodeWriters$class.getFile(BytecodeWriters.scala:31) at scala.tools.nsc.backend.jvm.BytecodeWriters$class.scala$tools$nsc$backend$jvm$BytecodeWriters$$getFile(BytecodeWriters.scala:37) at scala.tools.nsc.backend.jvm.BytecodeWriters$ClassBytecodeWriter$class.writeClass(BytecodeWriters.scala:89) at scala.tools.nsc.backend.jvm.GenASM$AsmPhase$$anon$4.writeClass(GenASM.scala:67) at scala.tools.nsc.backend.jvm.GenASM$JBuilder.writeIfNotTooBig(GenASM.scala:459) at scala.tools.nsc.backend.jvm.GenASM$JPlainBuilder.genClass(GenASM.scala:1413) at scala.tools.nsc.backend.jvm.GenASM$AsmPhase.run(GenASM.scala:120) at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1583) at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1557) at scala.tools.nsc.Global$Run.compileSources(Global.scala:1553) at org.apache.spark.repl.SparkIMain.compileSourcesKeepingRun(SparkIMain.scala:468) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.compileAndSaveRun(SparkIMain.scala:859) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.compile(SparkIMain.scala:815) at org.apache.spark.repl.SparkIMain$Request.compile$lzycompute(SparkIMain.scala:1009) at org.apache.spark.repl.SparkIMain$Request.compile(SparkIMain.scala:1004) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:644) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:801) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:846) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:758) at org.apache.spark.repl.SparkILoop$$anonfun$sparkCleanUp$1.apply(SparkILoop.scala:163) at org.apache.spark.repl.SparkILoop$$anonfun$sparkCleanUp$1.apply(SparkILoop.scala:163) at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:263) at org.apache.spark.repl.SparkILoop.sparkCleanUp(SparkILoop.scala:162) at org.apache.spark.repl.SparkILoop.closeInterpreter(SparkILoop.scala:169) at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$apply$mcZ$sp$7.apply$mcV$sp(SparkILoop.scala:942) at scala.sys.ShutdownHookThread$$anon$1.run(ShutdownHookThread.scala:33) ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/1266 [SPARK-2327] [SQL] Fix nullabilities of Join/Generate/Aggregate. Fix nullabilities of `Join`/`Generate`/`Aggregate` because: - Output attributes of opposite side of `OuterJoin` should be nullable. - Output attributes of generater side of `Generate` should be nullable if `join` is `true` and `outer` is `true`. - `AttributeReference` of `computedAggregates` of `Aggregate` should be the same as `aggregateExpression`'s. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-2327 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1266.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1266 commit f20f196e33e4bd6ab4adeb9bb4afea9b9256d902 Author: Takuya UESHIN ues...@happy-camper.st Date: 2014-06-30T08:43:37Z Fix Join output nullabilities. commit 09532ec6a67ba38b63deb6d0ea4607d7e8a07080 Author: Takuya UESHIN ues...@happy-camper.st Date: 2014-06-30T08:45:10Z Fix Generate output nullabilities. commit 0e31e377b4a8ba8c4bf698c2afea6a604a6c25bd Author: Takuya UESHIN ues...@happy-camper.st Date: 2014-06-30T08:48:52Z Fix Aggregate resultAttribute nullabilities. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-47512702 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-47512711 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-47512853 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16262/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-47512852 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
GitHub user CrazyJvm opened a pull request: https://github.com/apache/spark/pull/1267 make port of ConnectionManager configurable I encountered a port confliction problem due to ConnectionManager which is really annoying . So I make the ConnectionManager port configurable with default port still. I added a new configuration called `spark.network.connectionmanager.port` , and I'm not sure whether the name is OK or not : ) You can merge this pull request into a Git repository by running: $ git pull https://github.com/CrazyJvm/spark ConnectionManager-Port Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1267.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1267 commit e34fbe5003a97b693de7abb5144d907d01283fef Author: CrazyJvm crazy...@gmail.com Date: 2014-06-30T09:37:12Z make port of ConnectionManager configurable --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47513619 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47513604 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-47513607 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47513744 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47513746 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16263/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47514431 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47514421 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user adamosloizou commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-47514571 @aarondav great idea! Was not aware of the init commands. It works like a charm, thanks! Updated PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-47517256 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16264/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-47517255 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] SPARK-2311: Added additional GLMs (Poi...
Github user BaiGang commented on the pull request: https://github.com/apache/spark/pull/1237#issuecomment-47517562 Merging some of the features in https://github.com/apache/spark/pull/1243 to this PR via https://github.com/xwei-datageek/spark/pull/2. Please take a review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47517908 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47517909 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16265/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2328] [SQL] Add execution of `SHOW TABL...
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/1268 [SPARK-2328] [SQL] Add execution of `SHOW TABLES` before `TestHive.reset()`. `PruningSuite` is executed first of Hive tests unfortunately, `TestHive.reset()` breaks the test environment. To prevent this, we must run a query before calling reset the first time. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ueshin/apache-spark issues/SPARK-2328 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1268.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1268 commit 043ceac67cdd53cb784e83012fca7a91804dae10 Author: Takuya UESHIN ues...@happy-camper.st Date: 2014-06-30T10:36:18Z Add execution of `SHOW TABLES` before `TestHive.reset()`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2327] [SQL] Fix nullabilities of Join/G...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/1266#issuecomment-47518136 I guess this needs #1268 to pass Hive tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2328] [SQL] Add execution of `SHOW TABL...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1268#issuecomment-47518254 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2328] [SQL] Add execution of `SHOW TABL...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1268#issuecomment-47518247 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2328] [SQL] Add execution of `SHOW TABL...
Github user ueshin commented on the pull request: https://github.com/apache/spark/pull/1268#issuecomment-47518299 If #1226 is merged before this, this is not needed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-47520651 Hey !, This is okay. You might wanna do it for quit too ? My actual doubt was that why can't we add it here https://github.com/apache/spark/blob/master/repl/src/main/scala/org/apache/spark/repl/SparkILoop.scala#L286, Haven't tried it myself, but feels like the last argument can call closeInterpreter ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user adamosloizou commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-47525376 @ScrapCodes thanks for the feedback! I think the original issue was about offering a behaviour similar to the vanilla scala REPL. The `standardCommands` that you suggested are used with a `:` prepended such as `:quit`. The idea was to offer the scala REPL `exit` functionality (notice the lack of `:`) while also stopping the Spark context. I could add the quit too if you want but that is not part of the default scala REPL behavior: ``` scala quit console:8: error: not found: value quit quit ^ scala :quit [success] Total time: 170 s, completed 30-Jun-2014 14:25:50 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2328] [SQL] Add execution of `SHOW TABL...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1268#issuecomment-47525556 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16266/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2328] [SQL] Add execution of `SHOW TABL...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1268#issuecomment-4752 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2003] Fix python SparkContext example
Github user dianacarroll commented on the pull request: https://github.com/apache/spark/pull/1246#issuecomment-47537578 Why did you change the doc to recommend setting sc.master programmatically? AFAIK we are actively discouraging that, now that it can be set in the spark-submit script. I disagree that this is a doc bug. If SparkContext(sparkConf) works in Scala, it should also work in Python. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2199] [mllib] topic modeling
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1269#issuecomment-47538367 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2199] [mllib] topic modeling
GitHub user akopich opened a pull request: https://github.com/apache/spark/pull/1269 [SPARK-2199] [mllib] topic modeling I have implemented Probabilistic Latent Semantic Analysis (PLSA) and Robust PLSA with support of additive regularization (that actually means that I've implemented Latent Dirichlet Allocation too). You can merge this pull request into a Git repository by running: $ git pull https://github.com/akopich/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1269.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1269 commit 79dbb42e0edcf4194542faf9f409383a424f2ff4 Author: valerij acop...@gmail.com Date: 2014-06-27T17:44:00Z [SPARK-2199][mllib] commit 695c7b225450685c0d05438bb39ca2d998b688a0 Author: valerij acop...@gmail.com Date: 2014-06-30T13:29:35Z TopicModeling interface is changed commit 66b67e555cfde24decb1637d2a73b4b3dc7a61ce Author: valerij acop...@gmail.com Date: 2014-06-30T14:10:33Z [SPARK-2199][mllib] docs added commit c75a5eed4b23c852e3f1a6ce2819a4d0d8ee3ae8 Author: valerij acop...@gmail.com Date: 2014-06-30T14:12:29Z [SPARK-2199][mllib] document is a sparse verctor of Ints, not shorts commit 88b8da82ad4ecbfe783b32b7c28117b26283f824 Author: valerij acop...@gmail.com Date: 2014-06-30T14:14:54Z [SPARK-2199][mllib] style fixed commit 1c4288500933405020558f8928ce96d4aa459672 Author: valerij acop...@gmail.com Date: 2014-06-30T14:15:35Z [SPARK-2199][mllib] backgound is explicitly defined as val commit 112c65c149c24ed84af3ac3e98c47bd3a984e26e Author: valerij acop...@gmail.com Date: 2014-06-30T14:23:15Z [SPARK-2199][mllib] Short is replaced with int in Enumerator.scala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user cmccabe commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47543722 Looks good. +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] SPARK-2329 Add multi-label evaluation ...
GitHub user avulanov opened a pull request: https://github.com/apache/spark/pull/1270 [MLLIB] SPARK-2329 Add multi-label evaluation metrics Implementation of various multi-label classification measures, including: Hamming-loss, strict and default Accuracy, macro-averaged Precision, Recall and F1-measure based on documents and labels, micro-averaged measures: https://issues.apache.org/jira/browse/SPARK-2329 This request relates to multi-class measures pull request: https://github.com/apache/spark/pull/1155 You can merge this pull request into a Git repository by running: $ git pull https://github.com/avulanov/spark multilabelmetrics Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1270.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1270 commit 154164bc3aef49158451e169c3e7ccc13d267e09 Author: Alexander Ulanov na...@yandex.ru Date: 2014-06-30T10:33:25Z Multilabel evaluation metics and tests: macro precision and recall averaged by docs, micro and per-class precision and recall averaged by class commit ad62df02fe47042d54a7a3d2347262e88bba01f0 Author: Alexander Ulanov na...@yandex.ru Date: 2014-06-30T12:38:15Z Comments and scala style check commit 40593f5ac743feaa691e5ccdc0e6410d8dfa40d1 Author: Alexander Ulanov na...@yandex.ru Date: 2014-06-30T15:28:56Z Multi-label metrics: Hamming-loss, strict and normal accuracy, fix to macro measures, bunch of tests commit ca467652011b517b2c5f63d287e221964e27ded2 Author: Alexander Ulanov na...@yandex.ru Date: 2014-06-30T15:38:26Z Cosmetic changes: Apache header and parameter explanation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB] SPARK-2329 Add multi-label evaluation ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1270#issuecomment-47548849 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: make port of ConnectionManager configurable
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1267#issuecomment-47549589 can you please file a jira for this? What is your use case here when you ran into the issue, just in running the tests? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2304] tera sort example program for shu...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1242#issuecomment-47550102 The hadoop code for generating the data is out of date. It might not matter for your purposes, but if you want the up to date one look at sortbenchmark.org. I had filed jira to update Hadoop one but haven't gotten to it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP]SPARK-2098: All Spark processes should su...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1256#issuecomment-47550128 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP]SPARK-2098: All Spark processes should su...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1256#issuecomment-47550144 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47550403 Nice, this will be very helpful. Probably could also be added to yarn's ApplicationMaster.scala (both of them)? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP]SPARK-2098: All Spark processes should su...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1256#issuecomment-47555372 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16267/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [WIP]SPARK-2098: All Spark processes should su...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1256#issuecomment-47555371 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1264#issuecomment-47570582 Ok I'm merging this in mater branch-1.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-47570546 Ahh I think I understand your intention now. We can have this but there is one thing I had like to add is this exit functionality is deprecated and is not available for scala 2.11. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47570667 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47570648 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2322] Exception in resultHandler should...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1264 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47572047 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16268/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47572120 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47572632 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47572648 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47572045 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47573406 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16269/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47573404 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2234][SQL]Spark SQL basicOperators add ...
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/1151#issuecomment-47573444 You'll need to merge with master and resolve the conflicts. Something like: ``` git remote update git merge origin/master git add fixed files git commit ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47578645 LGTM, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-47579685 So you're saying there's no way out of the scala 2.11 repl? That's hardcore. On Mon, Jun 30, 2014 at 11:49 AM, Prashant Sharma notificati...@github.com wrote: Ahh I think I understand your intention now. We can have this but there is one thing I had like to add is this exit functionality is deprecated and is not available for scala 2.11. â Reply to this email directly or view it on GitHub https://github.com/apache/spark/pull/1230#issuecomment-47570546. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2332 [build] add exclusion for old servl...
GitHub user pdmack opened a pull request: https://github.com/apache/spark/pull/1271 SPARK-2332 [build] add exclusion for old servlet-api on hadoop-client in core You can merge this pull request into a Git repository by running: $ git pull https://github.com/pdmack/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1271.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1271 commit 6f39fecfe8e506f73374412c6999a7641b80df36 Author: Peter MacKinnon pmack...@redhat.com Date: 2014-06-30T19:49:42Z add exclusion for old servlet-api on hadoop-client in core --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2332 [build] add exclusion for old servl...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1271#issuecomment-47579967 I've been wrestling with this dependency for a while. However I thought we specifically wanted to retain servlet-api 3.0. What problem are you seeing? It's possible there's a different answer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2332 [build] add exclusion for old servl...
Github user pdmack commented on the pull request: https://github.com/apache/spark/pull/1271#issuecomment-47580217 Well, the build seems generally set up for jetty 8 which makes use of servlet-api 3.0.x. So I'm assuming we don't want the old 2.5 one since there are numerous explicit excludes littered through the poms for that version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47580286 Can we have HistoryServer too? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2159: Add support for stopping SparkCont...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/1230#issuecomment-47580323 I think he means `exit` is gone; you still have `:quit` (or so I hope). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2332 [build] add exclusion for old servl...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1271#issuecomment-47580342 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2332 [build] add exclusion for old servl...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1271#issuecomment-47580459 Yeah, saw your detail on the JIRA. Yes 2.5 should not be in there, I'm just surprised it is. The concern is knocking out 3.0 accidentally. Have you checked that is all OK still? I'll look too just because I've been staring at this particular dependency hell for a while now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1260#discussion_r14372270
--- Diff: core/src/main/scala/org/apache/spark/util/SignalLogger.scala ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import org.apache.commons.lang.SystemUtils
+import org.slf4j.Logger
+import sun.misc.{Signal, SignalHandler}
+
+/**
+ * Used to log signals received. This can be very useful in debugging
crashes or kills.
+ *
+ * Inspired by Colin Patrick McCabe's similar class from Hadoop.
+ */
+private[spark] object SignalLogger {
+
+ private var registered = false
+
+ /** Register a signal handler to log signals on UNIX-like systems. */
+ def register(log: Logger): Unit = if (SystemUtils.IS_OS_UNIX) {
--- End diff --
Could you indulge me by wrapping this in a synchronized? Also, that if
seems a bit too important / unexpected to keep hidden on the same line, so this
will help that too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47580697 LGTM, just two very minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2332 [build] add exclusion for old servl...
Github user pdmack commented on the pull request: https://github.com/apache/spark/pull/1271#issuecomment-47580937 mvn -X -Dhadoop.version=2.0.0-mr1-cdh4.1.2 -Dlabel=centos test I didn't see any jetty test failures and certainly none of the previous FilterRegistration security exceptions. Also 3.0 remains in tact: DEBUG] Invalidating /home/pmackinn/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar: class javax.servlet.http.HttpServletRequest now provided by /home/pmackinn/.m2/repository/org/eclipse/jetty/orbit/javax.servlet/3.0.0.v201112011016/javax.servlet-3.0.0.v201112011016.jar --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1260#discussion_r14375000
--- Diff: core/src/main/scala/org/apache/spark/util/SignalLogger.scala ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the License); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.util
+
+import org.apache.commons.lang.SystemUtils
+import org.slf4j.Logger
+import sun.misc.{Signal, SignalHandler}
+
+/**
+ * Used to log signals received. This can be very useful in debugging
crashes or kills.
+ *
+ * Inspired by Colin Patrick McCabe's similar class from Hadoop.
+ */
+private[spark] object SignalLogger {
+
+ private var registered = false
+
+ /** Register a signal handler to log signals on UNIX-like systems. */
+ def register(log: Logger): Unit = if (SystemUtils.IS_OS_UNIX) {
--- End diff --
done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47585835 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47585850 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47586502 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16270/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47586501 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47587402 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47587612 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47587596 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-2332 [build] add exclusion for old servl...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1271#issuecomment-47589343 Hey @pdmack as far as I can see, you're correct. javax.servlet:servlet-api is not currently a dependency for most (?) Hadoop versions. It seems to come in with versions of Hadoop 2.0.x via hadoop-hdfs. Excluding it should be fine since it is not desirable where it is brought in, and not brought in elsewhere. Just check that it all works, that the web UI starts. I have managed to accidentally exclude all copies of the servlet API accidentally before! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: Resolve sbt warnings during build � �
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/1153#issuecomment-47591417 This would be nice to get in before 1.0.1 just to clean up those ugly compiler warnings. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47592441 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16271/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47592440 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2318] When exiting on a signal, print t...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1260#issuecomment-47594260 I'm merging this in master. Thanks for everybody for taking a look! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
