[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145411566
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145411577
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145412905
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145412898
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145413368
  
  [Test build #43230 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43230/console)
 for   PR 8975 at commit 
[`88c9c3e`](https://github.com/apache/spark/commit/88c9c3ef407cecbe46ced9411d1d14ff70752d65).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class LogicalRelation(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145413370
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43230/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145413369
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145418831
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145418844
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9034] [SQL] Reflect field names defined...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8456#issuecomment-145419758
  
  [Test build #43227 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43227/console)
 for   PR 8456 at commit 
[`0382b3f`](https://github.com/apache/spark/commit/0382b3fd93b561c8ecd74ad0ffdd7ccc37139e2f).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9034] [SQL] Reflect field names defined...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8456#issuecomment-145419798
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43227/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9034] [SQL] Reflect field names defined...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8456#issuecomment-145419797
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8354#issuecomment-145420107
  
  [Test build #43228 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43228/console)
 for   PR 8354 at commit 
[`fd799a1`](https://github.com/apache/spark/commit/fd799a1163a47c2819ab8d91d2131968ab45871f).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8354#issuecomment-145420157
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43228/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8354#issuecomment-145420156
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8563#discussion_r41108645
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
 ---
@@ -402,4 +445,402 @@ class BlockMatrix @Since("1.3.0") (
 s"A.colsPerBlock: $colsPerBlock, B.rowsPerBlock: 
${other.rowsPerBlock}")
 }
   }
+
+  /** Schur Complement of a BlockMatrix.  For a matrix that is in 4 
partitions:
+*  A=[a11, a12; a21; a22], the Schur Complement S is S = a22 - (a21 * 
a11^-1 * a12).
+* The Schur Complement is always (n-1) x (n-1), which is the size of 
a22.
+*
+* @return BlockMatrix Schur Complement as BlockMatrix
+* @since 1.6.0
+  */
+private[mllib] def SchurComplement: BlockMatrix = {
+ require(this.numRowBlocks == this.numColBlocks, "Block Matrix must be 
square.")
+ require(this.numRowBlocks > 1, "Block Matrix must be larger than one 
block.")
+ val topRange = (0, 0); val botRange = (1, this.numColBlocks - 1)
+ val a11 = this.subBlock(topRange, topRange)
+ val a12 = this.subBlock(topRange, botRange)
+ val a21 = this.subBlock(botRange, topRange)
+ val a22 = this.subBlock(botRange, botRange)
+
+val a11Brz = inv(a11.toBreeze) // note that intermediate a11 calcs 
derive from inv(a11)
+val a11Mtx = Matrices.dense(a11.numRows.toInt, a11.numCols.toInt, 
a11Brz.toArray)
+val a11RDD = this.blocks.sparkContext.parallelize(Seq(((0, 0), 
a11Mtx)))
+val a11Inv = new BlockMatrix(a11RDD, this.rowsPerBlock, 
this.colsPerBlock)
+
+val S = a22.subtract(a21.multiply(a11Inv.multiply(a12)))
+return S
+  }
+
+  /** Returns a rectangular (sub)BlockMatrix with block ranges as 
specified.
+*
+* @param blockRowRange The lower and upper row ranges, as (Int,Int)
+* @param blockColRange The lower and upper col ranges, as (Int, Int)
+* @return a BlockMatrix with (0,0) as the upper leftmost block index
+* @since 1.6.0
+*/
+
+  private [mllib] def subBlock(blockRowRange: (Int, Int), blockColRange: 
(Int, Int)):
+  BlockMatrix = {
+//  Extracts BlockMatrix elements from a specified range of block 
indices
+//  Creating a Sub BlockMatrix of rectangular shape.
+//  Also reindexes so that the upper left block is always (0, 0)
+
+// JNDB: Add a require statement ...rowMax<=size..
+val rowMin = blockRowRange._1;val rowMax = blockRowRange._2
+val colMin = blockColRange._1 ;   val colMax = blockColRange._2
+val extractedSeq = this.blocks.filter{ case((x, y), matrix) =>
+  x >= rowMin && x<= rowMax && // finding blocks
+y >= colMin && y<= colMax }.map{   // shifting indices
+  case(((x, y), matrix) ) => ((x-rowMin, y-colMin), matrix)
+}
+return new BlockMatrix(extractedSeq, rowsPerBlock, colsPerBlock)
+  }
+
+  /** computes the LU decomposition of a Single Block from BlockMatrix 
using the
+* Breeze LU method.  The method (as written) operates -only- on the 
upper
+* left (0,0) corner of the BlockMatrix.
+*
+* @return List[BDM[Double]] of Breeze Matrices (BDM) (P,L,U) for 
blockLU method.
+* @since 1.6.0
+  */
+  private [mllib] def singleBlockPLU: List[BDM[Double]] = {
+// returns PA = LU factorization from Breeze
+val PLU = LU(this.subBlock((0, 0), (0, 0)).toBreeze)
+val k = PLU._1.cols
+val L = lowerTriangular(PLU._1) - diag(diag(PLU._1)) + 
diag(DenseVector.fill(k){1.0})
+val U = upperTriangular(PLU._1);
+var P = diag(DenseVector.fill(k){1.0})
+val Pi = diag(DenseVector.fill(k){1.0})
+// size of square matrix
+for(i <- 0 to (k - 1)) { // i test populating permutation matrix
+  val I = i match {case 0 => k - 1 case _ => i - 1}
+  val J = PLU._2(i) -1
+  if (i != J) {  Pi(i, J) += 1.0; Pi(J, i) += 1.0; Pi(i, i) -= 1.0; 
Pi(J, J) -= 1.0}
+  P = Pi * P  // constructor Pi*P for PA=LU
+  if (i != J) { Pi(i, J) -= 1.0; Pi(J, i) -= 1.0; Pi(i, i) += 1.0; 
Pi(J, J) += 1.0}
+}
+return List(P, L, U)
+  }
+
+
+  /** This method reassigns 'absolute' index locations (i,j), to 
sequences.  This is
+* designed to reconsitute the orignal block locations that were lost 
in the
+* subBlock method.
+*
+* @param rowMin The new lowest row value
+* @param colMin The new lowest column value
+* @return an RDD of Sequences with new block indexing
+* @since 1.6.0
+*
+*/
+  private [mllib] def shiftIndices(rowMin: Int, colMin: Int): RDD[((Int, 
Int), Matrix)] = {
+// This routine recovers the absolute indexing of the block matrices 
for 

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145420520
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145420516
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8563#discussion_r41108858
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
 ---
@@ -402,4 +445,402 @@ class BlockMatrix @Since("1.3.0") (
 s"A.colsPerBlock: $colsPerBlock, B.rowsPerBlock: 
${other.rowsPerBlock}")
 }
   }
+
+  /** Schur Complement of a BlockMatrix.  For a matrix that is in 4 
partitions:
+*  A=[a11, a12; a21; a22], the Schur Complement S is S = a22 - (a21 * 
a11^-1 * a12).
+* The Schur Complement is always (n-1) x (n-1), which is the size of 
a22.
+*
+* @return BlockMatrix Schur Complement as BlockMatrix
+* @since 1.6.0
+  */
+private[mllib] def SchurComplement: BlockMatrix = {
+ require(this.numRowBlocks == this.numColBlocks, "Block Matrix must be 
square.")
+ require(this.numRowBlocks > 1, "Block Matrix must be larger than one 
block.")
+ val topRange = (0, 0); val botRange = (1, this.numColBlocks - 1)
+ val a11 = this.subBlock(topRange, topRange)
+ val a12 = this.subBlock(topRange, botRange)
+ val a21 = this.subBlock(botRange, topRange)
+ val a22 = this.subBlock(botRange, botRange)
+
+val a11Brz = inv(a11.toBreeze) // note that intermediate a11 calcs 
derive from inv(a11)
+val a11Mtx = Matrices.dense(a11.numRows.toInt, a11.numCols.toInt, 
a11Brz.toArray)
+val a11RDD = this.blocks.sparkContext.parallelize(Seq(((0, 0), 
a11Mtx)))
+val a11Inv = new BlockMatrix(a11RDD, this.rowsPerBlock, 
this.colsPerBlock)
+
+val S = a22.subtract(a21.multiply(a11Inv.multiply(a12)))
+return S
+  }
+
+  /** Returns a rectangular (sub)BlockMatrix with block ranges as 
specified.
+*
+* @param blockRowRange The lower and upper row ranges, as (Int,Int)
+* @param blockColRange The lower and upper col ranges, as (Int, Int)
+* @return a BlockMatrix with (0,0) as the upper leftmost block index
+* @since 1.6.0
+*/
+
+  private [mllib] def subBlock(blockRowRange: (Int, Int), blockColRange: 
(Int, Int)):
+  BlockMatrix = {
+//  Extracts BlockMatrix elements from a specified range of block 
indices
+//  Creating a Sub BlockMatrix of rectangular shape.
+//  Also reindexes so that the upper left block is always (0, 0)
+
+// JNDB: Add a require statement ...rowMax<=size..
+val rowMin = blockRowRange._1;val rowMax = blockRowRange._2
+val colMin = blockColRange._1 ;   val colMax = blockColRange._2
+val extractedSeq = this.blocks.filter{ case((x, y), matrix) =>
+  x >= rowMin && x<= rowMax && // finding blocks
+y >= colMin && y<= colMax }.map{   // shifting indices
+  case(((x, y), matrix) ) => ((x-rowMin, y-colMin), matrix)
+}
+return new BlockMatrix(extractedSeq, rowsPerBlock, colsPerBlock)
+  }
+
+  /** computes the LU decomposition of a Single Block from BlockMatrix 
using the
+* Breeze LU method.  The method (as written) operates -only- on the 
upper
+* left (0,0) corner of the BlockMatrix.
+*
+* @return List[BDM[Double]] of Breeze Matrices (BDM) (P,L,U) for 
blockLU method.
+* @since 1.6.0
+  */
+  private [mllib] def singleBlockPLU: List[BDM[Double]] = {
+// returns PA = LU factorization from Breeze
+val PLU = LU(this.subBlock((0, 0), (0, 0)).toBreeze)
+val k = PLU._1.cols
+val L = lowerTriangular(PLU._1) - diag(diag(PLU._1)) + 
diag(DenseVector.fill(k){1.0})
+val U = upperTriangular(PLU._1);
+var P = diag(DenseVector.fill(k){1.0})
+val Pi = diag(DenseVector.fill(k){1.0})
+// size of square matrix
+for(i <- 0 to (k - 1)) { // i test populating permutation matrix
+  val I = i match {case 0 => k - 1 case _ => i - 1}
+  val J = PLU._2(i) -1
+  if (i != J) {  Pi(i, J) += 1.0; Pi(J, i) += 1.0; Pi(i, i) -= 1.0; 
Pi(J, J) -= 1.0}
+  P = Pi * P  // constructor Pi*P for PA=LU
+  if (i != J) { Pi(i, J) -= 1.0; Pi(J, i) -= 1.0; Pi(i, i) += 1.0; 
Pi(J, J) += 1.0}
+}
+return List(P, L, U)
+  }
+
+
+  /** This method reassigns 'absolute' index locations (i,j), to 
sequences.  This is
+* designed to reconsitute the orignal block locations that were lost 
in the
+* subBlock method.
+*
+* @param rowMin The new lowest row value
+* @param colMin The new lowest column value
+* @return an RDD of Sequences with new block indexing
+* @since 1.6.0
+*
+*/
+  private [mllib] def shiftIndices(rowMin: Int, colMin: Int): RDD[((Int, 
Int), Matrix)] = {
+// This routine recovers the absolute indexing of the block matrices 
for 

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8563#discussion_r41109096
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
 ---
@@ -402,4 +445,402 @@ class BlockMatrix @Since("1.3.0") (
 s"A.colsPerBlock: $colsPerBlock, B.rowsPerBlock: 
${other.rowsPerBlock}")
 }
   }
+
+  /** Schur Complement of a BlockMatrix.  For a matrix that is in 4 
partitions:
+*  A=[a11, a12; a21; a22], the Schur Complement S is S = a22 - (a21 * 
a11^-1 * a12).
+* The Schur Complement is always (n-1) x (n-1), which is the size of 
a22.
+*
+* @return BlockMatrix Schur Complement as BlockMatrix
+* @since 1.6.0
+  */
+private[mllib] def SchurComplement: BlockMatrix = {
+ require(this.numRowBlocks == this.numColBlocks, "Block Matrix must be 
square.")
+ require(this.numRowBlocks > 1, "Block Matrix must be larger than one 
block.")
+ val topRange = (0, 0); val botRange = (1, this.numColBlocks - 1)
+ val a11 = this.subBlock(topRange, topRange)
+ val a12 = this.subBlock(topRange, botRange)
+ val a21 = this.subBlock(botRange, topRange)
+ val a22 = this.subBlock(botRange, botRange)
+
+val a11Brz = inv(a11.toBreeze) // note that intermediate a11 calcs 
derive from inv(a11)
+val a11Mtx = Matrices.dense(a11.numRows.toInt, a11.numCols.toInt, 
a11Brz.toArray)
+val a11RDD = this.blocks.sparkContext.parallelize(Seq(((0, 0), 
a11Mtx)))
+val a11Inv = new BlockMatrix(a11RDD, this.rowsPerBlock, 
this.colsPerBlock)
+
+val S = a22.subtract(a21.multiply(a11Inv.multiply(a12)))
+return S
+  }
+
+  /** Returns a rectangular (sub)BlockMatrix with block ranges as 
specified.
+*
+* @param blockRowRange The lower and upper row ranges, as (Int,Int)
+* @param blockColRange The lower and upper col ranges, as (Int, Int)
+* @return a BlockMatrix with (0,0) as the upper leftmost block index
+* @since 1.6.0
+*/
+
+  private [mllib] def subBlock(blockRowRange: (Int, Int), blockColRange: 
(Int, Int)):
+  BlockMatrix = {
+//  Extracts BlockMatrix elements from a specified range of block 
indices
+//  Creating a Sub BlockMatrix of rectangular shape.
+//  Also reindexes so that the upper left block is always (0, 0)
+
+// JNDB: Add a require statement ...rowMax<=size..
+val rowMin = blockRowRange._1;val rowMax = blockRowRange._2
+val colMin = blockColRange._1 ;   val colMax = blockColRange._2
+val extractedSeq = this.blocks.filter{ case((x, y), matrix) =>
+  x >= rowMin && x<= rowMax && // finding blocks
+y >= colMin && y<= colMax }.map{   // shifting indices
+  case(((x, y), matrix) ) => ((x-rowMin, y-colMin), matrix)
+}
+return new BlockMatrix(extractedSeq, rowsPerBlock, colsPerBlock)
+  }
+
+  /** computes the LU decomposition of a Single Block from BlockMatrix 
using the
+* Breeze LU method.  The method (as written) operates -only- on the 
upper
+* left (0,0) corner of the BlockMatrix.
+*
+* @return List[BDM[Double]] of Breeze Matrices (BDM) (P,L,U) for 
blockLU method.
+* @since 1.6.0
+  */
+  private [mllib] def singleBlockPLU: List[BDM[Double]] = {
+// returns PA = LU factorization from Breeze
+val PLU = LU(this.subBlock((0, 0), (0, 0)).toBreeze)
+val k = PLU._1.cols
+val L = lowerTriangular(PLU._1) - diag(diag(PLU._1)) + 
diag(DenseVector.fill(k){1.0})
+val U = upperTriangular(PLU._1);
+var P = diag(DenseVector.fill(k){1.0})
+val Pi = diag(DenseVector.fill(k){1.0})
+// size of square matrix
+for(i <- 0 to (k - 1)) { // i test populating permutation matrix
+  val I = i match {case 0 => k - 1 case _ => i - 1}
+  val J = PLU._2(i) -1
+  if (i != J) {  Pi(i, J) += 1.0; Pi(J, i) += 1.0; Pi(i, i) -= 1.0; 
Pi(J, J) -= 1.0}
+  P = Pi * P  // constructor Pi*P for PA=LU
+  if (i != J) { Pi(i, J) -= 1.0; Pi(J, i) -= 1.0; Pi(i, i) += 1.0; 
Pi(J, J) += 1.0}
+}
+return List(P, L, U)
+  }
+
+
+  /** This method reassigns 'absolute' index locations (i,j), to 
sequences.  This is
+* designed to reconsitute the orignal block locations that were lost 
in the
+* subBlock method.
+*
+* @param rowMin The new lowest row value
+* @param colMin The new lowest column value
+* @return an RDD of Sequences with new block indexing
+* @since 1.6.0
+*
+*/
+  private [mllib] def shiftIndices(rowMin: Int, colMin: Int): RDD[((Int, 
Int), Matrix)] = {
+// This routine recovers the absolute indexing of the block matrices 
for 

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8563#discussion_r41109078
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
 ---
@@ -402,4 +445,402 @@ class BlockMatrix @Since("1.3.0") (
 s"A.colsPerBlock: $colsPerBlock, B.rowsPerBlock: 
${other.rowsPerBlock}")
 }
   }
+
+  /** Schur Complement of a BlockMatrix.  For a matrix that is in 4 
partitions:
+*  A=[a11, a12; a21; a22], the Schur Complement S is S = a22 - (a21 * 
a11^-1 * a12).
+* The Schur Complement is always (n-1) x (n-1), which is the size of 
a22.
+*
+* @return BlockMatrix Schur Complement as BlockMatrix
+* @since 1.6.0
+  */
+private[mllib] def SchurComplement: BlockMatrix = {
+ require(this.numRowBlocks == this.numColBlocks, "Block Matrix must be 
square.")
+ require(this.numRowBlocks > 1, "Block Matrix must be larger than one 
block.")
+ val topRange = (0, 0); val botRange = (1, this.numColBlocks - 1)
+ val a11 = this.subBlock(topRange, topRange)
+ val a12 = this.subBlock(topRange, botRange)
+ val a21 = this.subBlock(botRange, topRange)
+ val a22 = this.subBlock(botRange, botRange)
+
+val a11Brz = inv(a11.toBreeze) // note that intermediate a11 calcs 
derive from inv(a11)
+val a11Mtx = Matrices.dense(a11.numRows.toInt, a11.numCols.toInt, 
a11Brz.toArray)
+val a11RDD = this.blocks.sparkContext.parallelize(Seq(((0, 0), 
a11Mtx)))
+val a11Inv = new BlockMatrix(a11RDD, this.rowsPerBlock, 
this.colsPerBlock)
+
+val S = a22.subtract(a21.multiply(a11Inv.multiply(a12)))
+return S
+  }
+
+  /** Returns a rectangular (sub)BlockMatrix with block ranges as 
specified.
+*
+* @param blockRowRange The lower and upper row ranges, as (Int,Int)
+* @param blockColRange The lower and upper col ranges, as (Int, Int)
+* @return a BlockMatrix with (0,0) as the upper leftmost block index
+* @since 1.6.0
+*/
+
+  private [mllib] def subBlock(blockRowRange: (Int, Int), blockColRange: 
(Int, Int)):
+  BlockMatrix = {
+//  Extracts BlockMatrix elements from a specified range of block 
indices
+//  Creating a Sub BlockMatrix of rectangular shape.
+//  Also reindexes so that the upper left block is always (0, 0)
+
+// JNDB: Add a require statement ...rowMax<=size..
+val rowMin = blockRowRange._1;val rowMax = blockRowRange._2
+val colMin = blockColRange._1 ;   val colMax = blockColRange._2
+val extractedSeq = this.blocks.filter{ case((x, y), matrix) =>
+  x >= rowMin && x<= rowMax && // finding blocks
+y >= colMin && y<= colMax }.map{   // shifting indices
+  case(((x, y), matrix) ) => ((x-rowMin, y-colMin), matrix)
+}
+return new BlockMatrix(extractedSeq, rowsPerBlock, colsPerBlock)
+  }
+
+  /** computes the LU decomposition of a Single Block from BlockMatrix 
using the
+* Breeze LU method.  The method (as written) operates -only- on the 
upper
+* left (0,0) corner of the BlockMatrix.
+*
+* @return List[BDM[Double]] of Breeze Matrices (BDM) (P,L,U) for 
blockLU method.
+* @since 1.6.0
+  */
+  private [mllib] def singleBlockPLU: List[BDM[Double]] = {
+// returns PA = LU factorization from Breeze
+val PLU = LU(this.subBlock((0, 0), (0, 0)).toBreeze)
+val k = PLU._1.cols
+val L = lowerTriangular(PLU._1) - diag(diag(PLU._1)) + 
diag(DenseVector.fill(k){1.0})
+val U = upperTriangular(PLU._1);
+var P = diag(DenseVector.fill(k){1.0})
+val Pi = diag(DenseVector.fill(k){1.0})
+// size of square matrix
+for(i <- 0 to (k - 1)) { // i test populating permutation matrix
+  val I = i match {case 0 => k - 1 case _ => i - 1}
+  val J = PLU._2(i) -1
+  if (i != J) {  Pi(i, J) += 1.0; Pi(J, i) += 1.0; Pi(i, i) -= 1.0; 
Pi(J, J) -= 1.0}
+  P = Pi * P  // constructor Pi*P for PA=LU
+  if (i != J) { Pi(i, J) -= 1.0; Pi(J, i) -= 1.0; Pi(i, i) += 1.0; 
Pi(J, J) += 1.0}
+}
+return List(P, L, U)
+  }
+
+
+  /** This method reassigns 'absolute' index locations (i,j), to 
sequences.  This is
+* designed to reconsitute the orignal block locations that were lost 
in the
+* subBlock method.
+*
+* @param rowMin The new lowest row value
+* @param colMin The new lowest column value
+* @return an RDD of Sequences with new block indexing
+* @since 1.6.0
+*
+*/
+  private [mllib] def shiftIndices(rowMin: Int, colMin: Int): RDD[((Int, 
Int), Matrix)] = {
+// This routine recovers the absolute indexing of the block matrices 
for 

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8563#discussion_r41109118
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
 ---
@@ -402,4 +445,402 @@ class BlockMatrix @Since("1.3.0") (
 s"A.colsPerBlock: $colsPerBlock, B.rowsPerBlock: 
${other.rowsPerBlock}")
 }
   }
+
+  /** Schur Complement of a BlockMatrix.  For a matrix that is in 4 
partitions:
+*  A=[a11, a12; a21; a22], the Schur Complement S is S = a22 - (a21 * 
a11^-1 * a12).
+* The Schur Complement is always (n-1) x (n-1), which is the size of 
a22.
+*
+* @return BlockMatrix Schur Complement as BlockMatrix
+* @since 1.6.0
+  */
+private[mllib] def SchurComplement: BlockMatrix = {
+ require(this.numRowBlocks == this.numColBlocks, "Block Matrix must be 
square.")
+ require(this.numRowBlocks > 1, "Block Matrix must be larger than one 
block.")
+ val topRange = (0, 0); val botRange = (1, this.numColBlocks - 1)
+ val a11 = this.subBlock(topRange, topRange)
+ val a12 = this.subBlock(topRange, botRange)
+ val a21 = this.subBlock(botRange, topRange)
+ val a22 = this.subBlock(botRange, botRange)
+
+val a11Brz = inv(a11.toBreeze) // note that intermediate a11 calcs 
derive from inv(a11)
+val a11Mtx = Matrices.dense(a11.numRows.toInt, a11.numCols.toInt, 
a11Brz.toArray)
+val a11RDD = this.blocks.sparkContext.parallelize(Seq(((0, 0), 
a11Mtx)))
+val a11Inv = new BlockMatrix(a11RDD, this.rowsPerBlock, 
this.colsPerBlock)
+
+val S = a22.subtract(a21.multiply(a11Inv.multiply(a12)))
+return S
+  }
+
+  /** Returns a rectangular (sub)BlockMatrix with block ranges as 
specified.
+*
+* @param blockRowRange The lower and upper row ranges, as (Int,Int)
+* @param blockColRange The lower and upper col ranges, as (Int, Int)
+* @return a BlockMatrix with (0,0) as the upper leftmost block index
+* @since 1.6.0
+*/
+
+  private [mllib] def subBlock(blockRowRange: (Int, Int), blockColRange: 
(Int, Int)):
+  BlockMatrix = {
+//  Extracts BlockMatrix elements from a specified range of block 
indices
+//  Creating a Sub BlockMatrix of rectangular shape.
+//  Also reindexes so that the upper left block is always (0, 0)
+
+// JNDB: Add a require statement ...rowMax<=size..
+val rowMin = blockRowRange._1;val rowMax = blockRowRange._2
+val colMin = blockColRange._1 ;   val colMax = blockColRange._2
+val extractedSeq = this.blocks.filter{ case((x, y), matrix) =>
+  x >= rowMin && x<= rowMax && // finding blocks
+y >= colMin && y<= colMax }.map{   // shifting indices
+  case(((x, y), matrix) ) => ((x-rowMin, y-colMin), matrix)
+}
+return new BlockMatrix(extractedSeq, rowsPerBlock, colsPerBlock)
+  }
+
+  /** computes the LU decomposition of a Single Block from BlockMatrix 
using the
+* Breeze LU method.  The method (as written) operates -only- on the 
upper
+* left (0,0) corner of the BlockMatrix.
+*
+* @return List[BDM[Double]] of Breeze Matrices (BDM) (P,L,U) for 
blockLU method.
+* @since 1.6.0
+  */
+  private [mllib] def singleBlockPLU: List[BDM[Double]] = {
+// returns PA = LU factorization from Breeze
+val PLU = LU(this.subBlock((0, 0), (0, 0)).toBreeze)
+val k = PLU._1.cols
+val L = lowerTriangular(PLU._1) - diag(diag(PLU._1)) + 
diag(DenseVector.fill(k){1.0})
+val U = upperTriangular(PLU._1);
+var P = diag(DenseVector.fill(k){1.0})
+val Pi = diag(DenseVector.fill(k){1.0})
+// size of square matrix
+for(i <- 0 to (k - 1)) { // i test populating permutation matrix
--- End diff --

Forloop is slow. Please use while loop here. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145423163
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145423171
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145423172
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145423162
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145424739
  
  [Test build #43233 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43233/consoleFull)
 for   PR 8920 at commit 
[`61e7646`](https://github.com/apache/spark/commit/61e76468d0a79649e2fd7b49d378071ed4265764).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145425285
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145425290
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145426462
  
  [Test build #43235 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43235/consoleFull)
 for   PR 8975 at commit 
[`18cbf73`](https://github.com/apache/spark/commit/18cbf73b81023e8adac6326e3225799291cbf8b5).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145428451
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145428455
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43232/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145412116
  
  [Test build #43229 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43229/consoleFull)
 for   PR 8899 at commit 
[`b50b86e`](https://github.com/apache/spark/commit/b50b86e655794aeb3025930aadb544b187fb17de).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[shenh062326]

GitHub user shenh062326 opened a pull request:

https://github.com/apache/spark/pull/8975

[SPARK-10918] [CORE] Prevent task failed for executor kill by driver

  When dynamicAllocation is enabled, when a executor was idle timeout, it 
will be kill by driver, if a task offer to the executor at the same time, the 
task will failed due to executor lost.
  

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shenh062326/spark my_change20151005

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8975.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8975


commit 88c9c3ef407cecbe46ced9411d1d14ff70752d65
Author: hongshen 
Date:   2015-10-05T02:12:50Z

Prevent task failed for executor kill by driver




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145413159
  
  [Test build #43230 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43230/consoleFull)
 for   PR 8975 at commit 
[`88c9c3e`](https://github.com/apache/spark/commit/88c9c3ef407cecbe46ced9411d1d14ff70752d65).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145413578
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43229/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145413544
  
  [Test build #43229 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43229/console)
 for   PR 8899 at commit 
[`b50b86e`](https://github.com/apache/spark/commit/b50b86e655794aeb3025930aadb544b187fb17de).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145413577
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145419045
  
  [Test build #43231 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43231/consoleFull)
 for   PR 8920 at commit 
[`c76bf5f`](https://github.com/apache/spark/commit/c76bf5f53714dc724b211e93140d80c8a95ca4d7).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[nilmeier]

Github user nilmeier commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145419186
  
Addressed Fred's Review Comments in the most recent update.  I was not able 
to fully address all issues raised...they are described below:

Review comments:
1) --> Try making the portion of the examples with input data more 
condensed; perhaps
 by reading the matrices from string constants
 X Done.

2) --> There's an orphan/duplicate ScalaDoc comment at line 356 of 
BlockMatrix.scala
 X Done.


3) --> Remove carriage return on lines 370, 481, 569, 732
 X Done.


4) --> Recommend adding a call to the new subtract method to the MLLib
programmer's guide
 X Done.


5) --> New API calls to BlockMatrix should have corresponding PySpark APIs
 O Mike Dusenberry has an open JIRA I will be meeting with Mike Dusenberry 
who has
   done Python API work before.


6) --> Error message at line 394 should print out the block sizes that 
don't match
 X added error message to .add and .subtract


7) --> The code at line 384 should multiply every element of b.head by -1 
as far
as I can see
 X Fixed this error.

8) --> Line 456 and 465-471 have wrong indentation
 X Done.


9) --> Scaladoc at 474 should state that blockRowRange and blockColRange 
are block
indexes, not row/column indexes
 X Done.

10) --> In lines 460-463, consider making a single pass over the blocks 
instead of
4 passes
 O I couldn't see an easy way to do this that didn't introduce additional 
layers of recursion
   that may impact performance.  I would like to add this to a later 
revision, as the recursive
   building procedures are a general design issue to consider here.

11) --> Add a note to SchurComplement that the current implementation 
assumes that
a11 fits into memory on the driver
X Done.

12) --> Might want to use a case class in return type of blockLUtoSolver
 O Will try to do this in the near future, but I didn't include the changes 
in this revision.

13) --> Take a close look at the performance impact of the chains of
multiplications at line 811 when there are many levels of recursion

 O We did some studies, and the code would benefit from refactoring to 
remove recursion.
  It would require a significant rewrite, and might be beyond the scope of 
the initial
  submission.


14) --> In recursiveSequencesBuild, you may want to break off more than one 
block
from the upper left corner of the matrix; in many cases, the available 
memory
on the driver can comfortably hold, say 10x10 blocks. You should be able to
query the SparkContext's memory information to determine how much heap
space is available for the in-memory portion of the computation. On a
side note, it would be good to produce a user-friendly error if it looks
like there is not enough local heap to process one block's data locally.

 O This may require a significant rewrite to handle correctlyI would 
like to try this for the next revision.

15) --> Might want to reuse buffers for the local data allocated at lines 
623-629
to avoid triggering global GC at each level of recursion.

 O I would like to explore this as well in the future, but I didn't address 
it in this update.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145419978
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145419943
  
  [Test build #43231 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43231/console)
 for   PR 8920 at commit 
[`c76bf5f`](https://github.com/apache/spark/commit/c76bf5f53714dc724b211e93140d80c8a95ca4d7).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145419979
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43231/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8563#discussion_r41108769
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
 ---
@@ -402,4 +445,402 @@ class BlockMatrix @Since("1.3.0") (
 s"A.colsPerBlock: $colsPerBlock, B.rowsPerBlock: 
${other.rowsPerBlock}")
 }
   }
+
+  /** Schur Complement of a BlockMatrix.  For a matrix that is in 4 
partitions:
+*  A=[a11, a12; a21; a22], the Schur Complement S is S = a22 - (a21 * 
a11^-1 * a12).
+* The Schur Complement is always (n-1) x (n-1), which is the size of 
a22.
+*
+* @return BlockMatrix Schur Complement as BlockMatrix
+* @since 1.6.0
+  */
+private[mllib] def SchurComplement: BlockMatrix = {
+ require(this.numRowBlocks == this.numColBlocks, "Block Matrix must be 
square.")
+ require(this.numRowBlocks > 1, "Block Matrix must be larger than one 
block.")
+ val topRange = (0, 0); val botRange = (1, this.numColBlocks - 1)
+ val a11 = this.subBlock(topRange, topRange)
+ val a12 = this.subBlock(topRange, botRange)
+ val a21 = this.subBlock(botRange, topRange)
+ val a22 = this.subBlock(botRange, botRange)
+
+val a11Brz = inv(a11.toBreeze) // note that intermediate a11 calcs 
derive from inv(a11)
+val a11Mtx = Matrices.dense(a11.numRows.toInt, a11.numCols.toInt, 
a11Brz.toArray)
+val a11RDD = this.blocks.sparkContext.parallelize(Seq(((0, 0), 
a11Mtx)))
+val a11Inv = new BlockMatrix(a11RDD, this.rowsPerBlock, 
this.colsPerBlock)
+
+val S = a22.subtract(a21.multiply(a11Inv.multiply(a12)))
+return S
+  }
+
+  /** Returns a rectangular (sub)BlockMatrix with block ranges as 
specified.
+*
+* @param blockRowRange The lower and upper row ranges, as (Int,Int)
+* @param blockColRange The lower and upper col ranges, as (Int, Int)
+* @return a BlockMatrix with (0,0) as the upper leftmost block index
+* @since 1.6.0
+*/
+
+  private [mllib] def subBlock(blockRowRange: (Int, Int), blockColRange: 
(Int, Int)):
+  BlockMatrix = {
+//  Extracts BlockMatrix elements from a specified range of block 
indices
+//  Creating a Sub BlockMatrix of rectangular shape.
+//  Also reindexes so that the upper left block is always (0, 0)
+
+// JNDB: Add a require statement ...rowMax<=size..
+val rowMin = blockRowRange._1;val rowMax = blockRowRange._2
+val colMin = blockColRange._1 ;   val colMax = blockColRange._2
+val extractedSeq = this.blocks.filter{ case((x, y), matrix) =>
+  x >= rowMin && x<= rowMax && // finding blocks
+y >= colMin && y<= colMax }.map{   // shifting indices
+  case(((x, y), matrix) ) => ((x-rowMin, y-colMin), matrix)
+}
+return new BlockMatrix(extractedSeq, rowsPerBlock, colsPerBlock)
+  }
+
+  /** computes the LU decomposition of a Single Block from BlockMatrix 
using the
+* Breeze LU method.  The method (as written) operates -only- on the 
upper
+* left (0,0) corner of the BlockMatrix.
+*
+* @return List[BDM[Double]] of Breeze Matrices (BDM) (P,L,U) for 
blockLU method.
+* @since 1.6.0
+  */
+  private [mllib] def singleBlockPLU: List[BDM[Double]] = {
+// returns PA = LU factorization from Breeze
+val PLU = LU(this.subBlock((0, 0), (0, 0)).toBreeze)
+val k = PLU._1.cols
+val L = lowerTriangular(PLU._1) - diag(diag(PLU._1)) + 
diag(DenseVector.fill(k){1.0})
+val U = upperTriangular(PLU._1);
+var P = diag(DenseVector.fill(k){1.0})
+val Pi = diag(DenseVector.fill(k){1.0})
+// size of square matrix
+for(i <- 0 to (k - 1)) { // i test populating permutation matrix
+  val I = i match {case 0 => k - 1 case _ => i - 1}
+  val J = PLU._2(i) -1
+  if (i != J) {  Pi(i, J) += 1.0; Pi(J, i) += 1.0; Pi(i, i) -= 1.0; 
Pi(J, J) -= 1.0}
+  P = Pi * P  // constructor Pi*P for PA=LU
+  if (i != J) { Pi(i, J) -= 1.0; Pi(J, i) -= 1.0; Pi(i, i) += 1.0; 
Pi(J, J) += 1.0}
+}
+return List(P, L, U)
+  }
+
+
+  /** This method reassigns 'absolute' index locations (i,j), to 
sequences.  This is
+* designed to reconsitute the orignal block locations that were lost 
in the
+* subBlock method.
+*
+* @param rowMin The new lowest row value
+* @param colMin The new lowest column value
+* @return an RDD of Sequences with new block indexing
+* @since 1.6.0
+*
+*/
+  private [mllib] def shiftIndices(rowMin: Int, colMin: Int): RDD[((Int, 
Int), Matrix)] = {
+// This routine recovers the absolute indexing of the block matrices 
for 

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145420699
  
  [Test build #43232 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43232/consoleFull)
 for   PR 8563 at commit 
[`625ae03`](https://github.com/apache/spark/commit/625ae03aa64a74de0742a03209cf9fece262a324).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8563#discussion_r41109171
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
 ---
@@ -402,4 +445,402 @@ class BlockMatrix @Since("1.3.0") (
 s"A.colsPerBlock: $colsPerBlock, B.rowsPerBlock: 
${other.rowsPerBlock}")
 }
   }
+
+  /** Schur Complement of a BlockMatrix.  For a matrix that is in 4 
partitions:
+*  A=[a11, a12; a21; a22], the Schur Complement S is S = a22 - (a21 * 
a11^-1 * a12).
+* The Schur Complement is always (n-1) x (n-1), which is the size of 
a22.
+*
+* @return BlockMatrix Schur Complement as BlockMatrix
+* @since 1.6.0
+  */
+private[mllib] def SchurComplement: BlockMatrix = {
+ require(this.numRowBlocks == this.numColBlocks, "Block Matrix must be 
square.")
+ require(this.numRowBlocks > 1, "Block Matrix must be larger than one 
block.")
+ val topRange = (0, 0); val botRange = (1, this.numColBlocks - 1)
+ val a11 = this.subBlock(topRange, topRange)
+ val a12 = this.subBlock(topRange, botRange)
+ val a21 = this.subBlock(botRange, topRange)
+ val a22 = this.subBlock(botRange, botRange)
+
+val a11Brz = inv(a11.toBreeze) // note that intermediate a11 calcs 
derive from inv(a11)
+val a11Mtx = Matrices.dense(a11.numRows.toInt, a11.numCols.toInt, 
a11Brz.toArray)
+val a11RDD = this.blocks.sparkContext.parallelize(Seq(((0, 0), 
a11Mtx)))
+val a11Inv = new BlockMatrix(a11RDD, this.rowsPerBlock, 
this.colsPerBlock)
+
+val S = a22.subtract(a21.multiply(a11Inv.multiply(a12)))
+return S
+  }
+
+  /** Returns a rectangular (sub)BlockMatrix with block ranges as 
specified.
+*
+* @param blockRowRange The lower and upper row ranges, as (Int,Int)
+* @param blockColRange The lower and upper col ranges, as (Int, Int)
+* @return a BlockMatrix with (0,0) as the upper leftmost block index
+* @since 1.6.0
+*/
+
+  private [mllib] def subBlock(blockRowRange: (Int, Int), blockColRange: 
(Int, Int)):
+  BlockMatrix = {
+//  Extracts BlockMatrix elements from a specified range of block 
indices
+//  Creating a Sub BlockMatrix of rectangular shape.
+//  Also reindexes so that the upper left block is always (0, 0)
+
+// JNDB: Add a require statement ...rowMax<=size..
+val rowMin = blockRowRange._1;val rowMax = blockRowRange._2
+val colMin = blockColRange._1 ;   val colMax = blockColRange._2
+val extractedSeq = this.blocks.filter{ case((x, y), matrix) =>
+  x >= rowMin && x<= rowMax && // finding blocks
+y >= colMin && y<= colMax }.map{   // shifting indices
+  case(((x, y), matrix) ) => ((x-rowMin, y-colMin), matrix)
+}
+return new BlockMatrix(extractedSeq, rowsPerBlock, colsPerBlock)
+  }
+
+  /** computes the LU decomposition of a Single Block from BlockMatrix 
using the
+* Breeze LU method.  The method (as written) operates -only- on the 
upper
+* left (0,0) corner of the BlockMatrix.
+*
+* @return List[BDM[Double]] of Breeze Matrices (BDM) (P,L,U) for 
blockLU method.
+* @since 1.6.0
+  */
+  private [mllib] def singleBlockPLU: List[BDM[Double]] = {
+// returns PA = LU factorization from Breeze
+val PLU = LU(this.subBlock((0, 0), (0, 0)).toBreeze)
+val k = PLU._1.cols
+val L = lowerTriangular(PLU._1) - diag(diag(PLU._1)) + 
diag(DenseVector.fill(k){1.0})
+val U = upperTriangular(PLU._1);
+var P = diag(DenseVector.fill(k){1.0})
+val Pi = diag(DenseVector.fill(k){1.0})
+// size of square matrix
+for(i <- 0 to (k - 1)) { // i test populating permutation matrix
+  val I = i match {case 0 => k - 1 case _ => i - 1}
+  val J = PLU._2(i) -1
+  if (i != J) {  Pi(i, J) += 1.0; Pi(J, i) += 1.0; Pi(i, i) -= 1.0; 
Pi(J, J) -= 1.0}
--- End diff --

make it multiple lines.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[dbtsai]

Github user dbtsai commented on a diff in the pull request:

https://github.com/apache/spark/pull/8563#discussion_r41109156
  
--- Diff: 
mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/BlockMatrix.scala
 ---
@@ -402,4 +445,402 @@ class BlockMatrix @Since("1.3.0") (
 s"A.colsPerBlock: $colsPerBlock, B.rowsPerBlock: 
${other.rowsPerBlock}")
 }
   }
+
+  /** Schur Complement of a BlockMatrix.  For a matrix that is in 4 
partitions:
+*  A=[a11, a12; a21; a22], the Schur Complement S is S = a22 - (a21 * 
a11^-1 * a12).
+* The Schur Complement is always (n-1) x (n-1), which is the size of 
a22.
+*
+* @return BlockMatrix Schur Complement as BlockMatrix
+* @since 1.6.0
+  */
+private[mllib] def SchurComplement: BlockMatrix = {
+ require(this.numRowBlocks == this.numColBlocks, "Block Matrix must be 
square.")
+ require(this.numRowBlocks > 1, "Block Matrix must be larger than one 
block.")
+ val topRange = (0, 0); val botRange = (1, this.numColBlocks - 1)
+ val a11 = this.subBlock(topRange, topRange)
+ val a12 = this.subBlock(topRange, botRange)
+ val a21 = this.subBlock(botRange, topRange)
+ val a22 = this.subBlock(botRange, botRange)
+
+val a11Brz = inv(a11.toBreeze) // note that intermediate a11 calcs 
derive from inv(a11)
+val a11Mtx = Matrices.dense(a11.numRows.toInt, a11.numCols.toInt, 
a11Brz.toArray)
+val a11RDD = this.blocks.sparkContext.parallelize(Seq(((0, 0), 
a11Mtx)))
+val a11Inv = new BlockMatrix(a11RDD, this.rowsPerBlock, 
this.colsPerBlock)
+
+val S = a22.subtract(a21.multiply(a11Inv.multiply(a12)))
+return S
+  }
+
+  /** Returns a rectangular (sub)BlockMatrix with block ranges as 
specified.
+*
+* @param blockRowRange The lower and upper row ranges, as (Int,Int)
+* @param blockColRange The lower and upper col ranges, as (Int, Int)
+* @return a BlockMatrix with (0,0) as the upper leftmost block index
+* @since 1.6.0
+*/
+
+  private [mllib] def subBlock(blockRowRange: (Int, Int), blockColRange: 
(Int, Int)):
+  BlockMatrix = {
+//  Extracts BlockMatrix elements from a specified range of block 
indices
+//  Creating a Sub BlockMatrix of rectangular shape.
+//  Also reindexes so that the upper left block is always (0, 0)
+
+// JNDB: Add a require statement ...rowMax<=size..
+val rowMin = blockRowRange._1;val rowMax = blockRowRange._2
+val colMin = blockColRange._1 ;   val colMax = blockColRange._2
+val extractedSeq = this.blocks.filter{ case((x, y), matrix) =>
+  x >= rowMin && x<= rowMax && // finding blocks
+y >= colMin && y<= colMax }.map{   // shifting indices
+  case(((x, y), matrix) ) => ((x-rowMin, y-colMin), matrix)
+}
+return new BlockMatrix(extractedSeq, rowsPerBlock, colsPerBlock)
+  }
+
+  /** computes the LU decomposition of a Single Block from BlockMatrix 
using the
+* Breeze LU method.  The method (as written) operates -only- on the 
upper
+* left (0,0) corner of the BlockMatrix.
+*
+* @return List[BDM[Double]] of Breeze Matrices (BDM) (P,L,U) for 
blockLU method.
+* @since 1.6.0
+  */
+  private [mllib] def singleBlockPLU: List[BDM[Double]] = {
+// returns PA = LU factorization from Breeze
+val PLU = LU(this.subBlock((0, 0), (0, 0)).toBreeze)
+val k = PLU._1.cols
+val L = lowerTriangular(PLU._1) - diag(diag(PLU._1)) + 
diag(DenseVector.fill(k){1.0})
+val U = upperTriangular(PLU._1);
+var P = diag(DenseVector.fill(k){1.0})
+val Pi = diag(DenseVector.fill(k){1.0})
+// size of square matrix
+for(i <- 0 to (k - 1)) { // i test populating permutation matrix
+  val I = i match {case 0 => k - 1 case _ => i - 1}
--- End diff --

just if else here for both performance and reliability. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145424182
  
  [Test build #43234 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43234/consoleFull)
 for   PR 8563 at commit 
[`de594b1`](https://github.com/apache/spark/commit/de594b19fc6c86ef56c70702b68d595de21c8bad).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145428420
  
  [Test build #43232 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43232/console)
 for   PR 8563 at commit 
[`625ae03`](https://github.com/apache/spark/commit/625ae03aa64a74de0742a03209cf9fece262a324).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10889] [Streaming] Bump KCL to add Mill...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/8957


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10889] [Streaming] Bump KCL to add Mill...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/8957#issuecomment-145327961
  
Merged to master / 1.5


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9570] [DOCS] Consistent recommendation ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/8968


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10887] [SQL] Build HashedRelation outsi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8953#issuecomment-145320578
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10887] [SQL] Build HashedRelation outsi...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8953#issuecomment-145320579
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43223/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10887] [SQL] Build HashedRelation outsi...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8953#issuecomment-145320558
  
  [Test build #43223 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43223/console)
 for   PR 8953 at commit 
[`b9d5bc9`](https://github.com/apache/spark/commit/b9d5bc9efcbd3eae8ce6a114cf5fc16864ccbe0c).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class BinaryHashJoinNode(`
  * `case class BroadcastHashJoinNode(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10669] [Docs] Link to each language's A...

[keypointt]

GitHub user keypointt opened a pull request:

https://github.com/apache/spark/pull/8974

[SPARK-10669] [Docs] Link to each language's API in codetabs in ML docs: 
spark.mllib

my first PR, please let me know if more work is needed. Thanks a lot!

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/keypointt/spark SPARK-10669

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8974.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8974


commit f8289891d5b32fffdc6a4ce077d8d206e015119f
Author: Xin Ren 
Date:   2015-10-02T07:00:36Z

[SPARK-10669] test modify

commit 67c67158c93a4a8c5a963ecf821f5e85e1228bf3
Author: Xin Ren 
Date:   2015-10-04T16:54:35Z

[SPARK-10669] Link to each language API in codetabs in spark mllib docs

commit 31960f6790a75fd037ffd879a8d17e546c5fa6fa
Author: Xin Ren 
Date:   2015-10-04T16:59:56Z

[SPARK-10669] minor correction




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10669] [Docs] Link to each language's A...

[keypointt]

Github user keypointt commented on the pull request:

https://github.com/apache/spark/pull/8974#issuecomment-145368459
  
@srowen After checking the ticket you mentioned, now I know what is exactly 
expected. I'm closing this one and continue work on SPARK-10669. Thank you


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10669] [Docs] Link to each language's A...

[keypointt]

Github user keypointt closed the pull request at:

https://github.com/apache/spark/pull/8974


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8962#issuecomment-145368441
  
  [Test build #43225 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43225/consoleFull)
 for   PR 8962 at commit 
[`57cc150`](https://github.com/apache/spark/commit/57cc150e9cc78c0ac87bd347308e684e5f876f5f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7869][SQL] Adding Postgres JSON and JSO...

[0x0FFF]

Github user 0x0FFF commented on the pull request:

https://github.com/apache/spark/pull/8948#issuecomment-145356466
  
Adding @rxin


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10669] [Docs] Link to each language's A...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8974#issuecomment-145367212
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8962#issuecomment-145368122
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8962#issuecomment-145368117
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

[rerngvit]

Github user rerngvit commented on the pull request:

https://github.com/apache/spark/pull/8962#issuecomment-145368055
  
@sun-rui I revised according to your comments. Please have a look.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8374#issuecomment-145367948
  
  [Test build #43224 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43224/console)
 for   PR 8374 at commit 
[`d50bdf7`](https://github.com/apache/spark/commit/d50bdf796c96f827aa920c31dd3d5822414f178d).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8374#issuecomment-145368001
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43224/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8374#issuecomment-145368000
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8374#issuecomment-145358557
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8374#issuecomment-145358566
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8374#issuecomment-145358933
  
  [Test build #43224 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43224/consoleFull)
 for   PR 8374 at commit 
[`d50bdf7`](https://github.com/apache/spark/commit/d50bdf796c96f827aa920c31dd3d5822414f178d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10669] [Docs] Link to each language's A...

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/8974#issuecomment-145367909
  
@keypointt I don't this is what the JIRA was getting at. See 
https://issues.apache.org/jira/browse/SPARK-10670 and 
https://github.com/apache/spark/pull/8901 instead. I think you'd want to start 
over and close this one though some of the extra links are nice here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8962#issuecomment-145369661
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10101] [SQL] Add maxlength to JDBC fiel...

[rama-mullapudi]

Github user rama-mullapudi commented on the pull request:

https://github.com/apache/spark/pull/8374#issuecomment-145380014
  
Please review the pull for approval to merge.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8962#issuecomment-145369662
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43225/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10905][SparkR]: Export freqItems() for ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8962#issuecomment-145369619
  
  [Test build #43225 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43225/console)
 for   PR 8962 at commit 
[`57cc150`](https://github.com/apache/spark/commit/57cc150e9cc78c0ac87bd347308e684e5f876f5f).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9034] [SQL] Reflect field names defined...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8456#issuecomment-145408372
  
  [Test build #43227 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43227/consoleFull)
 for   PR 8456 at commit 
[`0382b3f`](https://github.com/apache/spark/commit/0382b3fd93b561c8ecd74ad0ffdd7ccc37139e2f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145408528
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10916] Set perm gen size when launching...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/8970#issuecomment-145400271
  
/cc @tgravescs @harishreedharan @sryza 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[navis]

Github user navis commented on a diff in the pull request:

https://github.com/apache/spark/pull/8899#discussion_r41105674
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JSONRelation.scala
 ---
@@ -79,12 +76,16 @@ private[sql] class JSONRelation(
 
   override val needConversion: Boolean = false
 
-  private def createBaseRdd(inputPaths: Array[FileStatus]): RDD[String] = {
+  private def createBaseRdd(
+  inputPaths: Array[FileStatus],
--- End diff --

Sorry for late reply. Yes, look not good and need not to be handed over. 
I'll change "paths = sources" to "paths = cachedSourcePaths()"


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[navis]

Github user navis commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145407288
  
@srowen Sorry, I've been on vacation. Addressed comment.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145407558
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9034] [SQL] Reflect field names defined...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8456#issuecomment-145407559
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-9034] [SQL] Reflect field names defined...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8456#issuecomment-145407569
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8354#issuecomment-145407561
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145407564
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8354#issuecomment-145407570
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10724] [SQL] SQL's floor() returns DOUB...

[navis]

Github user navis closed the pull request at:

https://github.com/apache/spark/pull/8893


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10151] [SQL] Support invocation of hive...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8354#issuecomment-145408422
  
  [Test build #43228 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43228/consoleFull)
 for   PR 8354 at commit 
[`fd799a1`](https://github.com/apache/spark/commit/fd799a1163a47c2819ab8d91d2131968ab45871f).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145408398
  
  [Test build #43226 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43226/consoleFull)
 for   PR 8899 at commit 
[`1a4ddec`](https://github.com/apache/spark/commit/1a4ddec869136b7b98d5bdc70526ad69ad61a954).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145408529
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43226/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10709] [SQL] When loading a json datase...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8899#issuecomment-145408526
  
  [Test build #43226 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43226/console)
 for   PR 8899 at commit 
[`1a4ddec`](https://github.com/apache/spark/commit/1a4ddec869136b7b98d5bdc70526ad69ad61a954).
 * This patch **fails Scala style tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class LogicalRelation(`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145429343
  
  [Test build #43234 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43234/console)
 for   PR 8563 at commit 
[`de594b1`](https://github.com/apache/spark/commit/de594b19fc6c86ef56c70702b68d595de21c8bad).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `final class ChiSqSelector(override val uid: String)`
  * `final class QuantileDiscretizer(override val uid: String)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145429498
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43234/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8514] LU factorization on BlockMatrix

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8563#issuecomment-145429492
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145430083
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43233/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145430081
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10836] [SparkR] Added sort(x, decreasin...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8920#issuecomment-145430069
  
  [Test build #43233 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43233/console)
 for   PR 8920 at commit 
[`61e7646`](https://github.com/apache/spark/commit/61e76468d0a79649e2fd7b49d378071ed4265764).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10921] [YARN] Completely remove the use...

[jaceklaskowski]

GitHub user jaceklaskowski opened a pull request:

https://github.com/apache/spark/pull/8976

[SPARK-10921] [YARN] Completely remove the use of SparkContext.prefer…

…redNodeLocationData

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jaceklaskowski/spark SPARK-10921

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/8976.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #8976


commit 1353686d6f963f6316a279f3425effd17973f61a
Author: Jacek Laskowski 
Date:   2015-10-05T04:12:05Z

[SPARK-10921] [YARN] Completely remove the use of 
SparkContext.preferredNodeLocationData




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10921] [YARN] Completely remove the use...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/8976#issuecomment-145432564
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-10918] [CORE] Prevent task failed for e...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/8975#issuecomment-145433645
  
  [Test build #43235 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43235/console)
 for   PR 8975 at commit 
[`18cbf73`](https://github.com/apache/spark/commit/18cbf73b81023e8adac6326e3225799291cbf8b5).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org