Re: VerifyError running Spark SQL code?

2015-11-25 Thread Josh Rosen 
I think I've also seen this issue as well, but in a different suite. I
wasn't able to easily get to the bottom of it, though. What JDK / JRE are
you using? I'm on


Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

on OSX.

On Wed, Nov 25, 2015 at 4:51 PM, Marcelo Vanzin  wrote:

> I've been running into this error when running Spark SQL recently; no
> matter what I try (completely clean build or anything else) doesn't
> seem to fix it. Anyone has some idea of what's wrong?
>
> [info] Exception encountered when attempting to run a suite with class
> name: org.apache.spark.sql.execution.ui.SQLListenerMemoryLeakSuite ***
> ABORTED *** (4 seconds, 111 milliseconds)
> [info]   java.lang.VerifyError: Bad  method call from inside of a
> branch
> [info] Exception Details:
> [info]   Location:
> [info]
>  
> org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.(Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;)V
> @82: invokespecial
>
> Same happens with spark shell (when instantiating SQLContext), so not
> an issue with the test code...
>
> --
> Marcelo
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> For additional commands, e-mail: dev-h...@spark.apache.org
>
>

VerifyError running Spark SQL code?

2015-11-25 Thread Marcelo Vanzin 
I've been running into this error when running Spark SQL recently; no
matter what I try (completely clean build or anything else) doesn't
seem to fix it. Anyone has some idea of what's wrong?

[info] Exception encountered when attempting to run a suite with class
name: org.apache.spark.sql.execution.ui.SQLListenerMemoryLeakSuite ***
ABORTED *** (4 seconds, 111 milliseconds)
[info]   java.lang.VerifyError: Bad  method call from inside of a branch
[info] Exception Details:
[info]   Location:
[info] 
org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.(Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;)V
@82: invokespecial

Same happens with spark shell (when instantiating SQLContext), so not
an issue with the test code...

-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: VerifyError running Spark SQL code?

2015-11-25 Thread Marcelo Vanzin 
$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)

(On Linux.) It's not that particular suite, though, it's anything I do
that touches Spark SQL...

On Wed, Nov 25, 2015 at 4:54 PM, Josh Rosen  wrote:
> I think I've also seen this issue as well, but in a different suite. I
> wasn't able to easily get to the bottom of it, though. What JDK / JRE are
> you using? I'm on
>
>
> Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
>
> on OSX.
>
> On Wed, Nov 25, 2015 at 4:51 PM, Marcelo Vanzin  wrote:
>>
>> I've been running into this error when running Spark SQL recently; no
>> matter what I try (completely clean build or anything else) doesn't
>> seem to fix it. Anyone has some idea of what's wrong?
>>
>> [info] Exception encountered when attempting to run a suite with class
>> name: org.apache.spark.sql.execution.ui.SQLListenerMemoryLeakSuite ***
>> ABORTED *** (4 seconds, 111 milliseconds)
>> [info]   java.lang.VerifyError: Bad  method call from inside of a
>> branch
>> [info] Exception Details:
>> [info]   Location:
>> [info]
>> org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.(Lorg/apache/spark/sql/catalyst/expressions/Expression;Lorg/apache/spark/sql/catalyst/expressions/Expression;)V
>> @82: invokespecial
>>
>> Same happens with spark shell (when instantiating SQLContext), so not
>> an issue with the test code...
>>
>> --
>> Marcelo
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>



-- 
Marcelo

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: VerifyError running Spark SQL code?

2015-11-25 Thread Marcelo Vanzin 
Seems to be some new thing with recent JDK updates according to the
intertubes. This patch seems to work around it:

--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/HyperLogLogPlusPlus.scala
@@ -63,11 +63,7 @@ case class HyperLogLogPlusPlus(
  def this(child: Expression, relativeSD: Expression) = {
this(
  child = child,
-  relativeSD = relativeSD match {
-case Literal(d: Double, DoubleType) => d
-case _ =>
-  throw new AnalysisException("The second argument should be
a double literal.")
-  },
+  relativeSD = HyperLogLogPlusPlus.validateRelativeSd(relativeSD),
  mutableAggBufferOffset = 0,
  inputAggBufferOffset = 0)
  }
@@ -448,4 +444,11 @@ object HyperLogLogPlusPlus {
Array(189083, 185696.913, 182348.774, 179035.946, 175762.762,
172526.444, 169329.754, 166166.099, 163043.269, 159958.91, 156907.912,
153906.845,
150924.199, 147996.568, 145093.457, 142239.233, 139421.475, 136632.27,
133889.588, 131174.2, 128511.619, 125868.621, 123265.385, 120721.061,
118181.76
9, 115709.456, 113252.446, 110840.198, 108465.099, 106126.164,
103823.469, 101556.618, 99308.004, 97124.508, 94937.803, 92833.731,
90745.061, 88677.62
7, 86617.47, 84650.442, 82697.833, 80769.132, 78879.629, 77014.432,
75215.626, 73384.587, 71652.482, 69895.93, 68209.301, 66553.669,
64921.981, 63310.
323, 61742.115, 60205.018, 58698.658, 57190.657, 55760.865, 54331.169,
52908.167, 51550.273, 50225.254, 48922.421, 47614.533, 46362.049,
45098.569, 43
926.083, 42736.03, 41593.473, 40425.26, 39316.237, 38243.651,
37170.617, 36114.609, 35084.19, 34117.233, 33206.509, 32231.505,
31318.728, 30403.404, 2
9540.055001, 28679.236, 27825.862, 26965.216, 26179.148, 25462.08,
24645.952, 23922.523, 23198.144, 22529.128, 21762.417999,
21134.779, 20459.
117, 19840.818, 19187.04, 18636.368999, 17982.831,
17439.738999, 16874.547, 16358.216999, 15835.684, 15352.914,
14823.681, 14329.313, 1381
6.897, 13342.874, 12880.882, 12491.648, 12021.254, 11625.392,
11293.761001, 10813.697, 10456.209, 10099.074, 9755.391,
9393.1850006, 9
047.5790003, 8657.984, 8395.8590005, 8033,
7736.9590003, 7430.5969995, 7258.4769996,
6924.5820005, 6691.293, 6
357.9250005, 6202.0570003, 5921.1970004, 5628.283,
5404.967, 5226.7110001, 4990.7560005,
4799.7739998, 4622.93
09998, 4472.478, 4171.7870001, 3957.462,
3868.9520005, 3691.1430004, 3474.6310005,
3341.6720002, 3109.141, 307
1.9740005, 2796.4039998, 2756.1779996, 2611.467,
2471.935, 2382.2639997, 2209.2240005,
2142.283, 2013.9610
001, 1911.184, 1818.2709995, 1668.4790005,
1519.6580005, 1469.6759998, 1367.1380004,
1248.5289998, 1181.236000
3, 1022.7190004, 1088.2070005, 959.0360008,
876.09599903, 791.18399892, 703.33700058,
731.9453, 586.8640006, 526.0
2499907, 323.00499888, 320.44800091, 340.67299952,
309.63899966, 216.60199955, 102.92299952,
19.23907, -0.1140
0059605, -32.624000689, -89.317999702, -153.49799905,
-64.297000205, -143.6956, -259.49799905,
-253.01799924, -213.948
00091, -397.5984, -434.00600052, -403.47500093,
-297.95800101, -404.31700039, -528.89899976,
-506.62100043, -513.2
0500075, -479.35100024, -596.13999898, -527.0163,
-664.68100099, -680.30600099, -704.0547,
-850.48600034, -757
.4320003, -713.30899892)
  )
  // scalastyle:on
+
+  private def validateRelativeSd(relativeSD: Expression): Double =
relativeSD match {
+case Literal(d: Double, DoubleType) => d
+case _ =>
+  throw new AnalysisException("The second argument should be a
double literal.")
+  }
+
}


On Wed, Nov 25, 2015 at 5:29 PM, Marcelo Vanzin  wrote:
> $ java -version
> java version "1.7.0_67"
> Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
>
> (On Linux.) It's not that particular suite, though, it's anything I do
> that touches Spark SQL...
>
> On Wed, Nov 25, 2015 at 4:54 PM, Josh Rosen  wrote:
>> I think I've also seen this issue as well, but in a different suite. I
>> wasn't able to easily get to the bottom of it, though. What JDK / JRE are
>> you using? I'm on
>>
>>
>> Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
>> Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
>>
>> on OSX.
>>
>> On Wed, Nov 25, 2015 at 4:51 PM, Marcelo Vanzin  wrote:
>>>
>>> I've been running into this error when running Spark SQL recently; no
>>> matter what I try (completely clean build or anything else) doesn't
>>> seem to fix it. Anyone has some