[jira] [Commented] (SPARK-17403) Fatal Error: Scan cached strings
[
https://issues.apache.org/jira/browse/SPARK-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16682591#comment-16682591
]
Eyal Farago commented on SPARK-17403:
-
[~paul_lysak], [~hvanhovell],
please notice the exception happens when building the column buffers, hence the
corrupt string comes from upstream - in the repro code this is the join, not
necessarily an in-mem-relation.
[~hvanhovell], I'd suspect that the upstream operator somehow overwrites the
row/buffer returned from the Iterator.next method while the downstream code
(in-mem column building code) doesn't take the necessary precautions (copy). if
this is the case I'd suspect the issue is somehow related to code generation
either in the join or InMemoryTableScanExec operator.
another option, [~paul_lysak] mentioned that the problem happens when the
application is 'big enough', given that DataSet's default persistence level is
_memory and disk_, is it possible that spark starts evacuating these blocks
from memory and then bad things start happening?
> Fatal Error: Scan cached strings
>
>
> Key: SPARK-17403
> URL: https://issues.apache.org/jira/browse/SPARK-17403
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0
> Environment: Spark standalone cluster (3 Workers, 47 cores)
> Ubuntu 14
> Java 8
>Reporter: Ruben Hernando
>Priority: Major
>
> The process creates views from JDBC (SQL server) source and combines them to
> create other views.
> Finally it dumps results via JDBC
> Error:
> {quote} # JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build
> 1.8.0_101-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode
> linux-amd64 )
> # Problematic frame:
> # J 4895 C1 org.apache.spark.unsafe.Platform.getLong(Ljava/lang/Object;J)J
> (9 bytes) @ 0x7fbb355dfd6c [0x7fbb355dfd60+0xc]
> #{quote}
> SQL Query plan (fields truncated):
> {noformat}
> == Parsed Logical Plan ==
> 'Project [*]
> +- 'UnresolvedRelation `COEQ_63`
> == Analyzed Logical Plan ==
> InstanceId: bigint, price: double, ZoneId: int, priceItemId: int, priceId: int
> Project [InstanceId#20236L, price#20237, ZoneId#20239, priceItemId#20242,
> priceId#20244]
> +- SubqueryAlias coeq_63
>+- Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- SubqueryAlias 6__input
> +-
> Relation[_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,SL_Xcl_C#152,SL_Css_Cojs#153L,SL_Config#154,SL_CREATEDON#
> .. 36 more fields] JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables]. ... [...] FROM [sch].[SLTables] [SLTables] JOIN sch.TPSLTables
> TPSLTables ON [TPSLTables].[_TableSL_SID] = [SLTables].[_TableSL_SID] where
> _TP = 24) input)
> def pers
> == Optimized Logical Plan ==
> Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], [SLTables].[TableSHID],
> [TPSLTables].[SL_ACT_GI_DTE], ... [...] FROM [sch].[SLTables] [SLTables]
> JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,...
> 36 more fields]
> == Physical Plan ==
> *Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryTableScan [_TableSL_SID#143L, SL_RD_ColR_N#189]
>: +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: : +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], ... [...] FROM [sch].[SLTables]
> [SLTables] JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,,...
> 36 more fields]
> {noformat}
--
This message was sent by
[jira] [Commented] (SPARK-17403) Fatal Error: Scan cached strings
[
https://issues.apache.org/jira/browse/SPARK-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15984987#comment-15984987
]
Paul Lysak commented on SPARK-17403:
Hope that helps - finally managed to reproduce it without using production data:
{code}
import org.apache.spark.sql.functions._
val leftRows = spark.sparkContext.parallelize(numSlices = 3000, seq = for
(k <- 1 to 1000; j <- 1 to 1000) yield Row(k, j))
.flatMap(r => (1 to 1000).map(l => Row.fromSeq(r.toSeq :+ l)))
val leftDf = spark.createDataFrame(leftRows, StructType(Seq(
StructField("k", IntegerType),
StructField("j", IntegerType),
StructField("l", IntegerType)
)))
.withColumn("combinedKey", expr("k*100 + j*1000 + l"))
.withColumn("fixedCol", lit("sampleVal"))
.withColumn("combKeyStr", format_number(col("combinedKey"), 0))
.withColumn("k100", expr("k*100"))
.withColumn("j100", expr("j*100"))
.withColumn("l100", expr("l*100"))
.withColumn("k_200", expr("k+200"))
.withColumn("j_200", expr("j+200"))
.withColumn("l_200", expr("l+200"))
.withColumn("strCol1_1", concat(lit("value of sample column number one
with which column k will be concatenated:" * 5), format_number(col("k"), 0)))
.withColumn("strCol1_2", concat(lit("value of sample column two one with
which column j will be concatenated:" * 5), format_number(col("j"), 0)))
.withColumn("strCol1_3", concat(lit("value of sample column three one
with which column r will be concatenated:" * 5), format_number(col("l"), 0)))
.withColumn("strCol2_1", concat(lit("value of sample column number one
with which column k will be concatenated:" * 5), format_number(col("k"), 0)))
.withColumn("strCol2_2", concat(lit("value of sample column two one with
which column j will be concatenated:" * 5), format_number(col("j"), 0)))
.withColumn("strCol2_3", concat(lit("value of sample column three one
with which column r will be concatenated:" * 5), format_number(col("l"), 0)))
.withColumn("strCol3_1", concat(lit("value of sample column number one
with which column k will be concatenated:" * 5), format_number(col("k"), 0)))
.withColumn("strCol3_2", concat(lit("value of sample column two one with
which column j will be concatenated:" * 5), format_number(col("j"), 0)))
.withColumn("strCol3_3", concat(lit("value of sample column three one
with which column r will be concatenated:" * 5), format_number(col("l"), 0)))
//if further columns commented out - error disappears
leftDf.cache()
println("= leftDf count:" + leftDf.count())
leftDf.show(10)
val rightRows = spark.sparkContext.parallelize((1 to 800).map(i => Row(i,
"k_" + i, "sampleVal")))
val rightDf = spark.createDataFrame(rightRows, StructType(Seq(
StructField("k", IntegerType),
StructField("kStr", StringType),
StructField("sampleCol", StringType)
)))
rightDf.cache()
println("= rightDf count:" + rightDf.count())
rightDf.show(10)
val joinedDf = leftDf.join(broadcast(rightDf), usingColumns = Seq("k"),
joinType = "left")
joinedDf.cache()
println("= joinedDf count:" + joinedDf.count())
joinedDf.show(10)
{code}
ApplicationMaster fails with such exception:
{noformat}
User class threw exception: org.apache.spark.SparkException: Job aborted due to
stage failure: Task 949 in stage 8.0 failed 4 times, most recent failure: Lost
task 949.3 in stage 8.0 (TID 4922, ip-10-35-162-219.ec2.internal, executor
139): java.lang.NegativeArraySizeException
at org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:229)
at org.apache.spark.unsafe.types.UTF8String.clone(UTF8String.java:826)
at
org.apache.spark.sql.execution.columnar.StringColumnStats.gatherStats(ColumnStats.scala:216)
at
org.apache.spark.sql.execution.columnar.NullableColumnBuilder$class.appendFrom(NullableColumnBuilder.scala:55)
at
org.apache.spark.sql.execution.columnar.NativeColumnBuilder.org$apache$spark$sql$execution$columnar$compression$CompressibleColumnBuilder$$super$appendFrom(ColumnBuilder.scala:97)
at
org.apache.spark.sql.execution.columnar.compression.CompressibleColumnBuilder$class.appendFrom(CompressibleColumnBuilder.scala:78)
at
org.apache.spark.sql.execution.columnar.NativeColumnBuilder.appendFrom(ColumnBuilder.scala:97)
at
org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.next(InMemoryRelation.scala:122)
at
org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.next(InMemoryRelation.scala:97)
at
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:957)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
at
[jira] [Commented] (SPARK-17403) Fatal Error: Scan cached strings
[ https://issues.apache.org/jira/browse/SPARK-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982595#comment-15982595 ] Paul Lysak commented on SPARK-17403: Looks like we have the same issue with Spark 2.1 on YARN (Amazon EMR release emr-5.4.0). Workaround that solves the issue for us (at the cost of some performance) is to use df.persist(StorageLevel.DISK_ONLY) instead of df.cache(). Depending on the node types, memory settings, storage level and some other factors I couldn't clearly identify it may appear as User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 158 in stage 504.0 failed 4 times, most recent failure: Lost task 158.3 in stage 504.0 (TID 427365, ip-10-35-162-171.ec2.internal, executor 83): java.lang.NegativeArraySizeException at org.apache.spark.unsafe.types.UTF8String.getBytes(UTF8String.java:229) at org.apache.spark.unsafe.types.UTF8String.clone(UTF8String.java:826) at org.apache.spark.sql.execution.columnar.StringColumnStats.gatherStats(ColumnStats.scala:217) at org.apache.spark.sql.execution.columnar.NullableColumnBuilder$class.appendFrom(NullableColumnBuilder.scala:55) at org.apache.spark.sql.execution.columnar.NativeColumnBuilder.org$apache$spark$sql$execution$columnar$compression$CompressibleColumnBuilder$$super$appendFrom(ColumnBuilder.scala:97) at org.apache.spark.sql.execution.columnar.compression.CompressibleColumnBuilder$class.appendFrom(CompressibleColumnBuilder.scala:78) at org.apache.spark.sql.execution.columnar.NativeColumnBuilder.appendFrom(ColumnBuilder.scala:97) at org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.next(InMemoryRelation.scala:122) at org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.next(InMemoryRelation.scala:97) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:957) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) at org.apache.spark.rdd.RDD.iterator(RDD.scala:285) or as User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 27 in stage 61.0 failed 4 times, most recent failure: Lost task 27.3 in stage 61.0 (TID 36167, ip-10-35-162-149.ec2.internal, executor 1): java.lang.OutOfMemoryError: Java heap space at org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder.grow(BufferHolder.java:73) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_38$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) at org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.next(InMemoryRelation.scala:107) at org.apache.spark.sql.execution.columnar.InMemoryRelation$$anonfun$1$$anon$1.next(InMemoryRelation.scala:97) at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:216) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:957) at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334) or as 2017-04-24 19:02:45,951 ERROR org.apache.spark.util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-3,5,main] java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder.grow(BufferHolder.java:73) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply_37$(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.execution.joins.HashJoin$$anonfun$join$1.apply(HashJoin.scala:217) at org.apache.spark.sql.execution.joins.HashJoin$$anonfun$join$1.apply(HashJoin.scala:215) at
[jira] [Commented] (SPARK-17403) Fatal Error: Scan cached strings
[
https://issues.apache.org/jira/browse/SPARK-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470311#comment-15470311
]
Herman van Hovell commented on SPARK-17403:
---
[~rhernando] My current hypothesis is a bit different. The in memory cache
compresses data, it uses dictionary encoding for strings. The strings that go
into the dictionary are probably not copied, this is fine for on-heap but is a
problem for off-heap because these strings might point to released memory.
> Fatal Error: Scan cached strings
>
>
> Key: SPARK-17403
> URL: https://issues.apache.org/jira/browse/SPARK-17403
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0
> Environment: Spark standalone cluster (3 Workers, 47 cores)
> Ubuntu 14
> Java 8
>Reporter: Ruben Hernando
>
> The process creates views from JDBC (SQL server) source and combines them to
> create other views.
> Finally it dumps results via JDBC
> Error:
> {quote}
> # JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build
> 1.8.0_101-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode
> linux-amd64 )
> # Problematic frame:
> # J 4895 C1 org.apache.spark.unsafe.Platform.getLong(Ljava/lang/Object;J)J (9
> bytes) @ 0x7fbb355dfd6c [0x7fbb355dfd60+0xc]
> #
> {quote}
> SQL Query plan (fields truncated):
> {noformat}
> == Parsed Logical Plan ==
> 'Project [*]
> +- 'UnresolvedRelation `COEQ_63`
> == Analyzed Logical Plan ==
> InstanceId: bigint, price: double, ZoneId: int, priceItemId: int, priceId: int
> Project [InstanceId#20236L, price#20237, ZoneId#20239, priceItemId#20242,
> priceId#20244]
> +- SubqueryAlias coeq_63
>+- Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- SubqueryAlias 6__input
> +-
> Relation[_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,SL_Xcl_C#152,SL_Css_Cojs#153L,SL_Config#154,SL_CREATEDON#
> .. 36 more fields] JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables]. ... [...] FROM [sch].[SLTables] [SLTables] JOIN sch.TPSLTables
> TPSLTables ON [TPSLTables].[_TableSL_SID] = [SLTables].[_TableSL_SID] where
> _TP = 24) input)
> == Optimized Logical Plan ==
> Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], [SLTables].[TableSHID],
> [TPSLTables].[SL_ACT_GI_DTE], ... [...] FROM [sch].[SLTables] [SLTables]
> JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,...
> 36 more fields]
> == Physical Plan ==
> *Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryTableScan [_TableSL_SID#143L, SL_RD_ColR_N#189]
>: +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: : +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], ... [...] FROM [sch].[SLTables]
> [SLTables] JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,,...
> 36 more fields]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17403) Fatal Error: Scan cached strings
[
https://issues.apache.org/jira/browse/SPARK-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470293#comment-15470293
]
Ruben Hernando commented on SPARK-17403:
I think the error happens when a cached data frame is being queried by 2
different tasks at once and both tasks try to cache that data frame
If I use serialised persistence I get "ConcurrentModificationException" writing
the serialised object
> Fatal Error: Scan cached strings
>
>
> Key: SPARK-17403
> URL: https://issues.apache.org/jira/browse/SPARK-17403
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0
> Environment: Spark standalone cluster (3 Workers, 47 cores)
> Ubuntu 14
> Java 8
>Reporter: Ruben Hernando
>
> The process creates views from JDBC (SQL server) source and combines them to
> create other views.
> Finally it dumps results via JDBC
> Error:
> {quote}
> # JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build
> 1.8.0_101-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode
> linux-amd64 )
> # Problematic frame:
> # J 4895 C1 org.apache.spark.unsafe.Platform.getLong(Ljava/lang/Object;J)J (9
> bytes) @ 0x7fbb355dfd6c [0x7fbb355dfd60+0xc]
> #
> {quote}
> SQL Query plan (fields truncated):
> {noformat}
> == Parsed Logical Plan ==
> 'Project [*]
> +- 'UnresolvedRelation `COEQ_63`
> == Analyzed Logical Plan ==
> InstanceId: bigint, price: double, ZoneId: int, priceItemId: int, priceId: int
> Project [InstanceId#20236L, price#20237, ZoneId#20239, priceItemId#20242,
> priceId#20244]
> +- SubqueryAlias coeq_63
>+- Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- SubqueryAlias 6__input
> +-
> Relation[_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,SL_Xcl_C#152,SL_Css_Cojs#153L,SL_Config#154,SL_CREATEDON#
> .. 36 more fields] JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables]. ... [...] FROM [sch].[SLTables] [SLTables] JOIN sch.TPSLTables
> TPSLTables ON [TPSLTables].[_TableSL_SID] = [SLTables].[_TableSL_SID] where
> _TP = 24) input)
> == Optimized Logical Plan ==
> Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], [SLTables].[TableSHID],
> [TPSLTables].[SL_ACT_GI_DTE], ... [...] FROM [sch].[SLTables] [SLTables]
> JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,...
> 36 more fields]
> == Physical Plan ==
> *Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryTableScan [_TableSL_SID#143L, SL_RD_ColR_N#189]
>: +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: : +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], ... [...] FROM [sch].[SLTables]
> [SLTables] JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,,...
> 36 more fields]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17403) Fatal Error: Scan cached strings
[
https://issues.apache.org/jira/browse/SPARK-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468528#comment-15468528
]
Ruben Hernando commented on SPARK-17403:
I'm sorry I can't share the data.
This is a 2 tables join, where each table has near 37 million records, and one
of them about 40 columns. As far as I can see, string columns are identifiers
(many null values)
btw, as problematic frame is "org.apache.spark.unsafe.Platform.getLong",
wouldn't it mean to be a Long field?
> Fatal Error: Scan cached strings
>
>
> Key: SPARK-17403
> URL: https://issues.apache.org/jira/browse/SPARK-17403
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0
> Environment: Spark standalone cluster (3 Workers, 47 cores)
> Ubuntu 14
> Java 8
>Reporter: Ruben Hernando
>
> The process creates views from JDBC (SQL server) source and combines them to
> create other views.
> Finally it dumps results via JDBC
> Error:
> {quote}
> # JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build
> 1.8.0_101-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode
> linux-amd64 )
> # Problematic frame:
> # J 4895 C1 org.apache.spark.unsafe.Platform.getLong(Ljava/lang/Object;J)J (9
> bytes) @ 0x7fbb355dfd6c [0x7fbb355dfd60+0xc]
> #
> {quote}
> SQL Query plan (fields truncated):
> {noformat}
> == Parsed Logical Plan ==
> 'Project [*]
> +- 'UnresolvedRelation `COEQ_63`
> == Analyzed Logical Plan ==
> InstanceId: bigint, price: double, ZoneId: int, priceItemId: int, priceId: int
> Project [InstanceId#20236L, price#20237, ZoneId#20239, priceItemId#20242,
> priceId#20244]
> +- SubqueryAlias coeq_63
>+- Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- SubqueryAlias 6__input
> +-
> Relation[_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,SL_Xcl_C#152,SL_Css_Cojs#153L,SL_Config#154,SL_CREATEDON#
> .. 36 more fields] JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables]. ... [...] FROM [sch].[SLTables] [SLTables] JOIN sch.TPSLTables
> TPSLTables ON [TPSLTables].[_TableSL_SID] = [SLTables].[_TableSL_SID] where
> _TP = 24) input)
> == Optimized Logical Plan ==
> Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], [SLTables].[TableSHID],
> [TPSLTables].[SL_ACT_GI_DTE], ... [...] FROM [sch].[SLTables] [SLTables]
> JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,...
> 36 more fields]
> == Physical Plan ==
> *Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryTableScan [_TableSL_SID#143L, SL_RD_ColR_N#189]
>: +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: : +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], ... [...] FROM [sch].[SLTables]
> [SLTables] JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,,...
> 36 more fields]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-17403) Fatal Error: Scan cached strings
[
https://issues.apache.org/jira/browse/SPARK-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468449#comment-15468449
]
Davies Liu commented on SPARK-17403:
[~rhernando] Could you pull out the string column (SL_RD_ColR_N) and dump it as
parquet file to reproduce the issue here? There could a bug in scanning cached
strings (depends on the data).
> Fatal Error: Scan cached strings
>
>
> Key: SPARK-17403
> URL: https://issues.apache.org/jira/browse/SPARK-17403
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.0.0
> Environment: Spark standalone cluster (3 Workers, 47 cores)
> Ubuntu 14
> Java 8
>Reporter: Ruben Hernando
>
> The process creates views from JDBC (SQL server) source and combines them to
> create other views.
> Finally it dumps results via JDBC
> Error:
> {quote}
> # JRE version: Java(TM) SE Runtime Environment (8.0_101-b13) (build
> 1.8.0_101-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.101-b13 mixed mode
> linux-amd64 )
> # Problematic frame:
> # J 4895 C1 org.apache.spark.unsafe.Platform.getLong(Ljava/lang/Object;J)J (9
> bytes) @ 0x7fbb355dfd6c [0x7fbb355dfd60+0xc]
> #
> {quote}
> SQL Query plan (fields truncated):
> {noformat}
> == Parsed Logical Plan ==
> 'Project [*]
> +- 'UnresolvedRelation `COEQ_63`
> == Analyzed Logical Plan ==
> InstanceId: bigint, price: double, ZoneId: int, priceItemId: int, priceId: int
> Project [InstanceId#20236L, price#20237, ZoneId#20239, priceItemId#20242,
> priceId#20244]
> +- SubqueryAlias coeq_63
>+- Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- SubqueryAlias 6__input
> +-
> Relation[_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,SL_Xcl_C#152,SL_Css_Cojs#153L,SL_Config#154,SL_CREATEDON#
> .. 36 more fields] JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables]. ... [...] FROM [sch].[SLTables] [SLTables] JOIN sch.TPSLTables
> TPSLTables ON [TPSLTables].[_TableSL_SID] = [SLTables].[_TableSL_SID] where
> _TP = 24) input)
> == Optimized Logical Plan ==
> Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], [SLTables].[TableSHID],
> [TPSLTables].[SL_ACT_GI_DTE], ... [...] FROM [sch].[SLTables] [SLTables]
> JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,TableP_DCID#148,TableSHID#149,SL_ACT_GI_DTE#150,SL_Xcl_C#151,...
> 36 more fields]
> == Physical Plan ==
> *Project [_TableSL_SID#143L AS InstanceId#20236L, SL_RD_ColR_N#189 AS
> price#20237, 24 AS ZoneId#20239, 6 AS priceItemId#20242, 63 AS priceId#20244]
> +- InMemoryTableScan [_TableSL_SID#143L, SL_RD_ColR_N#189]
>: +- InMemoryRelation [_TableSL_SID#143L, _TableP_DC_SID#144L,
> _TableSH_SID#145L, ID#146, Name#147, ... 36 more fields], true, 1,
> StorageLevel(disk, memory, deserialized, 1 replicas)
>: : +- *Scan JDBCRelation((select [SLTables].[_TableSL_SID],
> [SLTables].[_TableP_DC_SID], [SLTables].[_TableSH_SID], [SLTables].[ID],
> [SLTables].[Name], [SLTables].[TableP_DCID], ... [...] FROM [sch].[SLTables]
> [SLTables] JOIN sch.TPSLTables TPSLTables ON [TPSLTables].[_TableSL_SID] =
> [SLTables].[_TableSL_SID] where _TP = 24) input)
> [_TableSL_SID#143L,_TableP_DC_SID#144L,_TableSH_SID#145L,ID#146,Name#147,,...
> 36 more fields]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
