[jira] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails
[ https://issues.apache.org/jira/browse/SPARK-43106 ] jeanlyn deleted comment on SPARK-43106: - was (Author: jeanlyn): I think we also encountered similar problems, we circumvent this problem by using parameters *spark.sql.hive.convertInsertingPartitionedTable=false* > Data lost from the table if the INSERT OVERWRITE query fails > > > Key: SPARK-43106 > URL: https://issues.apache.org/jira/browse/SPARK-43106 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0 >Reporter: Vaibhav Beriwala >Priority: Major > > When we run an INSERT OVERWRITE query for an unpartitioned table on Spark-3, > Spark has the following behavior: > 1) It will first clean up all the data from the actual table path. > 2) It will then launch a job that performs the actual insert. > > There are 2 major issues with this approach: > 1) If the insert job launched in step 2 above fails for any reason, the data > from the original table is lost. > 2) If the insert job in step 2 above takes a huge time to complete, then > table data is unavailable to other readers for the entire duration the job > takes. > This behavior is the same even for the partitioned tables when using static > partitioning. For dynamic partitioning, we do not delete the table data > before the job launch. > > Is there a reason as to why we perform this delete before the job launch and > not as part of the Job commit operation? This issue is not there with Hive - > where the data is cleaned up as part of the Job commit operation probably. As > part of SPARK-19183, we did add a new hook in the commit protocol for this > exact same purpose, but seems like its default behavior is still to delete > the table data before the job launch. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-43106) Data lost from the table if the INSERT OVERWRITE query fails
[ https://issues.apache.org/jira/browse/SPARK-43106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790135#comment-17790135 ] jeanlyn commented on SPARK-43106: - I think we also encountered similar problems, we circumvent this problem by using parameters *spark.sql.hive.convertInsertingPartitionedTable=false* > Data lost from the table if the INSERT OVERWRITE query fails > > > Key: SPARK-43106 > URL: https://issues.apache.org/jira/browse/SPARK-43106 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.4.0 >Reporter: Vaibhav Beriwala >Priority: Major > > When we run an INSERT OVERWRITE query for an unpartitioned table on Spark-3, > Spark has the following behavior: > 1) It will first clean up all the data from the actual table path. > 2) It will then launch a job that performs the actual insert. > > There are 2 major issues with this approach: > 1) If the insert job launched in step 2 above fails for any reason, the data > from the original table is lost. > 2) If the insert job in step 2 above takes a huge time to complete, then > table data is unavailable to other readers for the entire duration the job > takes. > This behavior is the same even for the partitioned tables when using static > partitioning. For dynamic partitioning, we do not delete the table data > before the job launch. > > Is there a reason as to why we perform this delete before the job launch and > not as part of the Job commit operation? This issue is not there with Hive - > where the data is cleaned up as part of the Job commit operation probably. As > part of SPARK-19183, we did add a new hook in the commit protocol for this > exact same purpose, but seems like its default behavior is still to delete > the table data before the job launch. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35635) concurrent insert statements from multiple beeline fail with job aborted exception
[ https://issues.apache.org/jira/browse/SPARK-35635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741309#comment-17741309 ] jeanlyn commented on SPARK-35635: - When the tasks are running concurrently, the "_temporary" will be attempted to be deleted multiple times, which may result in job failure. Is it more appropriate to reopen this issue? [~gurwls223] > concurrent insert statements from multiple beeline fail with job aborted > exception > -- > > Key: SPARK-35635 > URL: https://issues.apache.org/jira/browse/SPARK-35635 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.1 > Environment: Spark 3.1.1 >Reporter: Chetan Bhat >Priority: Minor > > Create tables - > CREATE TABLE J1_TBL ( > i integer, > j integer, > t string > ) USING parquet; > CREATE TABLE J2_TBL ( > i integer, > k integer > ) USING parquet; > From 4 concurrent beeline sessions execute the insert into select queries - > INSERT INTO J1_TBL VALUES (1, 4, 'one'); > INSERT INTO J1_TBL VALUES (2, 3, 'two'); > INSERT INTO J1_TBL VALUES (3, 2, 'three'); > INSERT INTO J1_TBL VALUES (4, 1, 'four'); > INSERT INTO J1_TBL VALUES (5, 0, 'five'); > INSERT INTO J1_TBL VALUES (6, 6, 'six'); > INSERT INTO J1_TBL VALUES (7, 7, 'seven'); > INSERT INTO J1_TBL VALUES (8, 8, 'eight'); > INSERT INTO J1_TBL VALUES (0, NULL, 'zero'); > INSERT INTO J1_TBL VALUES (NULL, NULL, 'null'); > INSERT INTO J1_TBL VALUES (NULL, 0, 'zero'); > INSERT INTO J2_TBL VALUES (1, -1); > INSERT INTO J2_TBL VALUES (2, 2); > INSERT INTO J2_TBL VALUES (3, -3); > INSERT INTO J2_TBL VALUES (2, 4); > INSERT INTO J2_TBL VALUES (5, -5); > INSERT INTO J2_TBL VALUES (5, -5); > INSERT INTO J2_TBL VALUES (0, NULL); > INSERT INTO J2_TBL VALUES (NULL, NULL); > INSERT INTO J2_TBL VALUES (NULL, 0); > > Issue : concurrent insert statements from multiple beeline fail with job > aborted exception. > 0: jdbc:hive2://10.19.89.222:23040/> INSERT INTO J1_TBL VALUES (8, 8, > 'eight'); > Error: org.apache.hive.service.cli.HiveSQLException: Error running query: > org.apache.spark.SparkException: Job aborted. > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:366) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3$$Lambda$1781/750578465.apply$mcV$sp(Unknown > Source) > at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > at > org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78) > at > org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:45) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:263) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:258) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:272) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.spark.SparkException: Job aborted. > at > org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188) > at > org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:109) > at > org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:107) > at > org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:121) > at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228) > at org.apache.spa
[jira] [Commented] (SPARK-35635) concurrent insert statements from multiple beeline fail with job aborted exception
[ https://issues.apache.org/jira/browse/SPARK-35635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741296#comment-17741296 ] jeanlyn commented on SPARK-35635: - We encounter the same issue when concurrent writing in deference partition on same table. > concurrent insert statements from multiple beeline fail with job aborted > exception > -- > > Key: SPARK-35635 > URL: https://issues.apache.org/jira/browse/SPARK-35635 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.1 > Environment: Spark 3.1.1 >Reporter: Chetan Bhat >Priority: Minor > > Create tables - > CREATE TABLE J1_TBL ( > i integer, > j integer, > t string > ) USING parquet; > CREATE TABLE J2_TBL ( > i integer, > k integer > ) USING parquet; > From 4 concurrent beeline sessions execute the insert into select queries - > INSERT INTO J1_TBL VALUES (1, 4, 'one'); > INSERT INTO J1_TBL VALUES (2, 3, 'two'); > INSERT INTO J1_TBL VALUES (3, 2, 'three'); > INSERT INTO J1_TBL VALUES (4, 1, 'four'); > INSERT INTO J1_TBL VALUES (5, 0, 'five'); > INSERT INTO J1_TBL VALUES (6, 6, 'six'); > INSERT INTO J1_TBL VALUES (7, 7, 'seven'); > INSERT INTO J1_TBL VALUES (8, 8, 'eight'); > INSERT INTO J1_TBL VALUES (0, NULL, 'zero'); > INSERT INTO J1_TBL VALUES (NULL, NULL, 'null'); > INSERT INTO J1_TBL VALUES (NULL, 0, 'zero'); > INSERT INTO J2_TBL VALUES (1, -1); > INSERT INTO J2_TBL VALUES (2, 2); > INSERT INTO J2_TBL VALUES (3, -3); > INSERT INTO J2_TBL VALUES (2, 4); > INSERT INTO J2_TBL VALUES (5, -5); > INSERT INTO J2_TBL VALUES (5, -5); > INSERT INTO J2_TBL VALUES (0, NULL); > INSERT INTO J2_TBL VALUES (NULL, NULL); > INSERT INTO J2_TBL VALUES (NULL, 0); > > Issue : concurrent insert statements from multiple beeline fail with job > aborted exception. > 0: jdbc:hive2://10.19.89.222:23040/> INSERT INTO J1_TBL VALUES (8, 8, > 'eight'); > Error: org.apache.hive.service.cli.HiveSQLException: Error running query: > org.apache.spark.SparkException: Job aborted. > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:366) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:263) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3$$Lambda$1781/750578465.apply$mcV$sp(Unknown > Source) > at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) > at > org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:78) > at > org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:62) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:45) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:263) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:258) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) > at > org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:272) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.spark.SparkException: Job aborted. > at > org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:231) > at > org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:188) > at > org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:109) > at > org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:107) > at > org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:121) > at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228) > at org.apache.spark.sql.Dataset$$Lambda$1650/1168893915.apply(Unknown Source) > at org.apache.spark.sql.Dataset.$anonfun$with
[jira] [Commented] (SPARK-38230) InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions in most cases
[ https://issues.apache.org/jira/browse/SPARK-38230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735519#comment-17735519 ] jeanlyn commented on SPARK-38230: - We found Hive metastore crash frequently after upgrade Spark from 2.4.7 to 3.3.2. After investigation, I found `InsertIntoHadoopFsRelationCommand` will pull all partitions when using dynamicPartitionOverwrite, and i find this issue after solves the problem by using generate paths to get partitions to get partitions in our environment. So, I have submitted a new pull request, hoping to help you. > InsertIntoHadoopFsRelationCommand unnecessarily fetches details of partitions > in most cases > --- > > Key: SPARK-38230 > URL: https://issues.apache.org/jira/browse/SPARK-38230 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.0.2, 3.3.0, 3.4.0, 3.5.0 >Reporter: Coal Chan >Priority: Major > > In > `org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand`, > `sparkSession.sessionState.catalog.listPartitions` will call method > `org.apache.hadoop.hive.metastore.listPartitionsPsWithAuth` of hive metastore > client, this method will produce multiple queries per partition on hive > metastore db. So when you insert into a table which has too many > partitions(ie: 10k), it will produce too many queries on hive metastore > db(ie: n * 10k = 10nk), it puts a lot of strain on the database. > In fact, it calls method `listPartitions` in order to get locations of > partitions and get `customPartitionLocations`. But in most cases, we do not > have custom partitions, we can just get partition names, so we can call > method listPartitionNames. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14243) updatedBlockStatuses does not update correctly when removing blocks
[ https://issues.apache.org/jira/browse/SPARK-14243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15216229#comment-15216229 ] jeanlyn commented on SPARK-14243: - [~andrewor14] Let me know if the descriptions does not detail enough. Also, I will try to fix it these day. :-) > updatedBlockStatuses does not update correctly when removing blocks > --- > > Key: SPARK-14243 > URL: https://issues.apache.org/jira/browse/SPARK-14243 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.2, 1.6.1 >Reporter: jeanlyn > > Currently, *updatedBlockStatuses* of *TaskMetrics* does not update correctly > when removing blocks in *BlockManager.removeBlock* and the method invoke > *removeBlock*. See: > branch-1.6:https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1108 > branch-1.5:https://github.com/apache/spark/blob/branch-1.5/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1101 > We should make sure *updatedBlockStatuses* update correctly when: > * Block removed from BlockManager > * Block dropped from memory to disk > * Block added to BlockManager -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-14243) updatedBlockStatuses does not update correctly when removing blocks
[ https://issues.apache.org/jira/browse/SPARK-14243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-14243: Summary: updatedBlockStatuses does not update correctly when removing blocks (was: updatedBlockStatuses does not update correctly ) > updatedBlockStatuses does not update correctly when removing blocks > --- > > Key: SPARK-14243 > URL: https://issues.apache.org/jira/browse/SPARK-14243 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.2, 1.6.1 >Reporter: jeanlyn > > Currently, *updatedBlockStatuses* of *TaskMetrics* does not update correctly > when removing blocks in *BlockManager.removeBlock* and the method invoke > *removeBlock*. See: > branch-1.6:https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1108 > branch-1.5:https://github.com/apache/spark/blob/branch-1.5/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1101 > We should make sure *updatedBlockStatuses* update correctly when: > * Block removed from BlockManager > * Block dropped from memory to disk > * Block added to BlockManager -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-14243) updatedBlockStatuses does not update correctly
jeanlyn created SPARK-14243: --- Summary: updatedBlockStatuses does not update correctly Key: SPARK-14243 URL: https://issues.apache.org/jira/browse/SPARK-14243 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.6.1, 1.5.2 Reporter: jeanlyn Currently, *updatedBlockStatuses* of *TaskMetrics* does not update correctly when removing blocks in *BlockManager.removeBlock* and the method invoke *removeBlock*. See: branch-1.6:https://github.com/apache/spark/blob/branch-1.6/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1108 branch-1.5:https://github.com/apache/spark/blob/branch-1.5/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1101 We should make sure *updatedBlockStatuses* update correctly when: * Block removed from BlockManager * Block dropped from memory to disk * Block added to BlockManager -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13845) BlockStatus and StreamBlockId keep on growing result driver OOM
[ https://issues.apache.org/jira/browse/SPARK-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-13845: Summary: BlockStatus and StreamBlockId keep on growing result driver OOM (was: Driver OOM after few days when running streaming) > BlockStatus and StreamBlockId keep on growing result driver OOM > --- > > Key: SPARK-13845 > URL: https://issues.apache.org/jira/browse/SPARK-13845 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.2, 1.6.1 >Reporter: jeanlyn > > We have a streaming job using *FlumePollInputStream* always driver OOM after > few days, here is some driver heap dump before OOM > {noformat} > num #instances #bytes class name > -- >1: 13845916 553836640 org.apache.spark.storage.BlockStatus >2: 14020324 336487776 org.apache.spark.storage.StreamBlockId >3: 13883881 333213144 scala.collection.mutable.DefaultEntry >4: 8907 89043952 [Lscala.collection.mutable.HashEntry; >5: 62360 65107352 [B >6:163368 24453904 [Ljava.lang.Object; >7:293651 20342664 [C > ... > {noformat} > *BlockStatus* and *StreamBlockId* keep on growing, and the driver OOM in the > end. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-13845) Driver OOM after few days when running streaming
jeanlyn created SPARK-13845: --- Summary: Driver OOM after few days when running streaming Key: SPARK-13845 URL: https://issues.apache.org/jira/browse/SPARK-13845 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.6.1, 1.5.2 Reporter: jeanlyn We have a streaming job using *FlumePollInputStream* always driver OOM after few days, here is some driver heap dump before OOM {noformat} num #instances #bytes class name -- 1: 13845916 553836640 org.apache.spark.storage.BlockStatus 2: 14020324 336487776 org.apache.spark.storage.StreamBlockId 3: 13883881 333213144 scala.collection.mutable.DefaultEntry 4: 8907 89043952 [Lscala.collection.mutable.HashEntry; 5: 62360 65107352 [B 6:163368 24453904 [Ljava.lang.Object; 7:293651 20342664 [C ... {noformat} *BlockStatus* and *StreamBlockId* keep on growing, and the driver OOM in the end. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-13586) add config to skip generate down time batch when restart StreamingContext
[ https://issues.apache.org/jira/browse/SPARK-13586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn closed SPARK-13586. --- Resolution: Invalid > add config to skip generate down time batch when restart StreamingContext > - > > Key: SPARK-13586 > URL: https://issues.apache.org/jira/browse/SPARK-13586 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: jeanlyn >Priority: Minor > > If we restart streaming, which using checkpoint and has stopped for hours, it > will generate a lot of batch to the queue, and it need to take a while to > handle this batches. So i propose to add a config to control whether generate > down time batch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13586) add config to skip generate down time batch when restart StreamingContext
[ https://issues.apache.org/jira/browse/SPARK-13586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-13586: Priority: Minor (was: Major) > add config to skip generate down time batch when restart StreamingContext > - > > Key: SPARK-13586 > URL: https://issues.apache.org/jira/browse/SPARK-13586 > Project: Spark > Issue Type: Improvement > Components: Streaming >Affects Versions: 1.6.0 >Reporter: jeanlyn >Priority: Minor > > If we restart streaming, which using checkpoint and has stopped for hours, it > will generate a lot of batch to the queue, and it need to take a while to > handle this batches. So i propose to add a config to control whether generate > down time batch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-13586) add config to skip generate down time batch when restart StreamingContext
jeanlyn created SPARK-13586: --- Summary: add config to skip generate down time batch when restart StreamingContext Key: SPARK-13586 URL: https://issues.apache.org/jira/browse/SPARK-13586 Project: Spark Issue Type: Improvement Components: Streaming Affects Versions: 1.6.0 Reporter: jeanlyn If we restart streaming, which using checkpoint and has stopped for hours, it will generate a lot of batch to the queue, and it need to take a while to handle this batches. So i propose to add a config to control whether generate down time batch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-13356) WebUI missing input informations when recovering from dirver failure
jeanlyn created SPARK-13356: --- Summary: WebUI missing input informations when recovering from dirver failure Key: SPARK-13356 URL: https://issues.apache.org/jira/browse/SPARK-13356 Project: Spark Issue Type: Bug Affects Versions: 1.6.0, 1.5.2, 1.5.1, 1.5.0 Reporter: jeanlyn WebUI missing some input information when streaming recover from checkpoint, it may confuse people the data had lose when recover from failure. For example: !DirectKafkaScreenshot.jpg! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-13356) WebUI missing input informations when recovering from dirver failure
[ https://issues.apache.org/jira/browse/SPARK-13356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-13356: Attachment: DirectKafkaScreenshot.jpg > WebUI missing input informations when recovering from dirver failure > > > Key: SPARK-13356 > URL: https://issues.apache.org/jira/browse/SPARK-13356 > Project: Spark > Issue Type: Bug >Affects Versions: 1.5.0, 1.5.1, 1.5.2, 1.6.0 >Reporter: jeanlyn > Attachments: DirectKafkaScreenshot.jpg > > > WebUI missing some input information when streaming recover from checkpoint, > it may confuse people the data had lose when recover from failure. > For example: > !DirectKafkaScreenshot.jpg! -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8513) _temporary may be left undeleted when a write job committed with FileOutputCommitter fails due to a race condition
[ https://issues.apache.org/jira/browse/SPARK-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699117#comment-14699117 ] jeanlyn edited comment on SPARK-8513 at 8/17/15 7:30 AM: - I think i encountered the problem these day. Our job failed due to {{_temporary}} left, and when using the hive api update partitions will throw exception if it has nested directory. {noformat} 2015-08-12 07:07:07 INFO org.apache.hadoop.hive.ql.metadata.HiveException: checkPaths: hdfs://ns1/tmp/hive-dd_edw/hive_2015-08-12_07-02-20_902_7762418154833191311-1/-ext-1 has nested directoryhdfs://ns1/tmp/hive-dd_edw/hive_2015-08-12_07-02-20_902_7762418154833191311-1/-ext-1/_temporary at org.apache.hadoop.hive.ql.metadata.Hive.checkPaths(Hive.java:2080) at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2270) at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1222) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:233) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:124) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:266) at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1140) at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1140) at org.apache.spark.sql.DataFrame.(DataFrame.scala:147) at org.apache.spark.sql.DataFrame.(DataFrame.scala:130) at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:97) at org.apache.spark.sql.hive.thriftserver.AbstractSparkSQLDriver.run(AbstractSparkSQLDriver.scala:57) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:273) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:442) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:148) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:619) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {noformat} In our case, all of our task had finished except the speculative task. {noformat} 15/08/12 07:07:06 INFO TaskSetManager: Marking task 6 in stage 40.0 (on BJHC-HERA-17163.hadoop.local) as speculatable because it ran more than 33639 ms (speculative task)**15/08/12 07:07:06 INFO TaskSetManager: Starting task 6.1 in stage 40.0 (TID 165, BJHC-HERA-16580.hadoop.local, PROCESS_LOCAL, 1687 bytes)* 15/08/12 07:07:06 INFO BlockManagerInfo: Added broadcast_60_piece0 in memory on BJHC-HERA-16580.hadoop.local:48182 (size: 740.2 KB, free: 2.1 GB) 15/08/12 07:07:06 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to BJHC-HERA-16580.hadoop.local:9208 15/08/12 07:07:07 INFO TaskSetManager: Finished task 6.0 in stage 40.0 (TID 161) in 34449 ms on BJHC-HERA-17163.hadoop.local (10/10) 15/08/12 07:07:07 INFO DAGScheduler: ResultStage 40 (runJob at InsertIntoHiveTable.scala:83) finished in 34.457 s {noformat} However, i can not find any code to cancel the speculative task. So, if we want to fix this issue, do we also need to add the cancel logic(kill the speculative tasks) before making task cancellation synchronous when job finished? was (Author: jeanlyn): I think i encountered the problem these day. Our job failed due to {{_temporary}} left, and when using the hive api update partitions will throw exception if it has nested directory. {noformat} 2015-08-12 07:07:07 INFO org.apache.hadoop.hive.ql.metadata.HiveException: checkPaths: hdfs://ns1/tmp/hive-dd_edw/hive_2015-08-12_07-02-20_902_7762418154833191311-1/-ext-1 has nested directoryhdfs://ns1/tmp/hive-dd_edw/hive_2015-08-12_07-02-20_902_7762418154833191311-1/-ext-1/_temporary at org.apache.hadoop.hive.ql.metadata.Hive.checkPaths(Hive.java:2080) at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2270) at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1222) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:2
[jira] [Commented] (SPARK-8513) _temporary may be left undeleted when a write job committed with FileOutputCommitter fails due to a race condition
[ https://issues.apache.org/jira/browse/SPARK-8513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699117#comment-14699117 ] jeanlyn commented on SPARK-8513: I think i encountered the problem these day. Our job failed due to {{_temporary}} left, and when using the hive api update partitions will throw exception if it has nested directory. {noformat} 2015-08-12 07:07:07 INFO org.apache.hadoop.hive.ql.metadata.HiveException: checkPaths: hdfs://ns1/tmp/hive-dd_edw/hive_2015-08-12_07-02-20_902_7762418154833191311-1/-ext-1 has nested directoryhdfs://ns1/tmp/hive-dd_edw/hive_2015-08-12_07-02-20_902_7762418154833191311-1/-ext-1/_temporary at org.apache.hadoop.hive.ql.metadata.Hive.checkPaths(Hive.java:2080) at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2270) at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1222) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:233) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:124) at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.execute(InsertIntoHiveTable.scala:266) at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:1140) at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:1140) at org.apache.spark.sql.DataFrame.(DataFrame.scala:147) at org.apache.spark.sql.DataFrame.(DataFrame.scala:130) at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:97) at org.apache.spark.sql.hive.thriftserver.AbstractSparkSQLDriver.run(AbstractSparkSQLDriver.scala:57) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:273) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:507) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:442) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:148) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:619) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) {noformat} In our case, all of our task had finished except the speculative task. {code} 15/08/12 07:07:06 INFO TaskSetManager: Marking task 6 in stage 40.0 (on BJHC-HERA-17163.hadoop.local) as speculatable because it ran more than 33639 ms (speculative task)**15/08/12 07:07:06 INFO TaskSetManager: Starting task 6.1 in stage 40.0 (TID 165, BJHC-HERA-16580.hadoop.local, PROCESS_LOCAL, 1687 bytes)* 15/08/12 07:07:06 INFO BlockManagerInfo: Added broadcast_60_piece0 in memory on BJHC-HERA-16580.hadoop.local:48182 (size: 740.2 KB, free: 2.1 GB) 15/08/12 07:07:06 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to BJHC-HERA-16580.hadoop.local:9208 15/08/12 07:07:07 INFO TaskSetManager: Finished task 6.0 in stage 40.0 (TID 161) in 34449 ms on BJHC-HERA-17163.hadoop.local (10/10) 15/08/12 07:07:07 INFO DAGScheduler: ResultStage 40 (runJob at InsertIntoHiveTable.scala:83) finished in 34.457 s {code} However, i can not find any code to cancel the speculative task. So, if we want to fix this issue, do we also need to add the cancel logic(kill the speculative tasks) before making task cancellation synchronous when job finished? > _temporary may be left undeleted when a write job committed with > FileOutputCommitter fails due to a race condition > -- > > Key: SPARK-8513 > URL: https://issues.apache.org/jira/browse/SPARK-8513 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 1.2.2, 1.3.1, 1.4.0 >Reporter: Cheng Lian > > To reproduce this issue, we need a node with relatively more cores, say 32 > (e.g., Spark Jenkins builder is a good candidate). With such a node, the > following code should be relatively easy to reproduce this issue: > {code} > sqlContext.range(0, 10).repartition(32).select('id / > 0).write.mode("overwrite").parquet("file:///tmp/foo") > {code} > You may observe similar log lines
[jira] [Commented] (SPARK-6392) [SQL]class not found exception thows when `add jar` use spark cli
[ https://issues.apache.org/jira/browse/SPARK-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663264#comment-14663264 ] jeanlyn commented on SPARK-6392: I thought it fixed in my case, do you have more descriptions about the issue, or how can we reproduce it? > [SQL]class not found exception thows when `add jar` use spark cli > -- > > Key: SPARK-6392 > URL: https://issues.apache.org/jira/browse/SPARK-6392 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0 >Reporter: jeanlyn >Priority: Minor > > When we use spark cli to add jar dynamic,we will get the > *java.lang.ClassNotFoundException* when we use the class of jar to create > udf.For example: > {noformat} > spark-sql> add jar /home/jeanlyn/hello.jar; > spark-sql>create temporary function hello as 'hello'; > spark-sql>select hello(name) from person; > Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most > recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): > java.lang.ClassNotFoundException: hello > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-9591) Job failed for exception during getting Broadcast variable
jeanlyn created SPARK-9591: -- Summary: Job failed for exception during getting Broadcast variable Key: SPARK-9591 URL: https://issues.apache.org/jira/browse/SPARK-9591 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.4.1, 1.4.0, 1.3.1 Reporter: jeanlyn Job might failed for exception throw when we getting the broadcast variable especially using dynamic resource allocate. driver log {noformat} 2015-07-21 05:36:31 INFO 15/07/21 05:36:31 WARN TaskSetManager: Lost task 496.1 in stage 19.0 (TID 1715, XX): java.io.IOException: Failed to connect to X:27072 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:191) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156) at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:78) at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140) at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43) at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.ConnectException: Connection refused: xx:27072 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:208) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:287) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ... 1 more 15/07/21 05:36:32 WARN TaskSetManager: Lost task 496.2 in stage 19.0 (TID 1744, x): java.io.IOException: Failed to connect to /:34070 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:191) at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:156) at org.apache.spark.network.netty.NettyBlockTransferService$$anon$1.createAndStart(NettyBlockTransferService.scala:78) at org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140) at org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43) at org.apache.spark.network.shuffle.RetryingBlockFetcher$1.run(RetryingBlockFetcher.java:170) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.net.ConnectException: Connection refused: xxx:34070 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:208) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:287) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ... 1 more org.apache.spark.SparkException: Job aborted due to stage failure: Task 496 in stage 19.0 failed 4 times {noformat} executor log {noformat} 15/07/21 05:36:17 ERROR shuffle.RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks java.io.IOException: Failed to connect to xxx at org.apache.spark.network.client.TransportClientFactory.createClient(Tr
[jira] [Closed] (SPARK-6392) [SQL]class not found exception thows when `add jar` use spark cli
[ https://issues.apache.org/jira/browse/SPARK-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn closed SPARK-6392. -- Resolution: Fixed > [SQL]class not found exception thows when `add jar` use spark cli > -- > > Key: SPARK-6392 > URL: https://issues.apache.org/jira/browse/SPARK-6392 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0 >Reporter: jeanlyn >Priority: Minor > > When we use spark cli to add jar dynamic,we will get the > *java.lang.ClassNotFoundException* when we use the class of jar to create > udf.For example: > {noformat} > spark-sql> add jar /home/jeanlyn/hello.jar; > spark-sql>create temporary function hello as 'hello'; > spark-sql>select hello(name) from person; > Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most > recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): > java.lang.ClassNotFoundException: hello > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-6392) [SQL]class not found exception thows when `add jar` use spark cli
[ https://issues.apache.org/jira/browse/SPARK-6392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14633033#comment-14633033 ] jeanlyn commented on SPARK-6392: I think this issue is fixed by https://github.com/apache/spark/pull/4586. > [SQL]class not found exception thows when `add jar` use spark cli > -- > > Key: SPARK-6392 > URL: https://issues.apache.org/jira/browse/SPARK-6392 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0 >Reporter: jeanlyn >Priority: Minor > > When we use spark cli to add jar dynamic,we will get the > *java.lang.ClassNotFoundException* when we use the class of jar to create > udf.For example: > {noformat} > spark-sql> add jar /home/jeanlyn/hello.jar; > spark-sql>create temporary function hello as 'hello'; > spark-sql>select hello(name) from person; > Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most > recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): > java.lang.ClassNotFoundException: hello > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-8379) LeaseExpiredException when using dynamic partition with speculative execution
jeanlyn created SPARK-8379: -- Summary: LeaseExpiredException when using dynamic partition with speculative execution Key: SPARK-8379 URL: https://issues.apache.org/jira/browse/SPARK-8379 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.4.0, 1.3.1, 1.3.0 Reporter: jeanlyn when inserting to table using dynamic partitions with *spark.speculation=true* and there is a skew data of some partitions trigger the speculative tasks ,it will throws the exception like {code} org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): Lease mismatch on /tmp/hive-jeanlyn/hive_2015-06-15_15-20-44_734_8801220787219172413-1/-ext-1/ds=2015-06-15/type=2/part-00301.lzo owned by DFSClient_attempt_201506031520_0011_m_000189_0_-1513487243_53 but is accessed by DFSClient_attempt_201506031520_0011_m_42_0_-1275047721_57 {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8020) Spark SQL conf in spark-defaults.conf make metadataHive get constructed too early
[ https://issues.apache.org/jira/browse/SPARK-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570303#comment-14570303 ] jeanlyn edited comment on SPARK-8020 at 6/3/15 5:38 AM: I had tried to put the settings back to *spark-defaults.conf* just now,and i builded spark with rc4.I still got the same *ClassNotFoundException* excption as i mentioned about was (Author: jeanlyn): I had tried to put the settings back to *spark-defaults.conf* just now,and i builded spark with rc4.I still got the same * ClassNotFoundException* excption as i mentioned about > Spark SQL conf in spark-defaults.conf make metadataHive get constructed too > early > - > > Key: SPARK-8020 > URL: https://issues.apache.org/jira/browse/SPARK-8020 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.4.0 >Reporter: Yin Huai >Assignee: Yin Huai >Priority: Blocker > Fix For: 1.4.0 > > > To correctly construct a {{metadataHive}} object, we need two settings, > {{spark.sql.hive.metastore.version}} and {{spark.sql.hive.metastore.jars}}. > If users want to use Hive 0.12's metastore, they need to set > {{spark.sql.hive.metastore.version}} to {{0.12.0}} and set > {{spark.sql.hive.metastore.jars}} to {{maven}} or a classpath containing Hive > and Hadoop's jars. However, any spark sql setting in the > {{spark-defaults.conf}} will trigger the construction of {{metadataHive}} and > cause Spark SQL connect to the wrong metastore (e.g. connect to the local > derby metastore instead of a remove mysql Hive 0.12 metastore). Also, if > {{spark.sql.hive.metastore.version 0.12.0}} is the first conf set to SQL > conf, we will get > {code} > Exception in thread "main" java.lang.IllegalArgumentException: Builtin jars > can only be used when hive execution version == hive metastore version. > Execution: 0.13.1 != Metastore: 0.12.0. Specify a vaild path to the correct > hive jars using $HIVE_METASTORE_JARS or change > spark.sql.hive.metastore.version to 0.13.1. > at > org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:186) > at > org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:175) > at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:358) > at > org.apache.spark.sql.SQLContext$$anonfun$3.apply(SQLContext.scala:186) > at > org.apache.spark.sql.SQLContext$$anonfun$3.apply(SQLContext.scala:185) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at org.apache.spark.sql.SQLContext.(SQLContext.scala:185) > at org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:71) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:248) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:136) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8020) Spark SQL conf in spark-defaults.conf make metadataHive get constructed too early
[ https://issues.apache.org/jira/browse/SPARK-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570303#comment-14570303 ] jeanlyn edited comment on SPARK-8020 at 6/3/15 5:38 AM: I had tried to put the settings back to *spark-defaults.conf* just now,and i builded spark with rc4.I still got the same * ClassNotFoundException* excption as i mentioned about was (Author: jeanlyn): I had tried to put the settings back to *spark-defaults.conf* just now,and i builded spark with rc4.I still got the same excption as i mentioned about > Spark SQL conf in spark-defaults.conf make metadataHive get constructed too > early > - > > Key: SPARK-8020 > URL: https://issues.apache.org/jira/browse/SPARK-8020 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.4.0 >Reporter: Yin Huai >Assignee: Yin Huai >Priority: Blocker > Fix For: 1.4.0 > > > To correctly construct a {{metadataHive}} object, we need two settings, > {{spark.sql.hive.metastore.version}} and {{spark.sql.hive.metastore.jars}}. > If users want to use Hive 0.12's metastore, they need to set > {{spark.sql.hive.metastore.version}} to {{0.12.0}} and set > {{spark.sql.hive.metastore.jars}} to {{maven}} or a classpath containing Hive > and Hadoop's jars. However, any spark sql setting in the > {{spark-defaults.conf}} will trigger the construction of {{metadataHive}} and > cause Spark SQL connect to the wrong metastore (e.g. connect to the local > derby metastore instead of a remove mysql Hive 0.12 metastore). Also, if > {{spark.sql.hive.metastore.version 0.12.0}} is the first conf set to SQL > conf, we will get > {code} > Exception in thread "main" java.lang.IllegalArgumentException: Builtin jars > can only be used when hive execution version == hive metastore version. > Execution: 0.13.1 != Metastore: 0.12.0. Specify a vaild path to the correct > hive jars using $HIVE_METASTORE_JARS or change > spark.sql.hive.metastore.version to 0.13.1. > at > org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:186) > at > org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:175) > at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:358) > at > org.apache.spark.sql.SQLContext$$anonfun$3.apply(SQLContext.scala:186) > at > org.apache.spark.sql.SQLContext$$anonfun$3.apply(SQLContext.scala:185) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at org.apache.spark.sql.SQLContext.(SQLContext.scala:185) > at org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:71) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:248) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:136) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8020) Spark SQL conf in spark-defaults.conf make metadataHive get constructed too early
[ https://issues.apache.org/jira/browse/SPARK-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570303#comment-14570303 ] jeanlyn commented on SPARK-8020: I had tried to put the settings back to *spark-defaults.conf* just now,and i builded spark with rc4.I still got the same excption as i mentioned about > Spark SQL conf in spark-defaults.conf make metadataHive get constructed too > early > - > > Key: SPARK-8020 > URL: https://issues.apache.org/jira/browse/SPARK-8020 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.4.0 >Reporter: Yin Huai >Assignee: Yin Huai >Priority: Blocker > Fix For: 1.4.0 > > > To correctly construct a {{metadataHive}} object, we need two settings, > {{spark.sql.hive.metastore.version}} and {{spark.sql.hive.metastore.jars}}. > If users want to use Hive 0.12's metastore, they need to set > {{spark.sql.hive.metastore.version}} to {{0.12.0}} and set > {{spark.sql.hive.metastore.jars}} to {{maven}} or a classpath containing Hive > and Hadoop's jars. However, any spark sql setting in the > {{spark-defaults.conf}} will trigger the construction of {{metadataHive}} and > cause Spark SQL connect to the wrong metastore (e.g. connect to the local > derby metastore instead of a remove mysql Hive 0.12 metastore). Also, if > {{spark.sql.hive.metastore.version 0.12.0}} is the first conf set to SQL > conf, we will get > {code} > Exception in thread "main" java.lang.IllegalArgumentException: Builtin jars > can only be used when hive execution version == hive metastore version. > Execution: 0.13.1 != Metastore: 0.12.0. Specify a vaild path to the correct > hive jars using $HIVE_METASTORE_JARS or change > spark.sql.hive.metastore.version to 0.13.1. > at > org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:186) > at > org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:175) > at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:358) > at > org.apache.spark.sql.SQLContext$$anonfun$3.apply(SQLContext.scala:186) > at > org.apache.spark.sql.SQLContext$$anonfun$3.apply(SQLContext.scala:185) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at org.apache.spark.sql.SQLContext.(SQLContext.scala:185) > at org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:71) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:53) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:248) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:136) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8020) Spark SQL in spark-defaults.conf make metadataHive get constructed too early
[ https://issues.apache.org/jira/browse/SPARK-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568427#comment-14568427 ] jeanlyn edited comment on SPARK-8020 at 6/2/15 3:16 AM: [~yhuai],I set *spark.sql.hive.metastore.jars* in spark-defaults.conf i got errors like yours.But when i set *spark.sql.hive.metastore.jars* in *hive-site.xml* i got {code} 5/06/02 10:42:04 INFO storage.BlockManagerMaster: Trying to register BlockManager 15/06/02 10:42:04 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:41416 with 706.6 MB RAM, BlockManagerId(driver, localhost, 41416) 15/06/02 10:42:04 INFO storage.BlockManagerMaster: Registered BlockManager SET spark.sql.hive.metastore.version=0.12.0 15/06/02 10:42:04 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.u ris if you are connecting to a remote metastore. 15/06/02 10:42:04 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead 15/06/02 10:42:04 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using maven. Ivy Default Cache set to: /home/dd_edw/.ivy2/cache The jars for the packages stored in: /home/dd_edw/.ivy2/jars http://www.datanucleus.org/downloads/maven2 added as a remote repository with the name: repo-1 :: loading settings :: url = jar:file:/data0/spark-1.3.0-bin-2.2.0/lib/spark-assembly-1.4.0-SNAPSHOT-hadoop2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.apache.hive#hive-metastore added as a dependency org.apache.hive#hive-exec added as a dependency org.apache.hive#hive-common added as a dependency org.apache.hive#hive-serde added as a dependency com.google.guava#guava added as a dependency org.apache.hadoop#hadoop-client added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.apache.hive#hive-metastore;0.12.0 in central found org.antlr#antlr;3.4 in central found org.antlr#antlr-runtime;3.4 in central xception in thread "main" java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: fi le:/tmp/hive3795822184995995241vv12/aopalliance_aopalliance-1.0.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hive_hive-exec-0.12.0.jar, file:/tmp/hive3795822184995995 241vv12/org.apache.thrift_libfb303-0.9.0.jar, file:/tmp/hive3795822184995995241vv12/commons-digester_commons-digester-1.8.jar, file:/tmp/hive3795822184995995241vv12/com.sun.je rsey_jersey-client-1.9.jar, file:/tmp/hive3795822184995995241vv12/org.apache.httpcomponents_httpclient-4.2.5.jar, file:/tmp/hive3795822184995995241vv12/org.antlr_stringtemplat e-3.2.1.jar, file:/tmp/hive3795822184995995241vv12/commons-logging_commons-logging-1.1.3.jar, file:/tmp/hive3795822184995995241vv12/org.antlr_antlr-runtime-3.4.jar, file:/tmp/ hive3795822184995995241vv12/org.mockito_mockito-all-1.8.2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.derby_derby-10.4.2.0.jar, file:/tmp/hive3795822184995995241vv12 /antlr_antlr-2.7.7.jar, file:/tmp/hive3795822184995995241vv12/commons-net_commons-net-3.1.jar, file:/tmp/hive3795822184995995241vv12/org.slf4j_slf4j-log4j12-1.7.5.jar, file:/t mp/hive3795822184995995241vv12/junit_junit-3.8.1.jar, file:/tmp/hive3795822184995995241vv12/org.codehaus.jackson_jackson-jaxrs-1.8.8.jar, file:/tmp/hive3795822184995995241vv12 /commons-cli_commons-cli-1.2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hive_hive-serde-0.12.0.jar, file:/tmp/hive3795822184995995241vv12/org.codehaus.jettison_jett ison-1.1.jar, file:/tmp/hive3795822184995995241vv12/javax.xml.stream_stax-api-1.0-2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.avro_avro-1.7.4.jar, file:/tmp/hive37 95822184995995241vv12/org.apache.hadoop_hadoop-mapreduce-client-app-2.4.0.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hadoop_hadoop-mapreduce-client-common-2.4.0.jar , file:/tmp/hive3795822184995995241vv12/org.codehaus.jackson_jackson-xc-1.8.8.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hadoop_hadoop-annotations-2.4.0.jar, file:/ tmp/hive3795822184995995241vv12/org.mortbay.jetty_jetty-util-6.1.26.jar, file:/tmp/hive3795822184995995241vv12/org.apache.commons_commons-math3-3.1.1.jar, file:/tmp/hive379582 2184995995241vv12/javax.transaction_jta-1.1.jar, file:/tmp/hive3795822184995995241vv12/commons-httpclient_commons-httpclient-3.1.jar, file:/tmp/hive3795822184995995241vv12/xml enc_xmlenc-0.52.jar, file:/tmp/hive3795822184995995241vv12/org.sonatype.sisu.inject_cglib-2.2.1-v20090111.jar, file:/tmp/hive3795822184995995241vv12/com.google.code.findbugs_j sr305-1.3.9.jar, file:/tmp/hive3795822184995995241vv12/commons-codec_commons-codec-1.4.jar, fi
[jira] [Comment Edited] (SPARK-8020) Spark SQL in spark-defaults.conf make metadataHive get constructed too early
[ https://issues.apache.org/jira/browse/SPARK-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568427#comment-14568427 ] jeanlyn edited comment on SPARK-8020 at 6/2/15 2:48 AM: [~yhuai],I set *spark.sql.hive.metastore.jars* in spark-defaults.conf i got errors like yours.But when i set *spark.sql.hive.metastore.jars* in *hive-set.xml* i got {code} 5/06/02 10:42:04 INFO storage.BlockManagerMaster: Trying to register BlockManager 15/06/02 10:42:04 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:41416 with 706.6 MB RAM, BlockManagerId(driver, localhost, 41416) 15/06/02 10:42:04 INFO storage.BlockManagerMaster: Registered BlockManager SET spark.sql.hive.metastore.version=0.12.0 15/06/02 10:42:04 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.u ris if you are connecting to a remote metastore. 15/06/02 10:42:04 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead 15/06/02 10:42:04 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using maven. Ivy Default Cache set to: /home/dd_edw/.ivy2/cache The jars for the packages stored in: /home/dd_edw/.ivy2/jars http://www.datanucleus.org/downloads/maven2 added as a remote repository with the name: repo-1 :: loading settings :: url = jar:file:/data0/spark-1.3.0-bin-2.2.0/lib/spark-assembly-1.4.0-SNAPSHOT-hadoop2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.apache.hive#hive-metastore added as a dependency org.apache.hive#hive-exec added as a dependency org.apache.hive#hive-common added as a dependency org.apache.hive#hive-serde added as a dependency com.google.guava#guava added as a dependency org.apache.hadoop#hadoop-client added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.apache.hive#hive-metastore;0.12.0 in central found org.antlr#antlr;3.4 in central found org.antlr#antlr-runtime;3.4 in central xception in thread "main" java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: fi le:/tmp/hive3795822184995995241vv12/aopalliance_aopalliance-1.0.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hive_hive-exec-0.12.0.jar, file:/tmp/hive3795822184995995 241vv12/org.apache.thrift_libfb303-0.9.0.jar, file:/tmp/hive3795822184995995241vv12/commons-digester_commons-digester-1.8.jar, file:/tmp/hive3795822184995995241vv12/com.sun.je rsey_jersey-client-1.9.jar, file:/tmp/hive3795822184995995241vv12/org.apache.httpcomponents_httpclient-4.2.5.jar, file:/tmp/hive3795822184995995241vv12/org.antlr_stringtemplat e-3.2.1.jar, file:/tmp/hive3795822184995995241vv12/commons-logging_commons-logging-1.1.3.jar, file:/tmp/hive3795822184995995241vv12/org.antlr_antlr-runtime-3.4.jar, file:/tmp/ hive3795822184995995241vv12/org.mockito_mockito-all-1.8.2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.derby_derby-10.4.2.0.jar, file:/tmp/hive3795822184995995241vv12 /antlr_antlr-2.7.7.jar, file:/tmp/hive3795822184995995241vv12/commons-net_commons-net-3.1.jar, file:/tmp/hive3795822184995995241vv12/org.slf4j_slf4j-log4j12-1.7.5.jar, file:/t mp/hive3795822184995995241vv12/junit_junit-3.8.1.jar, file:/tmp/hive3795822184995995241vv12/org.codehaus.jackson_jackson-jaxrs-1.8.8.jar, file:/tmp/hive3795822184995995241vv12 /commons-cli_commons-cli-1.2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hive_hive-serde-0.12.0.jar, file:/tmp/hive3795822184995995241vv12/org.codehaus.jettison_jett ison-1.1.jar, file:/tmp/hive3795822184995995241vv12/javax.xml.stream_stax-api-1.0-2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.avro_avro-1.7.4.jar, file:/tmp/hive37 95822184995995241vv12/org.apache.hadoop_hadoop-mapreduce-client-app-2.4.0.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hadoop_hadoop-mapreduce-client-common-2.4.0.jar , file:/tmp/hive3795822184995995241vv12/org.codehaus.jackson_jackson-xc-1.8.8.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hadoop_hadoop-annotations-2.4.0.jar, file:/ tmp/hive3795822184995995241vv12/org.mortbay.jetty_jetty-util-6.1.26.jar, file:/tmp/hive3795822184995995241vv12/org.apache.commons_commons-math3-3.1.1.jar, file:/tmp/hive379582 2184995995241vv12/javax.transaction_jta-1.1.jar, file:/tmp/hive3795822184995995241vv12/commons-httpclient_commons-httpclient-3.1.jar, file:/tmp/hive3795822184995995241vv12/xml enc_xmlenc-0.52.jar, file:/tmp/hive3795822184995995241vv12/org.sonatype.sisu.inject_cglib-2.2.1-v20090111.jar, file:/tmp/hive3795822184995995241vv12/com.google.code.findbugs_j sr305-1.3.9.jar, file:/tmp/hive3795822184995995241vv12/commons-codec_commons-codec-1.4.jar, fil
[jira] [Commented] (SPARK-8020) Spark SQL in spark-defaults.conf make metadataHive get constructed too early
[ https://issues.apache.org/jira/browse/SPARK-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568427#comment-14568427 ] jeanlyn commented on SPARK-8020: [~yhuai],I set *spark.sql.hive.metastore.jars* in spark-defaults.conf i got errors like yours.But when i set *spark.sql.hive.metastore.jars* in *hive-set.xml* i got {code} 5/06/02 10:42:04 INFO storage.BlockManagerMaster: Trying to register BlockManager 15/06/02 10:42:04 INFO storage.BlockManagerMasterEndpoint: Registering block manager localhost:41416 with 706.6 MB RAM, BlockManagerId(driver, localhost, 41416) 15/06/02 10:42:04 INFO storage.BlockManagerMaster: Registered BlockManager SET spark.sql.hive.metastore.version=0.12.0 15/06/02 10:42:04 WARN conf.HiveConf: DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.u ris if you are connecting to a remote metastore. 15/06/02 10:42:04 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no longer has any effect. Use hive.hmshandler.retry.* instead 15/06/02 10:42:04 INFO hive.HiveContext: Initializing HiveMetastoreConnection version 0.12.0 using maven. Ivy Default Cache set to: /home/dd_edw/.ivy2/cache The jars for the packages stored in: /home/dd_edw/.ivy2/jars http://www.datanucleus.org/downloads/maven2 added as a remote repository with the name: repo-1 :: loading settings :: url = jar:file:/data0/spark-1.3.0-bin-2.2.0/lib/spark-assembly-1.4.0-SNAPSHOT-hadoop2.2.0.jar!/org/apache/ivy/core/settings/ivysettings.xml org.apache.hive#hive-metastore added as a dependency org.apache.hive#hive-exec added as a dependency org.apache.hive#hive-common added as a dependency org.apache.hive#hive-serde added as a dependency com.google.guava#guava added as a dependency org.apache.hadoop#hadoop-client added as a dependency :: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0 confs: [default] found org.apache.hive#hive-metastore;0.12.0 in central found org.antlr#antlr;3.4 in central found org.antlr#antlr-runtime;3.4 in central xception in thread "main" java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: com/google/common/base/Preconditions when creating Hive client using classpath: fi le:/tmp/hive3795822184995995241vv12/aopalliance_aopalliance-1.0.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hive_hive-exec-0.12.0.jar, file:/tmp/hive3795822184995995 241vv12/org.apache.thrift_libfb303-0.9.0.jar, file:/tmp/hive3795822184995995241vv12/commons-digester_commons-digester-1.8.jar, file:/tmp/hive3795822184995995241vv12/com.sun.je rsey_jersey-client-1.9.jar, file:/tmp/hive3795822184995995241vv12/org.apache.httpcomponents_httpclient-4.2.5.jar, file:/tmp/hive3795822184995995241vv12/org.antlr_stringtemplat e-3.2.1.jar, file:/tmp/hive3795822184995995241vv12/commons-logging_commons-logging-1.1.3.jar, file:/tmp/hive3795822184995995241vv12/org.antlr_antlr-runtime-3.4.jar, file:/tmp/ hive3795822184995995241vv12/org.mockito_mockito-all-1.8.2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.derby_derby-10.4.2.0.jar, file:/tmp/hive3795822184995995241vv12 /antlr_antlr-2.7.7.jar, file:/tmp/hive3795822184995995241vv12/commons-net_commons-net-3.1.jar, file:/tmp/hive3795822184995995241vv12/org.slf4j_slf4j-log4j12-1.7.5.jar, file:/t mp/hive3795822184995995241vv12/junit_junit-3.8.1.jar, file:/tmp/hive3795822184995995241vv12/org.codehaus.jackson_jackson-jaxrs-1.8.8.jar, file:/tmp/hive3795822184995995241vv12 /commons-cli_commons-cli-1.2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hive_hive-serde-0.12.0.jar, file:/tmp/hive3795822184995995241vv12/org.codehaus.jettison_jett ison-1.1.jar, file:/tmp/hive3795822184995995241vv12/javax.xml.stream_stax-api-1.0-2.jar, file:/tmp/hive3795822184995995241vv12/org.apache.avro_avro-1.7.4.jar, file:/tmp/hive37 95822184995995241vv12/org.apache.hadoop_hadoop-mapreduce-client-app-2.4.0.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hadoop_hadoop-mapreduce-client-common-2.4.0.jar , file:/tmp/hive3795822184995995241vv12/org.codehaus.jackson_jackson-xc-1.8.8.jar, file:/tmp/hive3795822184995995241vv12/org.apache.hadoop_hadoop-annotations-2.4.0.jar, file:/ tmp/hive3795822184995995241vv12/org.mortbay.jetty_jetty-util-6.1.26.jar, file:/tmp/hive3795822184995995241vv12/org.apache.commons_commons-math3-3.1.1.jar, file:/tmp/hive379582 2184995995241vv12/javax.transaction_jta-1.1.jar, file:/tmp/hive3795822184995995241vv12/commons-httpclient_commons-httpclient-3.1.jar, file:/tmp/hive3795822184995995241vv12/xml enc_xmlenc-0.52.jar, file:/tmp/hive3795822184995995241vv12/org.sonatype.sisu.inject_cglib-2.2.1-v20090111.jar, file:/tmp/hive3795822184995995241vv12/com.google.code.findbugs_j sr305-1.3.9.jar, file:/tmp/hive3795822184995995241vv12/commons-codec_commons-codec-1.4.jar, file:/tmp/hive3795822184995995241vv12/com.google.g
[jira] [Created] (SPARK-7885) add config to control map aggregation in spark sql
jeanlyn created SPARK-7885: -- Summary: add config to control map aggregation in spark sql Key: SPARK-7885 URL: https://issues.apache.org/jira/browse/SPARK-7885 Project: Spark Issue Type: Improvement Affects Versions: 1.3.1, 1.2.2, 1.2.0 Reporter: jeanlyn For now, *execution.HashAggregation* add the map aggregation in oder to decrease the shuffle data.However,we found gc problem when we use this optimization and finally the executor crash.For example, {noformat} select sale_ord_id as order_id, coalesce(sum(sku_offer_amount),0.0) as sku_offer_amount, coalesce(sum(suit_offer_amount),0.0) as suit_offer_amount, coalesce(sum(flash_gp_offer_amount),0.0) + coalesce(sum(gp_offer_amount),0.0) as gp_offer_amount, coalesce(sum(flash_gp_offer_amount),0.0) as flash_gp_offer_amount, coalesce(sum(full_minus_offer_amount),0.0) as full_rebate_offer_amount, 0.0 as telecom_point_offer_amount, coalesce(sum(coupon_pay_amount),0.0) as dq_and_jq_pay_amount, coalesce(sum(jq_pay_amount),0.0) + coalesce(sum(pop_shop_jq_pay_amount),0.0) + coalesce(sum(lim_cate_jq_pay_amount),0.0) as jq_pay_amount, coalesce(sum(dq_pay_amount),0.0) + coalesce(sum(pop_shop_dq_pay_amount),0.0) + coalesce(sum(lim_cate_dq_pay_amount),0.0) as dq_pay_amount, coalesce(sum(gift_cps_pay_amount),0.0) as gift_cps_pay_amount , coalesce(sum(mobile_red_packet_pay_amount),0.0) as mobile_red_packet_pay_amount, coalesce(sum(acct_bal_pay_amount),0.0) as acct_bal_pay_amount, coalesce(sum(jbean_pay_amount),0.0) as jbean_pay_amount, coalesce(sum(sku_rebate_amount),0.0) as sku_rebate_amount, coalesce(sum(yixun_point_pay_amount),0.0) as yixun_point_pay_amount, coalesce(sum(sku_freight_coupon_amount),0.0) as freight_coupon_amount fromord_at_det_di where ds = '2015-05-20' group by sale_ord_id {noformat} the sql scan two text files and each file is 360MB,we use 6 executor, each executor has 8GB memory and 2 cpu. We can add a config control map aggregation to avoid it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-6392) [SQL]class not found exception thows when `add jar` use spark cli
jeanlyn created SPARK-6392: -- Summary: [SQL]class not found exception thows when `add jar` use spark cli Key: SPARK-6392 URL: https://issues.apache.org/jira/browse/SPARK-6392 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.2.0 Reporter: jeanlyn Priority: Minor When we use spark cli to add jar dynamic,we will get the *java.lang.ClassNotFoundException* when we use the class of jar to create udf.For example: {noformat} spark-sql> add jar /home/jeanlyn/hello.jar; spark-sql>create temporary function hello as 'hello'; spark-sql>select hello(name) from person; Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost): java.lang.ClassNotFoundException: hello {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-5498) [SPARK-SQL]when the partition schema does not match table schema,it throws java.lang.ClassCastException and so on
jeanlyn created SPARK-5498: -- Summary: [SPARK-SQL]when the partition schema does not match table schema,it throws java.lang.ClassCastException and so on Key: SPARK-5498 URL: https://issues.apache.org/jira/browse/SPARK-5498 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.2.0 Reporter: jeanlyn when the partition schema does not match table schema,it will thows exception when the task is running.For example,we modify the type of column from int to bigint by the sql *ALTER TABLE table_with_partition CHANGE COLUMN key key BIGINT* ,then we query the patition data which was stored before the changing,we would get the exception: {noformat} org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 27.0 failed 4 times, most recent failure: Lost task 0.3 in stage 27.0 (TID 30, BJHC-HADOOP-HERA-16950.jeanlyn.local): java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.MutableLong cannot be cast to org.apache.spark.sql.catalyst.expressions.MutableInt at org.apache.spark.sql.catalyst.expressions.SpecificMutableRow.setInt(SpecificMutableRow.scala:241) at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$13$$anonfun$apply$4.apply(TableReader.scala:286) at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$13$$anonfun$apply$4.apply(TableReader.scala:286) at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$1.apply(TableReader.scala:322) at org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$1.apply(TableReader.scala:314) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$$anon$10.next(Iterator.scala:312) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) at scala.collection.AbstractIterator.to(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) at org.apache.spark.sql.execution.Limit$$anonfun$4.apply(basicOperators.scala:141) at org.apache.spark.sql.execution.Limit$$anonfun$4.apply(basicOperators.scala:141) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1314) at org.apache.spark.SparkContext$$anonfun$runJob$4.apply(SparkContext.scala:1314) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) at org.apache.spark.scheduler.Task.run(Task.scala:56) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420) at akka.actor.Actor$class.aroundReceive(Actor.scala:465) at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375) at a
[jira] [Updated] (SPARK-5068) When the path not found in the hdfs,we can't get the result
[ https://issues.apache.org/jira/browse/SPARK-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-5068: --- Fix Version/s: (was: 1.2.1) > When the path not found in the hdfs,we can't get the result > --- > > Key: SPARK-5068 > URL: https://issues.apache.org/jira/browse/SPARK-5068 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0 >Reporter: jeanlyn > > when the partion path was found in the metastore but not found in the hdfs,it > will casue some problems as follow: > {noformat} > hive> show partitions partition_test; > OK > dt=1 > dt=2 > dt=3 > dt=4 > Time taken: 0.168 seconds, Fetched: 4 row(s) > {noformat} > {noformat} > hive> dfs -ls /user/jeanlyn/warehouse/partition_test; > Found 3 items > drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 > /user/jeanlyn/warehouse/partition_test/dt=1 > drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 > /user/jeanlyn/warehouse/partition_test/dt=3 > drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 17:42 > /user/jeanlyn/warehouse/partition_test/dt=4 > {noformat} > when i run the sql > {noformat} > select * from partition_test limit 10 > {noformat} in *hive*,i got no problem,but when i run in *spark-sql* i get > the error as follow: > {noformat} > Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: > Input path does not exist: > hdfs://jeanlyn:9000/user/jeanlyn/warehouse/partition_test/dt=2 > at > org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) > at > org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) > at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at > org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) > at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.List.foreach(List.scala:318) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:66) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328) > at org.apache.spark.rdd.RDD.collect(RDD.scala:780) > at > org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:84) > at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:444) > at org.apache.spark.sql.hive.testpartition$.main(test.scala:23) > at org.apache.spark.sql.hive.testpartition.main(test.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) -
[jira] [Commented] (SPARK-5084) when mysql is used as the metadata storage for spark-sql, Exception occurs when HiveQuerySuite is excute
[ https://issues.apache.org/jira/browse/SPARK-5084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14264183#comment-14264183 ] jeanlyn commented on SPARK-5084: more description? > when mysql is used as the metadata storage for spark-sql, Exception occurs > when HiveQuerySuite is excute > - > > Key: SPARK-5084 > URL: https://issues.apache.org/jira/browse/SPARK-5084 > Project: Spark > Issue Type: Bug > Components: SQL >Reporter: baishuo > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-5068) When the path not found in the hdfs,we can't get the result
[ https://issues.apache.org/jira/browse/SPARK-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-5068: --- Fix Version/s: 1.2.1 > When the path not found in the hdfs,we can't get the result > --- > > Key: SPARK-5068 > URL: https://issues.apache.org/jira/browse/SPARK-5068 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.2.0 >Reporter: jeanlyn > Fix For: 1.2.1 > > > when the partion path was found in the metastore but not found in the hdfs,it > will casue some problems as follow: > {noformat} > hive> show partitions partition_test; > OK > dt=1 > dt=2 > dt=3 > dt=4 > Time taken: 0.168 seconds, Fetched: 4 row(s) > {noformat} > {noformat} > hive> dfs -ls /user/jeanlyn/warehouse/partition_test; > Found 3 items > drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 > /user/jeanlyn/warehouse/partition_test/dt=1 > drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 > /user/jeanlyn/warehouse/partition_test/dt=3 > drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 17:42 > /user/jeanlyn/warehouse/partition_test/dt=4 > {noformat} > when i run the sql > {noformat} > select * from partition_test limit 10 > {noformat} in *hive*,i got no problem,but when i run in *spark-sql* i get > the error as follow: > {noformat} > Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: > Input path does not exist: > hdfs://jeanlyn:9000/user/jeanlyn/warehouse/partition_test/dt=2 > at > org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) > at > org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) > at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at > org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) > at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at scala.collection.immutable.List.foreach(List.scala:318) > at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.AbstractTraversable.map(Traversable.scala:105) > at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:66) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) > at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) > at scala.Option.getOrElse(Option.scala:120) > at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) > at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328) > at org.apache.spark.rdd.RDD.collect(RDD.scala:780) > at > org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:84) > at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:444) > at org.apache.spark.sql.hive.testpartition$.main(test.scala:23) > at org.apache.spark.sql.hive.testpartition.main(test.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332
[jira] [Updated] (SPARK-5068) When the path not found in the hdfs,we can't get the result
[ https://issues.apache.org/jira/browse/SPARK-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-5068: --- Description: when the partion path was found in the metastore but not found in the hdfs,it will casue some problems as follow: {noformat} hive> show partitions partition_test; OK dt=1 dt=2 dt=3 dt=4 Time taken: 0.168 seconds, Fetched: 4 row(s) {noformat} {noformat} hive> dfs -ls /user/jeanlyn/warehouse/partition_test; Found 3 items drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=1 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=3 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 17:42 /user/jeanlyn/warehouse/partition_test/dt=4 {noformat} when i run the sql {noformat} select * from partition_test limit 10 {noformat} in *hive*,i got no problem,but when i run in *spark-sql* i get the error as follow: {noformat} Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://jeanlyn:9000/user/jeanlyn/warehouse/partition_test/dt=2 at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:66) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328) at org.apache.spark.rdd.RDD.collect(RDD.scala:780) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:84) at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:444) at org.apache.spark.sql.hive.testpartition$.main(test.scala:23) at org.apache.spark.sql.hive.testpartition.main(test.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) {noformat} was: when the partion path was found in the metastore but not found in the hdfs,it will casue some problems as follow: ``` hive> show partitions partition_test; OK dt=1 dt=2 dt=3 dt=4 Time taken: 0.168 seconds, Fetched: 4 row(s) ``` ``` hive> dfs -ls /user/jeanlyn/warehouse/partition_test; Found 3 items drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=1 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=3 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 17:42 /user/jeanlyn/warehouse/partition_test/dt=4 ``` when
[jira] [Updated] (SPARK-5068) When the path not found in the hdfs,we can't get the result
[ https://issues.apache.org/jira/browse/SPARK-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-5068: --- Description: when the partion path was found in the metastore but not found in the hdfs,it will casue some problems as follow: ``` hive> show partitions partition_test; OK dt=1 dt=2 dt=3 dt=4 Time taken: 0.168 seconds, Fetched: 4 row(s) ``` ``` hive> dfs -ls /user/jeanlyn/warehouse/partition_test; Found 3 items drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=1 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=3 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 17:42 /user/jeanlyn/warehouse/partition_test/dt=4 ``` when i run the sq `select * from partition_test limit 10` in **hive**,i got no problem,but when i run in spark-sql i get the error as follow: ``` Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://jeanlyn:9000/user/jeanlyn/warehouse/partition_test/dt=2 at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:66) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328) at org.apache.spark.rdd.RDD.collect(RDD.scala:780) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:84) at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:444) at org.apache.spark.sql.hive.testpartition$.main(test.scala:23) at org.apache.spark.sql.hive.testpartition.main(test.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) ``` was: when the partion path was found in the metastore but not found in the hdfs,it will casue some problems as follow: ``` hive> show partitions partition_test; OK dt=1 dt=2 dt=3 dt=4 Time taken: 0.168 seconds, Fetched: 4 row(s) ``` ``` hive> dfs -ls /user/jeanlyn/warehouse/partition_test; Found 3 items drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=1 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=3 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 17:42 /user/jeanlyn/warehouse/partition_test/dt=4 ``` when i run the sq `select * from partition_test limit 10`l in **hiv
[jira] [Created] (SPARK-5068) When the path not found in the hdfs,we can't get the result
jeanlyn created SPARK-5068: -- Summary: When the path not found in the hdfs,we can't get the result Key: SPARK-5068 URL: https://issues.apache.org/jira/browse/SPARK-5068 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 1.2.0 Reporter: jeanlyn when the partion path was found in the metastore but not found in the hdfs,it will casue some problems as follow: ``` hive> show partitions partition_test; OK dt=1 dt=2 dt=3 dt=4 Time taken: 0.168 seconds, Fetched: 4 row(s) ``` ``` hive> dfs -ls /user/jeanlyn/warehouse/partition_test; Found 3 items drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=1 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 16:29 /user/jeanlyn/warehouse/partition_test/dt=3 drwxr-xr-x - jeanlyn supergroup 0 2014-12-02 17:42 /user/jeanlyn/warehouse/partition_test/dt=4 ``` when i run the sq `select * from partition_test limit 10`l in **hive**,i got no problem,but when i run in spark-sql i get the error as follow: ``` Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://jeanlyn:9000/user/jeanlyn/warehouse/partition_test/dt=2 at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:251) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:32) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply(UnionRDD.scala:66) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.AbstractTraversable.map(Traversable.scala:105) at org.apache.spark.rdd.UnionRDD.getPartitions(UnionRDD.scala:66) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328) at org.apache.spark.rdd.RDD.collect(RDD.scala:780) at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:84) at org.apache.spark.sql.SchemaRDD.collect(SchemaRDD.scala:444) at org.apache.spark.sql.hive.testpartition$.main(test.scala:23) at org.apache.spark.sql.hive.testpartition.main(test.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) ``` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181057#comment-14181057 ] jeanlyn commented on SPARK-3967: dsa dsa > Spark applications fail in yarn-cluster mode when the directories configured > in yarn.nodemanager.local-dirs are located on different disks/partitions > - > > Key: SPARK-3967 > URL: https://issues.apache.org/jira/browse/SPARK-3967 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Christophe PRÉAUD > Attachments: spark-1.1.0-utils-fetch.patch, > spark-1.1.0-yarn_cluster_tmpdir.patch > > > Spark applications fail from time to time in yarn-cluster mode (but not in > yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is > set to a comma-separated list of directories which are located on different > disks/partitions. > Steps to reproduce: > 1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of > directories located on different partitions (the more you set, the more > likely it will be to reproduce the bug): > (...) > > yarn.nodemanager.local-dirs > > file:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir > > (...) > 2. Launch (several times) an application in yarn-cluster mode, it will fail > (apparently randomly) from time to time -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jeanlyn updated SPARK-3967: --- Comment: was deleted (was: dsa dsa) > Spark applications fail in yarn-cluster mode when the directories configured > in yarn.nodemanager.local-dirs are located on different disks/partitions > - > > Key: SPARK-3967 > URL: https://issues.apache.org/jira/browse/SPARK-3967 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Christophe PRÉAUD > Attachments: spark-1.1.0-utils-fetch.patch, > spark-1.1.0-yarn_cluster_tmpdir.patch > > > Spark applications fail from time to time in yarn-cluster mode (but not in > yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is > set to a comma-separated list of directories which are located on different > disks/partitions. > Steps to reproduce: > 1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of > directories located on different partitions (the more you set, the more > likely it will be to reproduce the bug): > (...) > > yarn.nodemanager.local-dirs > > file:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir > > (...) > 2. Launch (several times) an application in yarn-cluster mode, it will fail > (apparently randomly) from time to time -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178291#comment-14178291 ] jeanlyn commented on SPARK-3967: I think you this pull request also can be referenced: https://github.com/apache/spark/pull/1616 > Spark applications fail in yarn-cluster mode when the directories configured > in yarn.nodemanager.local-dirs are located on different disks/partitions > - > > Key: SPARK-3967 > URL: https://issues.apache.org/jira/browse/SPARK-3967 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Christophe PRÉAUD > Attachments: spark-1.1.0-utils-fetch.patch, > spark-1.1.0-yarn_cluster_tmpdir.patch > > > Spark applications fail from time to time in yarn-cluster mode (but not in > yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is > set to a comma-separated list of directories which are located on different > disks/partitions. > Steps to reproduce: > 1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of > directories located on different partitions (the more you set, the more > likely it will be to reproduce the bug): > (...) > > yarn.nodemanager.local-dirs > > file:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir > > (...) > 2. Launch (several times) an application in yarn-cluster mode, it will fail > (apparently randomly) from time to time -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-3967) Spark applications fail in yarn-cluster mode when the directories configured in yarn.nodemanager.local-dirs are located on different disks/partitions
[ https://issues.apache.org/jira/browse/SPARK-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14178221#comment-14178221 ] jeanlyn commented on SPARK-3967: This issue also hapens to me.but i want to know why this issue don't happens to the *yarn-client* mode > Spark applications fail in yarn-cluster mode when the directories configured > in yarn.nodemanager.local-dirs are located on different disks/partitions > - > > Key: SPARK-3967 > URL: https://issues.apache.org/jira/browse/SPARK-3967 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: Christophe PRÉAUD > Attachments: spark-1.1.0-utils-fetch.patch, > spark-1.1.0-yarn_cluster_tmpdir.patch > > > Spark applications fail from time to time in yarn-cluster mode (but not in > yarn-client mode) when yarn.nodemanager.local-dirs (Hadoop YARN config) is > set to a comma-separated list of directories which are located on different > disks/partitions. > Steps to reproduce: > 1. Set yarn.nodemanager.local-dirs (in yarn-site.xml) to a list of > directories located on different partitions (the more you set, the more > likely it will be to reproduce the bug): > (...) > > yarn.nodemanager.local-dirs > > file:/d1/yarn/local/nm-local-dir,file:/d2/yarn/local/nm-local-dir,file:/d3/yarn/local/nm-local-dir,file:/d4/yarn/local/nm-local-dir,file:/d5/yarn/local/nm-local-dir,file:/d6/yarn/local/nm-local-dir,file:/d7/yarn/local/nm-local-dir > > (...) > 2. Launch (several times) an application in yarn-cluster mode, it will fail > (apparently randomly) from time to time -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org