[jira] [Comment Edited] (SPARK-27169) number of active tasks is negative on executors page
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797070#comment-16797070 ] acupple edited comment on SPARK-27169 at 3/22/19 3:12 AM: -- Thanks for your suggestion, and I will try increment the queue size and reproduce the case。 was (Author: acupple): Thanks for you suggestion, and I will try increment the queue size and reproduce the case。 > number of active tasks is negative on executors page > > > Key: SPARK-27169 > URL: https://issues.apache.org/jira/browse/SPARK-27169 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.2 >Reporter: acupple >Priority: Minor > Attachments: QQ20190315-102215.png, QQ20190315-102235.png, > image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, > job_1924.log, stage_3511.log > > > I use spark to process some data in HDFS and HBASE, I use one thread consume > message from a queue, and then submit to a thread pool（16 fix size）for spark > processor. > But when run for some time, the active jobs will be thousands, and number of > active tasks are negative. > Actually, these jobs are already done when I check driver logs。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27169) number of active tasks is negative on executors page
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16797066#comment-16797066 ] shahid edited comment on SPARK-27169 at 3/20/19 11:26 AM: -- Yes. that means many event drops happens. Can you try increasing the queue size, "spark.scheduler.listenerbus.eventqueue.capacity" (default 1) might helps. If event drop happens, then UI display weirdly, I'm not sure, from the UI side we can do anything. Do you have any reproducible steps for that, so that I can try? was (Author: shahid): Yes. that means many event drops happens. Can you try increasing the queue size, "spark.scheduler.listenerbus.eventqueue.capacity" (default 1) might helps. If event drop happens, then UI display weirdly only, I'm not sure, from the UI side we can do anything. Do you have any reproducible steps for that, so that I can try? > number of active tasks is negative on executors page > > > Key: SPARK-27169 > URL: https://issues.apache.org/jira/browse/SPARK-27169 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.2 >Reporter: acupple >Priority: Minor > Attachments: QQ20190315-102215.png, QQ20190315-102235.png, > image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, > job_1924.log, stage_3511.log > > > I use spark to process some data in HDFS and HBASE, I use one thread consume > message from a queue, and then submit to a thread pool（16 fix size）for spark > processor. > But when run for some time, the active jobs will be thousands, and number of > active tasks are negative. > Actually, these jobs are already done when I check driver logs。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27169) number of active tasks is negative on executors page
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795878#comment-16795878 ] shahid edited comment on SPARK-27169 at 3/19/19 9:38 AM: - Thank you. Could you please provide full event log if possible? was (Author: shahid): Thank you. Could you please provide full event log if possible? I think, some task events are missed in the eventlog > number of active tasks is negative on executors page > > > Key: SPARK-27169 > URL: https://issues.apache.org/jira/browse/SPARK-27169 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.2 >Reporter: acupple >Priority: Minor > Attachments: QQ20190315-102215.png, QQ20190315-102235.png, > image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, > job_1924.log > > > I use spark to process some data in HDFS and HBASE, I use one thread consume > message from a queue, and then submit to a thread pool（16 fix size）for spark > processor. > But when run for some time, the active jobs will be thousands, and number of > active tasks are negative. > Actually, these jobs are already done when I check driver logs。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27169) number of active tasks is negative on executors page
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795774#comment-16795774 ] acupple edited comment on SPARK-27169 at 3/19/19 7:22 AM: -- I have enabled event log, on the spark UI, the job has not completed, but according to event log, the job has completed. !image-2019-03-19-15-17-25-522.png|width=872,height=48! the attachment is event log [^job_1924.log] And there is one unknown stage !image-2019-03-19-15-21-03-766.png|width=610,height=70! was (Author: acupple): I have enabled event log, on the spark UI, the job has not completed, but according to event log, the job has completed. !image-2019-03-19-15-17-25-522.png|width=872,height=48! the attachment is event log [^job_1924.log] And there is one unknown stage > number of active tasks is negative on executors page > > > Key: SPARK-27169 > URL: https://issues.apache.org/jira/browse/SPARK-27169 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.2 >Reporter: acupple >Priority: Minor > Attachments: QQ20190315-102215.png, QQ20190315-102235.png, > image-2019-03-19-15-17-25-522.png, image-2019-03-19-15-21-03-766.png, > job_1924.log > > > I use spark to process some data in HDFS and HBASE, I use one thread consume > message from a queue, and then submit to a thread pool（16 fix size）for spark > processor. > But when run for some time, the active jobs will be thousands, and number of > active tasks are negative. > Actually, these jobs are already done when I check driver logs。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-27169) number of active tasks is negative on executors page
[ https://issues.apache.org/jira/browse/SPARK-27169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793675#comment-16793675 ] shahid edited comment on SPARK-27169 at 3/16/19 8:40 AM: - Seems event drop has happened. Could you please provide logs,? was (Author: shahid): Seems event drop has happened. > number of active tasks is negative on executors page > > > Key: SPARK-27169 > URL: https://issues.apache.org/jira/browse/SPARK-27169 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.2 >Reporter: acupple >Priority: Minor > Attachments: QQ20190315-102215.png, QQ20190315-102235.png > > > I use spark to process some data in HDFS and HBASE, I use one thread consume > message from a queue, and then submit to a thread pool（16 fix size）for spark > processor. > But when run for some time, the active jobs will be thousands, and number of > active tasks are negative. > Actually, these jobs are already done when I check driver logs。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
