[jira] [Comment Edited] (SPARK-18085) SPIP: Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125150#comment-16125150 ] duyanghao edited comment on SPARK-18085 at 8/14/17 2:32 AM: [~vanzin] so what you mean is that your project has done nothing about loading large event logs(100G+) and loading history summary page when the event log directory is large at the first time.What you have done is for quick loading logs for subsequent use. I am afraid that only solves one part of above problems(for subsequent use),and one more important problem is that how to quick load logs at the first time. was (Author: duyanghao): [~vanzin] so what you mean is that your project has done nothing about loading large event logs(100G+) and loading history summary page when the event log directory is large at the first time.What you have done is for quick loading logs for subsequent use. I am afraid that only solves one part of above problems(for subsequent use),one more important problem is that how to quick load logs at the first time. > SPIP: Better History Server scalability for many / large applications > - > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Labels: SPIP > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18085) SPIP: Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125150#comment-16125150 ] duyanghao edited comment on SPARK-18085 at 8/14/17 2:31 AM: [~vanzin] so what you mean is that your project has done nothing about loading large event logs(100G+) and loading history summary page when the event log directory is large at the first time.What you have done is for quick loading logs for subsequent use. I am afraid that only solves one part of above problems(for subsequent use),one more important problem is that how to quick load logs at the first time. was (Author: duyanghao): [~vanzin] so what you mean is that your project has done nothing about loading large event logs(100G+) and loading history summary page when the event log directory is large at the first time.What you have done is for quick loading logs for subsequent use. > SPIP: Better History Server scalability for many / large applications > - > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Labels: SPIP > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18085) SPIP: Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125150#comment-16125150 ] duyanghao edited comment on SPARK-18085 at 8/14/17 2:28 AM: [~vanzin] so what you mean is that your project has done nothing about loading large event logs(100G+) and loading history summary page when the event log directory is large at the first time.What you have done is for quick loading logs for subsequent use. was (Author: duyanghao): [~vanzin] so what you mean is that your project has done nothing about loading large event logs(100G+) and loading history summary page when the event log directory is large at the first time. > SPIP: Better History Server scalability for many / large applications > - > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Labels: SPIP > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16125150#comment-16125150 ] duyanghao commented on SPARK-18085: --- [~vanzin] so what you mean is that your project has done nothing about loading large event logs(100G+) and loading history summary page when the event log directory is large at the first time. > SPIP: Better History Server scalability for many / large applications > - > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Labels: SPIP > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18085) SPIP: Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124421#comment-16124421 ] duyanghao edited comment on SPARK-18085 at 8/12/17 3:11 AM: [~vanzin] Could you have a summary of your progress(solved problems and unsolved problems) since we all have the same problems for loading large event logs(100G+) and loading history summary page when the event log directory is large. We all want to know how good your solution is(of course from your test results) and your plans to do it. was (Author: duyanghao): [~vanzin] Could you have a summary of your progress(solved problems and unsolved problems) since we all have the same problems for loading large event logs(100G+) and loading history summary page when the event log directory is large. We all want to know how good your solution is(of course from your test results)? > SPIP: Better History Server scalability for many / large applications > - > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Labels: SPIP > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124421#comment-16124421 ] duyanghao commented on SPARK-18085: --- [~vanzin] Could you have a summary of your progress(solved problems and unsolved problems) since we all have the same problems for loading large event logs(100G+) and loading history summary page when the event log directory is large. We all want to know how good your solution is(of course from your test results)? > SPIP: Better History Server scalability for many / large applications > - > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Labels: SPIP > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21348) Spark history Server load too slow when eventlog is large
[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079421#comment-16079421 ] duyanghao edited comment on SPARK-21348 at 7/9/17 2:36 AM: --- [~jerryshao] ok, i will follow SPARK-18085 to trace this issue. was (Author: duyanghao): [~jerryshao] ok, i will follow [SPARK-18085](https://issues.apache.org/jira/browse/SPARK-18085) to trace this issue. > Spark history Server load too slow when eventlog is large > - > > Key: SPARK-21348 > URL: https://issues.apache.org/jira/browse/SPARK-21348 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: duyanghao >Priority: Critical > > There is a eventlog file in hdfs as below: > ``` > -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 > hdfs://x/shared/spark-logs/x/.lz4 > ``` > and i start a spark history server to loading that eventlog file,but it is > too slow to get spark web ui through that spark history server. > it has requested for several hours but nothing about spark application web ui > appears in the browser > And the cpu usage of spark history server is 105.3% > total used-memory in host is 9G > The host has 48 Cores+130G Memory,enough Computing resources for spark > history server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21348) Spark history Server load too slow when eventlog is large
[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079421#comment-16079421 ] duyanghao commented on SPARK-21348: --- @Saisai Shao ok, i will follow [SPARK-18085](https://issues.apache.org/jira/browse/SPARK-18085) to trace this issue. > Spark history Server load too slow when eventlog is large > - > > Key: SPARK-21348 > URL: https://issues.apache.org/jira/browse/SPARK-21348 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: duyanghao >Priority: Critical > > There is a eventlog file in hdfs as below: > ``` > -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 > hdfs://x/shared/spark-logs/x/.lz4 > ``` > and i start a spark history server to loading that eventlog file,but it is > too slow to get spark web ui through that spark history server. > it has requested for several hours but nothing about spark application web ui > appears in the browser > And the cpu usage of spark history server is 105.3% > total used-memory in host is 9G > The host has 48 Cores+130G Memory,enough Computing resources for spark > history server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21348) Spark history Server load too slow when eventlog is large
[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079421#comment-16079421 ] duyanghao edited comment on SPARK-21348 at 7/9/17 2:35 AM: --- [~jerryshao] ok, i will follow [SPARK-18085](https://issues.apache.org/jira/browse/SPARK-18085) to trace this issue. was (Author: duyanghao): @Saisai Shao ok, i will follow [SPARK-18085](https://issues.apache.org/jira/browse/SPARK-18085) to trace this issue. > Spark history Server load too slow when eventlog is large > - > > Key: SPARK-21348 > URL: https://issues.apache.org/jira/browse/SPARK-21348 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: duyanghao >Priority: Critical > > There is a eventlog file in hdfs as below: > ``` > -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 > hdfs://x/shared/spark-logs/x/.lz4 > ``` > and i start a spark history server to loading that eventlog file,but it is > too slow to get spark web ui through that spark history server. > it has requested for several hours but nothing about spark application web ui > appears in the browser > And the cpu usage of spark history server is 105.3% > total used-memory in host is 9G > The host has 48 Cores+130G Memory,enough Computing resources for spark > history server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-21348) Spark history Server load too slow when eventlog is large
[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] duyanghao closed SPARK-21348. - Resolution: Duplicate > Spark history Server load too slow when eventlog is large > - > > Key: SPARK-21348 > URL: https://issues.apache.org/jira/browse/SPARK-21348 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: duyanghao >Priority: Critical > > There is a eventlog file in hdfs as below: > ``` > -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 > hdfs://x/shared/spark-logs/x/.lz4 > ``` > and i start a spark history server to loading that eventlog file,but it is > too slow to get spark web ui through that spark history server. > it has requested for several hours but nothing about spark application web ui > appears in the browser > And the cpu usage of spark history server is 105.3% > total used-memory in host is 9G > The host has 48 Cores+130G Memory,enough Computing resources for spark > history server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21348) Spark history Server load too slow when eventlog is large
[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] duyanghao updated SPARK-21348: -- Description: There is a eventlog file in hdfs as below: ``` -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://x/shared/spark-logs/x/.lz4 ``` and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server. it has requested for several hours but nothing about spark application web ui appears in the browser And the cpu usage of spark history server is 105.3% total used-memory in host is 9G The host has 48 Cores+130G Memory,enough Computing resources for spark history server. was: There is a eventlog file in hdfs as below: ``` -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://x/shared/spark-logs/x/.lz4 ``` and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server. it has requested for several hours but nothing about spark application web ui appears in the browser And the cpu usage of spark history server is 105.3% total used-memory in host is 9G The host have 48 Cores+130G Memory,enough Computing resources for spark history server. > Spark history Server load too slow when eventlog is large > - > > Key: SPARK-21348 > URL: https://issues.apache.org/jira/browse/SPARK-21348 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: duyanghao >Priority: Critical > > There is a eventlog file in hdfs as below: > ``` > -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 > hdfs://x/shared/spark-logs/x/.lz4 > ``` > and i start a spark history server to loading that eventlog file,but it is > too slow to get spark web ui through that spark history server. > it has requested for several hours but nothing about spark application web ui > appears in the browser > And the cpu usage of spark history server is 105.3% > total used-memory in host is 9G > The host has 48 Cores+130G Memory,enough Computing resources for spark > history server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21348) Spark history Server load too slow when eventlog is large
[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] duyanghao updated SPARK-21348: -- Description: There is a eventlog file in hdfs as below: ``` -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://x/shared/spark-logs/x/.lz4 ``` and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server. it has requested for several hours but nothing about spark application web ui appears in the browser And the cpu usage of spark history server is 105.3% total used-memory in host is 9G The host is 48 Cores+130G Memory,enough Computing resources for spark history server. was: There is a eventlog file in hdfs as below: ``` -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://x/shared/spark-logs/x/.lz4 ``` and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server. it has requested for several hours but nothing about spark application web ui appears in the browser And the usage of spark history server is 105.3% The host is 48 Cores+130G Memory,enough Computing resources for spark history server. > Spark history Server load too slow when eventlog is large > - > > Key: SPARK-21348 > URL: https://issues.apache.org/jira/browse/SPARK-21348 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: duyanghao >Priority: Critical > > There is a eventlog file in hdfs as below: > ``` > -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 > hdfs://x/shared/spark-logs/x/.lz4 > ``` > and i start a spark history server to loading that eventlog file,but it is > too slow to get spark web ui through that spark history server. > it has requested for several hours but nothing about spark application web ui > appears in the browser > And the cpu usage of spark history server is 105.3% > total used-memory in host is 9G > The host is 48 Cores+130G Memory,enough Computing resources for spark history > server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21348) Spark history Server load too slow when eventlog is large
[ https://issues.apache.org/jira/browse/SPARK-21348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] duyanghao updated SPARK-21348: -- Description: There is a eventlog file in hdfs as below: ``` -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://x/shared/spark-logs/x/.lz4 ``` and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server. it has requested for several hours but nothing about spark application web ui appears in the browser And the cpu usage of spark history server is 105.3% total used-memory in host is 9G The host have 48 Cores+130G Memory,enough Computing resources for spark history server. was: There is a eventlog file in hdfs as below: ``` -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://x/shared/spark-logs/x/.lz4 ``` and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server. it has requested for several hours but nothing about spark application web ui appears in the browser And the cpu usage of spark history server is 105.3% total used-memory in host is 9G The host is 48 Cores+130G Memory,enough Computing resources for spark history server. > Spark history Server load too slow when eventlog is large > - > > Key: SPARK-21348 > URL: https://issues.apache.org/jira/browse/SPARK-21348 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.1.0 >Reporter: duyanghao >Priority: Critical > > There is a eventlog file in hdfs as below: > ``` > -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 > hdfs://x/shared/spark-logs/x/.lz4 > ``` > and i start a spark history server to loading that eventlog file,but it is > too slow to get spark web ui through that spark history server. > it has requested for several hours but nothing about spark application web ui > appears in the browser > And the cpu usage of spark history server is 105.3% > total used-memory in host is 9G > The host have 48 Cores+130G Memory,enough Computing resources for spark > history server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21348) Spark history Server load too slow when eventlog is large
duyanghao created SPARK-21348: - Summary: Spark history Server load too slow when eventlog is large Key: SPARK-21348 URL: https://issues.apache.org/jira/browse/SPARK-21348 Project: Spark Issue Type: Bug Components: Web UI Affects Versions: 2.1.0 Reporter: duyanghao Priority: Critical There is a eventlog file in hdfs as below: ``` -rwxrwx--- 3 root supergroup 9634238709 2017-07-08 04:37 hdfs://x/shared/spark-logs/x/.lz4 ``` and i start a spark history server to loading that eventlog file,but it is too slow to get spark web ui through that spark history server. it has requested for several hours but nothing about spark application web ui appears in the browser And the usage of spark history server is 105.3% The host is 48 Cores+130G Memory,enough Computing resources for spark history server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
