[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940406#comment-14940406 ] Li Lu commented on YARN-3942: - YARN-4219 opened to track the new leveldb cache storage separately. > Timeline

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940307#comment-14940307 ] Jason Lowe commented on YARN-3942: -- I think the best way forward is to get the original patch cleaned up

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940314#comment-14940314 ] Li Lu commented on YARN-3942: - Thanks for the info [~jlowe]! Sure I can open a new JIRA for the leveldb cache

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940500#comment-14940500 ] Hadoop QA commented on YARN-3942: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935549#comment-14935549 ] Li Lu commented on YARN-3942: - Hi folks, I'm trying to figure out out next plan on this JIRA. Are we planning

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933864#comment-14933864 ] Jason Lowe commented on YARN-3942: -- Greg wrote: bq. I was even able to create the problem with the REST/WS

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908503#comment-14908503 ] Li Lu commented on YARN-3942: - Ah I see. BTW, seems like the cache loading process in refreshCache is properly

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908423#comment-14908423 ] Greg Senia commented on YARN-3942: -- Noticed this in the timeline server logs with certain hive jobs..

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908435#comment-14908435 ] Li Lu commented on YARN-3942: - H... Seems quite likely, but let me double check it... > Timeline store to

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908482#comment-14908482 ] Greg Senia commented on YARN-3942: -- I see the issue.. Our security folks mandated a umask change and my

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908790#comment-14908790 ] Greg Senia commented on YARN-3942: -- I was even able to create the problem with the REST/WS endpoint...

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908845#comment-14908845 ] Jonathan Eagles commented on YARN-3942: --- Also, I'm getting an error while starting up the

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908765#comment-14908765 ] Greg Senia commented on YARN-3942: -- Got beyond that and now getting this: 2015-09-25 17:43:32,249 - ERROR

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908855#comment-14908855 ] Li Lu commented on YARN-3942: - [~gss2002] the exception stack looks related to security settings? Not sure if

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14908893#comment-14908893 ] Greg Senia commented on YARN-3942: -- [~gtCarrera9] that is all that is in the ATS log w/ debug enabled on

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906561#comment-14906561 ] Jason Lowe commented on YARN-3942: -- Thanks for the patch, Li! I looks good to me after a brief overview

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14906794#comment-14906794 ] Li Lu commented on YARN-3942: - Thanks [~jlowe]! Unfortunately we haven't got a chance to test the latency for

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907612#comment-14907612 ] Greg Senia commented on YARN-3942: -- Also [~gtCarrera] I was not able to get your patch to compile on my

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907610#comment-14907610 ] Greg Senia commented on YARN-3942: -- [~jlowe] The patch worked correctly. I meant to state ATS was crashing

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905398#comment-14905398 ] Xuan Gong commented on YARN-3942: - The patch while implements LevelDBCacheTimelineStore as cache used in

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901388#comment-14901388 ] Li Lu commented on YARN-3942: - BTW, the patch apply to the existing YARN-3942.001.patch. > Timeline store to

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743621#comment-14743621 ] Jason Lowe commented on YARN-3942: -- Thanks for trying out the patch, [~gss2002]! Sorry to hear it's been

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14742159#comment-14742159 ] Hitesh Shah commented on YARN-3942: --- Thanks [~gss2002]. One point to note - if you use long running Hive

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741848#comment-14741848 ] Greg Senia commented on YARN-3942: -- I have placed this patch and the Tez patch into our test environments

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14730887#comment-14730887 ] Jason Lowe commented on YARN-3942: -- bq. Since we know the file size to be read, could we return a message

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14729815#comment-14729815 ] Hitesh Shah commented on YARN-3942: --- Some ideas from an offline discussion with [~bikassaha] and

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14729843#comment-14729843 ] Jason Lowe commented on YARN-3942: -- Option 1 will add some latency (not clear how much yet) to

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14729917#comment-14729917 ] Bikas Saha commented on YARN-3942: -- In Option 1 the latency would be the time to read the entire session

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720668#comment-14720668 ] Jason Lowe commented on YARN-3942: -- Yeah that's going to be tricky, especially if we need

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14720574#comment-14720574 ] Hitesh Shah commented on YARN-3942: --- [~jlowe] [~rajesh.balamohan] observed that the

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14701242#comment-14701242 ] Jason Lowe commented on YARN-3942: -- [~rajesh] the initial exception looks like an issue

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700657#comment-14700657 ] Rajesh Balamohan commented on YARN-3942: Should this be resilient to cluster

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652753#comment-14652753 ] Sangjin Lee commented on YARN-3942: --- +1 with this idea as a way to mitigate the

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647873#comment-14647873 ] Jason Lowe commented on YARN-3942: -- I think so. We can probably create a new

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648057#comment-14648057 ] Jason Lowe commented on YARN-3942: -- The logs are created based on app attempt. It helps

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648048#comment-14648048 ] Zhijie Shen commented on YARN-3942: --- Yeah, I prefer creating a TimelineEntityFileClient

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647217#comment-14647217 ] Zhijie Shen commented on YARN-3942: --- [~jlowe], thanks for sharing more information about

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14644394#comment-14644394 ] Jason Lowe commented on YARN-3942: -- bq. if you'd like to elaborate the drawback a bit, it

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643790#comment-14643790 ] Zhijie Shen commented on YARN-3942: --- Thanks for this work. Agree it's a good interim step

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634079#comment-14634079 ] Vinod Kumar Vavilapalli commented on YARN-3942: --- Tx for posting this

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634058#comment-14634058 ] Jason Lowe commented on YARN-3942: -- TEZ-2628 has the corresponding changes to allow Tez

[jira] [Commented] (YARN-3942) Timeline store to read events from HDFS

[ https://issues.apache.org/jira/browse/YARN-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634071#comment-14634071 ] Jason Lowe commented on YARN-3942: -- To setup for the Tez UI,