[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16133719#comment-16133719 ] Vrushali C commented on YARN-5304: -- Thanks [~rohithsharma] , yes I agree, removed the label. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Vrushali C > Labels: YARN-5355 > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16131869#comment-16131869 ] Rohith Sharma K S commented on YARN-5304: - Hi [~vrushalic] Is this merge blocker? I see this as improvement, so could we remove label? > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Vrushali C > Labels: YARN-5355, yarn-5355-atsv2-blocker > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095225#comment-16095225 ] Vrushali C commented on YARN-5304: -- Went over this jira with [~jrottinghuis] and here is a summary of what we discussed: - Provide a default hbase site via timeline service (perhaps called hbase-site-timelineservice.xml) which can be used to start up a single node hbase setup - ask the user to set HBASE_HOME to point to where the hbase installation is - ask the user to set yarn.timeline-service.hbase.configuration.file config param to where the hbase-site-timelineservice.xml is located - ask the user to ensure that hbase is set up on a node which is in the ZK quorum setting in the config for the clients to be able to connect to this cluster. it could be the node as yarn.timeline-service.hostname or it could be a node like RM. - ask the user to ensure that hbase clients have this yarn.timeline-service.hbase.configuration.file in their classpath. - Provide a set of steps so that a user can start up / stop a single node hbase cluster (ensuring that the NN address is in the hbase root dir setting). - Also, ensure that it's clear that some steps are one-time only, like copying the jars for coprocessor and creating the schema. They need not be done each time the cluster is started/stopped/restarted. The steps should be made as clear as we can make them. The idea is that these steps should be such that users can just copy/paste these commands (with minimal, clearly explained variable modifications) and run them. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Vrushali C > Labels: YARN-5355, yarn-5355-merge-blocker > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830747#comment-15830747 ] Sangjin Lee commented on YARN-5304: --- >From the client perspective, we need to look at some timeout configuration >values. If I recall, the default values are too long in that a put may not >fail for 30 minutes or that long. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Vrushali C > Labels: YARN-5355, yarn-5355-merge-blocker > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830743#comment-15830743 ] Vrushali C commented on YARN-5304: -- bq. If it is not too much, it would be great if we can add all the necessary defaults as part of this JIRA. Let me know what you think. I am not so sure that we can aim to cover "all necessary defaults" as part of this jira. Some defaults can be added but for some it may be good to add them after having run them for some time on our clusters. Let me know if you have any specific setting suggestions (other than hfile one). > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Vrushali C > Labels: YARN-5355, yarn-5355-merge-blocker > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[
https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830702#comment-15830702
]
Sangjin Lee commented on YARN-5304:
---
We should review and set some of the recommended properties in the property xml
file that should be created as part of this JIRA. For example, we should
explicitly set {{hfile.format.version}} to 3 (see YARN-6094).
If it is not too much, it would be great if we can add all the necessary
defaults as part of this JIRA. Let me know what you think.
> Ship single node HBase config option with single startup command
>
>
> Key: YARN-5304
> URL: https://issues.apache.org/jira/browse/YARN-5304
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Joep Rottinghuis
>Assignee: Vrushali C
> Labels: YARN-5355, yarn-5355-merge-blocker
>
> For small to medium Hadoop deployments we should make it dead-simple to use
> the timeline service v2. We should have a single command to launch and stop
> the timelineservice back-end for the default HBase implementation.
> A default config with all the values should be packaged that launches all the
> needed daemons (on the RM node) with a single command with all the
> recommended settings.
> Having a timeline admin command, perhaps an init command might be needed, or
> perhaps the timeline service can even auto-detect that and create tables,
> deploy needed coprocessors etc.
> The overall purpose is to ensure nobody needs to be an HBase expert to get
> this going. For those cluster operators with HBase experience, they can
> choose their own more sophisticated deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[
https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819904#comment-15819904
]
Sangjin Lee commented on YARN-5304:
---
Thanks for the summary [~vrushalic]. It is a good summary of the discussion.
Just to add a couple of more fine points,
- we would package this timeline service specific hbase configuration file in
hadoop
- this file would now be required to be present; that would also entail making
{{TIMELINE_SERVICE_HBASE_CONFIGURATION_FILE}} a required config and the file it
points to required, or
{{HBaseTimelineStorageUtils.getTimelineServiceHBaseConf()}} should fail
- bringing up hbase would require using this config file via the {{--config}}
option (i.e. {{"hbase --config ...}})
> Ship single node HBase config option with single startup command
>
>
> Key: YARN-5304
> URL: https://issues.apache.org/jira/browse/YARN-5304
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Joep Rottinghuis
>Assignee: Vrushali C
> Labels: YARN-5355, yarn-5355-merge-blocker
>
> For small to medium Hadoop deployments we should make it dead-simple to use
> the timeline service v2. We should have a single command to launch and stop
> the timelineservice back-end for the default HBase implementation.
> A default config with all the values should be packaged that launches all the
> needed daemons (on the RM node) with a single command with all the
> recommended settings.
> Having a timeline admin command, perhaps an init command might be needed, or
> perhaps the timeline service can even auto-detect that and create tables,
> deploy needed coprocessors etc.
> The overall purpose is to ensure nobody needs to be an HBase expert to get
> this going. For those cluster operators with HBase experience, they can
> choose their own more sophisticated deployment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817376#comment-15817376 ] Vrushali C commented on YARN-5304: -- [~jrottinghuis], [~sjlee0] and I had an offline discussion about this. Summarizing the discussion here: - Objective is to make things easier for the simple user who does not know about HBase to get going with ATSv2 . - Have a sample HBase config with default settings supplied with timeline service that enables settings for timeline service v2. Ensure that documentation has steps clarifying use of this HBase config for the HBase cluster setup. We discussed about having different HBase config files for the server side and for clients. The server side HBase config can be made simple enough if it’s a standalone deployment, say in case we are bringing up a HBase setup on the RM node itself. We discussed about providing a yarn-daemon command that can spin up a ATSv2 HBase backend using the sample HBase config supplied with the code. But this is not feasible since this would mean ensuring the pids for HBase daemons are handled by the yarn command. Also, HBase jars are needed to be made available, which is outside of YARN/ATSv2 in any case. In case of more complex deployment scenarios, like we have, say if there exists an HBase cluster for ATSv2 data separate from other HBase clusters, then we need a way to have different HBase configs such that there is a way to connect from an application on a particular compute node on a hadoop cluster to two different HBase clusters. One connection is for writing timeline service data and another for the application to read/write from/to the other HBase cluster for it’s own purpose, which [~jrottinghuis] addressed in YARN-5265. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Vrushali C > Labels: YARN-5355, yarn-5355-merge-blocker > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15735772#comment-15735772 ] Vrushali C commented on YARN-5304: -- Thanks Joep, I will take a look at this along with YARN-5980, might perhaps close that one as a duplicate depending on what I see. I will start on this most likely next week but will aim to close this out as soon as possible. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355 > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390217#comment-15390217 ] Joep Rottinghuis commented on YARN-5304: The goal is the simplest possible setup, but not any simpler. [~stack] you mentioned we could consider filing an HBase jira to make MiniHBaseCluster persist files to HDFS rather than just local storage. One challenge with that is that this class, and its MiniHBaseClusterRegionServer and SingleFileSystemShutdownThread subclasses, sit in src/test/java directory and hence gets packaged in hbase-server-X.Y.Z-tests.jar It seems that does leave us with the approach of a setup with 3 JVMs getting launched, even in the simplest setup: 1 for the Master, 1 for the RegionServer, 1 for ZK. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355 > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15375807#comment-15375807 ] Karthik Kambatla commented on YARN-5304: I spoke to [~esteban] about this. In his opinion, the minicluster approach (master, RS etc. in a single process) is discouraged. I am assuming the goal is to do a pseudo-distributed setup of HBase - Master and RegionServer in different processes. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > Labels: YARN-5355 > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5304) Ship single node HBase config option with single startup command
[ https://issues.apache.org/jira/browse/YARN-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15358463#comment-15358463 ] Joep Rottinghuis commented on YARN-5304: When we tackle this issue well, then that will address at least some of the concerns raised in YARN-5281. > Ship single node HBase config option with single startup command > > > Key: YARN-5304 > URL: https://issues.apache.org/jira/browse/YARN-5304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: YARN-2928 >Reporter: Joep Rottinghuis >Assignee: Joep Rottinghuis > > For small to medium Hadoop deployments we should make it dead-simple to use > the timeline service v2. We should have a single command to launch and stop > the timelineservice back-end for the default HBase implementation. > A default config with all the values should be packaged that launches all the > needed daemons (on the RM node) with a single command with all the > recommended settings. > Having a timeline admin command, perhaps an init command might be needed, or > perhaps the timeline service can even auto-detect that and create tables, > deploy needed coprocessors etc. > The overall purpose is to ensure nobody needs to be an HBase expert to get > this going. For those cluster operators with HBase experience, they can > choose their own more sophisticated deployment. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
