> On July 26, 2017, 7:39 p.m., Dmytro Grinenko wrote: > > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py > > Lines 131 (patched) > > <https://reviews.apache.org/r/61146/diff/1/?file=1782632#file1782632line175> > > > > it is not overkill to post big json each time with request or with > > cluster-env posted? i believe that this structure is not something subject > > for often updates and proly better to put it to resources and sync on > > changes or post to agent initial version and after just post on changes? > > > > > > Current request r not small entites, and proly it is not good idea to > > make them even more larger. > > Jonathan Hurley wrote: > The problem with storing it as a resource is that we'd need to update it > for new stacks. That means we'd need to read from resources and then write > back to resources dynamically instead of just updating a property. > > Also, I timed it: > !!! Decoding the JSON took 1ms > > So, not very long :)
it is not really about decoding, i'm more concerning about amount of transfered data with each task and amount of spam on 2-3k node cluster. Roughtly, this json eating 23 kb of the space, what means 44 mb of additional data on 2k cluster per stage from one task. About updating data with new stacks, putting to resource is just what comes first to my head. We can sent data on agent registration for initial session and later with harbeats or even real commands for non-restart updates, this will allow us to instantly update cache. It is ok to keep those data on cluster-env, but prolly we should limit huge properties to be posted often and update them only when they really updated. - Dmytro ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61146/#review181481 ----------------------------------------------------------- On July 26, 2017, 7:04 p.m., Jonathan Hurley wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/61146/ > ----------------------------------------------------------- > > (Updated July 26, 2017, 7:04 p.m.) > > > Review request for Ambari, Alejandro Fernandez, Dmytro Grinenko, Dmitro > Lisnichenko, and Nate Cole. > > > Bugs: AMBARI-21580 > https://issues.apache.org/jira/browse/AMBARI-21580 > > > Repository: ambari > > > Description > ------- > > Currently, both the stack-select and conf-select python files have hard-coded > structures inside of them which contain mappings for things like > configuration directories and stack-select component names. > > This is mainly used for pairing the Ambari role name (defined from > metainfo.xml) with the {{stack-select}} package name. There are places, like > {{params.py}} where we have not yet entered the target python file (like > {{AccumuloScript}}) where we could get this information. Some components need > this to be able to build things like the Hadoop conf dir: > {code:title=stack_select.py} > SERVER_ROLE_DIRECTORY_MAP = { > 'ACCUMULO_MASTER' : 'accumulo-master', > 'ACCUMULO_MONITOR' : 'accumulo-monitor', > 'ACCUMULO_GC' : 'accumulo-gc', > 'ACCUMULO_TRACER' : 'accumulo-tracer', > 'ACCUMULO_TSERVER' : 'accumulo-tablet', > 'ATLAS_SERVER' : 'atlas-server', > 'FLUME_HANDLER' : 'flume-server', > 'FALCON_SERVER' : 'falcon-server', > ... > {code} > > With the coming of management packs replacing stacks, we can no longer hard > code this in Python. My suggestion is to begin moving this data into a > property of some sort. Today, {{cluster-env}} exists and would be the only > place for it (where the stack-feature tools are now). However, I believe that > {{cluster-env}} is also going away and being replaced with something similar > to "Cluster Settings". > > In any event, this Jira is to track the work needed to replace this logic. > > > Diffs > ----- > > > ambari-common/src/main/python/resource_management/libraries/functions/stack_select.py > 265e7df9a0 > > ambari-common/src/main/python/resource_management/libraries/script/script.py > cce3ac461f > > ambari-server/src/main/java/org/apache/ambari/server/controller/internal/BlueprintConfigurationProcessor.java > 50cea9e00b > > ambari-server/src/main/java/org/apache/ambari/server/state/ConfigHelper.java > 974ad4f29e > > ambari-server/src/main/java/org/apache/ambari/server/upgrade/FinalUpgradeCatalog.java > dad0ecf8b0 > > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_client.py > 67ca525f15 > > ambari-server/src/main/resources/common-services/ACCUMULO/1.6.1.2.2.0/package/scripts/accumulo_script.py > ebd418d855 > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/atlas_client.py > d01ff84842 > > ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata_server.py > 38f9a4118d > > ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/druid_node.py > 7c6bf39d5f > > ambari-server/src/main/resources/common-services/DRUID/0.9.2/package/scripts/superset.py > b837b2492c > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_client.py > 365f661a1d > > ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/falcon_server.py > 5b2db4477b > > ambari-server/src/main/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume_handler.py > 107ce6d8ae > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_client.py > f18a96af8d > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_master.py > 815157220b > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/hbase_regionserver.py > 9194991aa9 > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/phoenix_queryserver.py > b1bdb785a9 > > ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/upgrade.py > 43c7ff478b > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/datanode.py > c7b813f3c6 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_client.py > 87a6f52d61 > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py > 0805ff48dd > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py > 7f7e30cb7a > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/nfsgateway.py > 03a497a91c > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/snamenode.py > ac45ffd55e > > ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/zkfc_slave.py > db68544bda > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hcat_client.py > b37698e9cf > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_client.py > 3d9bfd7c3e > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py > 633e82ea56 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py > 8f7d06879a > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server_interactive.py > c9582a3965 > > ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/webhcat_server.py > da5e82b3bc > > ambari-server/src/main/resources/common-services/KAFKA/0.8.1/package/scripts/kafka_broker.py > 4512038b9c > > ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py > c91b394c6f > > ambari-server/src/main/resources/common-services/MAHOUT/1.0.0.2.3/package/scripts/mahout_client.py > 3486add01c > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py > 9bbca9ffed > > ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py > b3a8643562 > > ambari-server/src/main/resources/common-services/PIG/0.12.0.2.0/package/scripts/pig_client.py > 3233381b1b > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_admin.py > ecbacbffc3 > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/ranger_tagsync.py > a474e7641e > > ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/upgrade.py > a07a1fd952 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/kms_server.py > 829a998d71 > > ambari-server/src/main/resources/common-services/RANGER_KMS/0.5.0.2.3/package/scripts/upgrade.py > 8478bb8644 > > ambari-server/src/main/resources/common-services/SLIDER/0.60.0.2.2/package/scripts/slider_client.py > 08c856963d > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/job_history_server.py > 4fe2f54bfc > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/livy_server.py > 4ca9304a28 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_client.py > 31bf4c63e7 > > ambari-server/src/main/resources/common-services/SPARK/1.2.1/package/scripts/spark_thrift_server.py > 0c82e6fdef > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/job_history_server.py > d2b32aedee > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/livy2_server.py > cb4f5eee2a > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_client.py > 0a437507ed > > ambari-server/src/main/resources/common-services/SPARK2/2.0.0/package/scripts/spark_thrift_server.py > 6fdd32430c > > ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/sqoop_client.py > d420fab27e > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/drpc_server.py > d0af415199 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus.py > ab6dc3e5ba > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/nimbus_prod.py > 39bda4df67 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/pacemaker.py > 6da204eb70 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor.py > da900f93b6 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/supervisor_prod.py > d6c3545ef0 > > ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/ui_server.py > 5d0c30399c > > ambari-server/src/main/resources/common-services/SYSTEMML/0.10.0/package/scripts/systemml_client.py > 2d45b689a5 > > ambari-server/src/main/resources/common-services/TEZ/0.4.0.2.1/package/scripts/tez_client.py > 8018f0f960 > > ambari-server/src/main/resources/common-services/TITAN/1.0.0/package/scripts/titan_client.py > 9bb1aad67c > > ambari-server/src/main/resources/common-services/TITAN/1.0.0/package/scripts/titan_server.py > 5dcc7e9a45 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py > a299e25e46 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/historyserver.py > 36417a8506 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/mapreduce2_client.py > 424157b128 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/nodemanager.py > 7be7c36800 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py > ecd81478f1 > > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn_client.py > 4d65a404a2 > > ambari-server/src/main/resources/common-services/ZEPPELIN/0.6.0.2.5/package/scripts/master.py > ba46dc80df > > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_client.py > 1d4e89c33d > > ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/zookeeper_server.py > 0af0345532 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/configuration/cluster-env.xml > c6b091d189 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py > e9f22834b5 > > ambari-server/src/main/resources/stacks/HDP/2.0.6/properties/stack_select_packages.json > PRE-CREATION > > > Diff: https://reviews.apache.org/r/61146/diff/1/ > > > Testing > ------- > > PENDING > > > Thanks, > > Jonathan Hurley > >
