1: I guest it is related to your login user and super user. 2: Could you share with us your griffin log , I suspect some exception happened when try to connect with ES.
On Fri, Sep 11, 2020 at 5:14 PM Sunil Muniyal <[email protected]> wrote: > Hello William, > > Tried as suggested. > > 1. Ingested data into Hive tables using the provided script. > The ownership still show as is (Source with Admin and Target with Root) > > 2. Updated env-batch.json and env-streaming.json files with IP address for > ES and rebuilt Griffin. > Still no metrics for the jobs executed. > ES does have data as confirmed yesterday. > > Please help. > > Thanks and Regards, > Sunil Muniyal > > > On Thu, Sep 10, 2020 at 7:41 PM William Guo <[email protected]> wrote: > >> please enter ip directly. >> not sure whether hostname can be resolved correctly or not. >> >> On Thu, Sep 10, 2020 at 10:06 PM Sunil Muniyal < >> [email protected]> wrote: >> >>> Hi William, >>> >>> Thank you for the reply. >>> >>> Regarding points 2 and 3. Possible to share some more details. I believe >>> the env_batch.json is configured as it is expected. What exactly needs to >>> be updated correctly? ES Hostname or shall I enter IP or something else? >>> Please help. >>> >>> Thanks and Regards, >>> Sunil Muniyal >>> >>> >>> On Thu, Sep 10, 2020 at 7:30 PM William Guo <[email protected]> wrote: >>> >>>> 1 OK, We will fix this issue soon. >>>> 2 Could you try ping es from your spark environment and input ES >>>> endpoint correctly in env_batch.json >>>> 3 Please put your es endpoint in env_batch.json >>>> 6 Please try the following script to build your env. >>>> ``` >>>> >>>> #!/bin/bash >>>> #create table >>>> hive -f create-table.hqlecho "create table done" >>>> #current hoursudo ./gen_demo_data.shcur_date=`date >>>> +%Y%m%d%H`dt=${cur_date:0:8}hour=${cur_date:8:2}partition_date="dt='$dt',hour='$hour'" >>>> sed s/PARTITION_DATE/$partition_date/ ./insert-data.hql.template > >>>> insert-data.hql >>>> hive -f >>>> insert-data.hqlsrc_done_path=/griffin/data/batch/demo_src/dt=${dt}/hour=${hour}/_DONEtgt_done_path=/griffin/data/batch/demo_tgt/dt=${dt}/hour=${hour}/_DONE >>>> hadoop fs -mkdir -p /griffin/data/batch/demo_src/dt=${dt}/hour=${hour} >>>> hadoop fs -mkdir -p /griffin/data/batch/demo_tgt/dt=${dt}/hour=${hour} >>>> hadoop fs -touchz ${src_done_path} >>>> hadoop fs -touchz ${tgt_done_path}echo "insert data [$partition_date] done" >>>> #last hoursudo ./gen_demo_data.shcur_date=`date -d '1 hour ago' >>>> +%Y%m%d%H`dt=${cur_date:0:8}hour=${cur_date:8:2}partition_date="dt='$dt',hour='$hour'" >>>> sed s/PARTITION_DATE/$partition_date/ ./insert-data.hql.template > >>>> insert-data.hql >>>> hive -f >>>> insert-data.hqlsrc_done_path=/griffin/data/batch/demo_src/dt=${dt}/hour=${hour}/_DONEtgt_done_path=/griffin/data/batch/demo_tgt/dt=${dt}/hour=${hour}/_DONE >>>> hadoop fs -mkdir -p /griffin/data/batch/demo_src/dt=${dt}/hour=${hour} >>>> hadoop fs -mkdir -p /griffin/data/batch/demo_tgt/dt=${dt}/hour=${hour} >>>> hadoop fs -touchz ${src_done_path} >>>> hadoop fs -touchz ${tgt_done_path}echo "insert data [$partition_date] done" >>>> #next hoursset +ewhile truedo >>>> sudo ./gen_demo_data.sh >>>> cur_date=`date +%Y%m%d%H` >>>> next_date=`date -d "+1hour" '+%Y%m%d%H'` >>>> dt=${next_date:0:8} >>>> hour=${next_date:8:2} >>>> partition_date="dt='$dt',hour='$hour'" >>>> sed s/PARTITION_DATE/$partition_date/ ./insert-data.hql.template > >>>> insert-data.hql >>>> hive -f insert-data.hql >>>> src_done_path=/griffin/data/batch/demo_src/dt=${dt}/hour=${hour}/_DONE >>>> tgt_done_path=/griffin/data/batch/demo_tgt/dt=${dt}/hour=${hour}/_DONE >>>> hadoop fs -mkdir -p /griffin/data/batch/demo_src/dt=${dt}/hour=${hour} >>>> hadoop fs -mkdir -p /griffin/data/batch/demo_tgt/dt=${dt}/hour=${hour} >>>> hadoop fs -touchz ${src_done_path} >>>> hadoop fs -touchz ${tgt_done_path} >>>> echo "insert data [$partition_date] done" >>>> sleep 3600doneset -e >>>> >>>> >>>> >>>> Thanks, >>>> >>>> William >>>> >>>> >>>> On Thu, Sep 10, 2020 at 4:58 PM Sunil Muniyal < >>>> [email protected]> wrote: >>>> >>>>> 1. Since I was able to get ElasticSearch 6.8.x integrated, does it >>>>> mean that only ES upto 6.8.x is supported for Griffin as of now? If yes, >>>>> what are the plans further? Is there a page from which I could get >>>>> updates? >>>>> --please file a jira ticket for us to make our code ES compatible. >>>>> [SM] GRIFFIN-346 - Support for Elastic Search latest version (7.9.1) >>>>> <https://issues.apache.org/jira/browse/GRIFFIN-346> is submitted >>>>> >>>>> 2. I still do not see the metrics available (please refer below >>>>> screenshots). Though the measure is now listed in the drop down of *DQ >>>>> Metrics* tab. But when I selected the test measure, nothing came up. >>>>> --could you check the ES whether metrics have been injected or not. >>>>> [SM] I used the link below and got the index that is created in ES. I >>>>> believe the data is loaded. However, please correct if I >>>>> understood incorrectly >>>>> *"http://<ES Public IP>:9200/_cat/indices?v"* >>>>> --------------> POC env is on public cloud so using Public IP. >>>>> >>>>> health status index uuid pri rep docs.count >>>>> docs.deleted store.size pri.store.size >>>>> yellow open griffin ur_Kd3XFQBCsPzIM84j87Q 5 2 0 >>>>> 0 1.2kb 1.2kb >>>>> >>>>> >>>>> Docs in the index:* "http://<ES Public IP>:9200/griffin/_search"* >>>>> >>>>> {"took":44,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}} >>>>> >>>>> >>>>> Index Mapping: *"http://<ES Public IP>:9200/griffin"* >>>>> >>>>> {"griffin":{"aliases":{},"mappings":{"accuracy":{"properties":{"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"tmst":{"type":"date"}}}},"settings":{"index":{"creation_date":"1599567930578","number_of_shards":"5","number_of_replicas":"2","uuid":"ur_Kd3XFQBCsPzIM84j87Q","version":{"created":"6081299"},"provided_name":"griffin"}}}} >>>>> >>>>> >>>>> 3. At a step in deployment guide it is suggested to check URL: >>>>> "*http://<ES >>>>> HOST IP>:9200/griffin/accuracy"* When navigated to this URL, I get >>>>> below error. Please advise >>>>> *{"error":"Incorrect HTTP method for uri [/griffin/accuracy] and >>>>> method [GET], allowed: [POST]","status":405}* >>>>> *-- it seems you need to use POST method.* >>>>> [SM] I am using the POST method as suggested in the article. Below is >>>>> the JSON of *env_batch.JSON* >>>>> * {* >>>>> * "type": "ELASTICSEARCH",* >>>>> * "config": {* >>>>> * "method": "post",* >>>>> * "api": "http://<ES Host Name>:9200/griffin/accuracy", >>>>> ---------> *do we need IP here? >>>>> * "connection.timeout": "1m",* >>>>> * "retry": 10* >>>>> * }* >>>>> * }* >>>>> >>>>> 6. I also noticed that in Data Assets, *demo_src* is owned by Admin >>>>> whereas, *demo-tgt* by root. Would that make any difference? If yes, >>>>> how to correct it? Reload HIVE data? >>>>> -- could you show me your script for dataset setup? >>>>> <https://issues.apache.org/jira/browse/GRIFFIN-346> >>>>> [SM] Attached are the 3 scripts. gen-hive-data.sh is the master script >>>>> which triggers demo_data and it further triggers delta_src. >>>>> Have done it as it is instructed in the Github article and >>>>> gen-hive-data.sh is triggered as root in the terminal. >>>>> >>>>> >>>>> Please advise. >>>>> >>>>> Thanks and Regards, >>>>> Sunil Muniyal >>>>> >>>>> >>>>> On Wed, Sep 9, 2020 at 8:41 PM William Guo <[email protected]> wrote: >>>>> >>>>>> *Request you to please advise further on below points:* >>>>>> 1. Since I was able to get ElasticSearch 6.8.x integrated, does it >>>>>> mean that only ES upto 6.8.x is supported for Griffin as of now? If yes, >>>>>> what are the plans further? Is there a page from which I could get >>>>>> updates? >>>>>> --please file a jira ticket for us to make our code ES compatible. >>>>>> >>>>>> 2. I still do not see the metrics available (please refer below >>>>>> screenshots). Though the measure is now listed in the drop down of *DQ >>>>>> Metrics* tab. But when I selected the test measure, nothing came up. >>>>>> --could you check the ES whether metrics have been injected or not. >>>>>> >>>>>> 3. At a step in deployment guide it is suggested to check URL: http://<ES >>>>>> HOST IP>:9200/griffin/accuracy >>>>>> <http://13.126.127.141:9200/griffin/accuracy> When navigated to this >>>>>> URL, I get below error. Please advise >>>>>> *{"error":"Incorrect HTTP method for uri [/griffin/accuracy] and >>>>>> method [GET], allowed: [POST]","status":405}* >>>>>> *-- it seems you need to use POST method.* >>>>>> >>>>>> 6. I also noticed that in Data Assets, *demo_src* is owned by Admin >>>>>> whereas, *demo-tgt* by root. Would that make any difference? If yes, >>>>>> how to correct it? Reload HIVE data? >>>>>> >>>>>> -- could you show me your script for dataset setup? >>>>>> >>>>>> >>>>>> On Tue, Sep 8, 2020 at 9:02 PM Sunil Muniyal < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi William, >>>>>>> >>>>>>> I was finally able to get Griffin up and ElasticSearch integrated >>>>>>> along with Hadoop. Thanks a lot for your help and guidance so far. >>>>>>> >>>>>>> I have created a test measure and a job which gets triggered at >>>>>>> every 4 mins automatically (have referred to the user guide available on >>>>>>> GitHub at this link >>>>>>> <https://github.com/apache/griffin/blob/master/griffin-doc/ui/user-guide.md> >>>>>>> .) >>>>>>> >>>>>>> *Request you to please advise further on below points:* >>>>>>> 1. Since I was able to get ElasticSearch 6.8.x integrated, does it >>>>>>> mean that only ES upto 6.8.x is supported for Griffin as of now? If yes, >>>>>>> what are the plans further? Is there a page from which I could get >>>>>>> updates? >>>>>>> 2. I still do not see the metrics available (please refer below >>>>>>> screenshots). Though the measure is now listed in the drop down of *DQ >>>>>>> Metrics* tab. But when I selected the test measure, nothing came up. >>>>>>> 3. At a step in deployment guide it is suggested to check URL: >>>>>>> http://<ES >>>>>>> HOST IP>:9200/griffin/accuracy >>>>>>> <http://13.126.127.141:9200/griffin/accuracy> When navigated to >>>>>>> this URL, I get below error. Please advise >>>>>>> *{"error":"Incorrect HTTP method for uri [/griffin/accuracy] and >>>>>>> method [GET], allowed: [POST]","status":405}* >>>>>>> 6. I also noticed that in Data Assets, *demo_src* is owned by Admin >>>>>>> whereas, *demo-tgt* by root. Would that make any difference? If >>>>>>> yes, how to correct it? Reload HIVE data? >>>>>>> >>>>>>> *Screenshots:* >>>>>>> *Data Assets:* >>>>>>> [image: image.png] >>>>>>> >>>>>>> *DQ Metrics (Test Measure selected):* >>>>>>> [image: image.png] >>>>>>> >>>>>>> *Job Triggered multiple times:* >>>>>>> [image: image.png] >>>>>>> >>>>>>> *Metrics page from job directly:* >>>>>>> [image: image.png] >>>>>>> >>>>>>> Thanks and Regards, >>>>>>> Sunil Muniyal >>>>>>> >>>>>>> >>>>>>> On Tue, Sep 8, 2020 at 4:38 PM Sunil Muniyal < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I am unable to get repos for 6.4.1 instead I found 6.8.x. Will try >>>>>>>> with this version of Elastic Search in sometime. >>>>>>>> >>>>>>>> In the meantime, would it be possible to confirm if 6.4.x or 6.8.x >>>>>>>> is the only supported version for Griffin? Reason I am asking is, the >>>>>>>> GitHub article for griffin deployment points to the latest version of >>>>>>>> ES. >>>>>>>> >>>>>>>> Thanks and Regards, >>>>>>>> Sunil Muniyal >>>>>>>> >>>>>>>> >>>>>>>> On Tue, Sep 8, 2020 at 4:06 PM Sunil Muniyal < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> I will need to redeploy ElasticSearch, correct? >>>>>>>>> >>>>>>>>> Thanks and Regards, >>>>>>>>> Sunil Muniyal >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Sep 8, 2020 at 4:05 PM William Guo <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Could you try with this version? >>>>>>>>>> <elasticsearch.version>6.4.1</elasticsearch.version> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> William >>>>>>>>>> >>>>>>>>>> On Tue, Sep 8, 2020 at 5:59 PM Sunil Muniyal < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi William / Dev group, >>>>>>>>>>> >>>>>>>>>>> I have deployed ES 7.9 - latest version (single node) and the >>>>>>>>>>> same is configured. I also get the default page when hitting >>>>>>>>>>> http://<ES HOST IP>:9200/ >>>>>>>>>>> >>>>>>>>>>> Upon creating the griffin configurations using the JSON string >>>>>>>>>>> given >>>>>>>>>>> >>>>>>>>>>> curl -k -H "Content-Type: application/json" -X PUT http://<replaced >>>>>>>>>>> with my ES host IP>:9200/griffin \ >>>>>>>>>>> -d '{ >>>>>>>>>>> "aliases": {}, >>>>>>>>>>> "mappings": { >>>>>>>>>>> "accuracy": { >>>>>>>>>>> "properties": { >>>>>>>>>>> "name": { >>>>>>>>>>> "fields": { >>>>>>>>>>> "keyword": { >>>>>>>>>>> "ignore_above": 256, >>>>>>>>>>> "type": "keyword" >>>>>>>>>>> } >>>>>>>>>>> }, >>>>>>>>>>> "type": "text" >>>>>>>>>>> }, >>>>>>>>>>> "tmst": { >>>>>>>>>>> "type": "date" >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> }, >>>>>>>>>>> "settings": { >>>>>>>>>>> "index": { >>>>>>>>>>> "number_of_replicas": "2", >>>>>>>>>>> "number_of_shards": "5" >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> }' >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *I get below error:* >>>>>>>>>>> >>>>>>>>>>> *{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root >>>>>>>>>>> mapping definition has unsupported parameters: [accuracy : >>>>>>>>>>> {properties={name={fields={keyword={ignore_above=256, >>>>>>>>>>> type=keyword}}, >>>>>>>>>>> type=text}, >>>>>>>>>>> tmst={type=date}}}]"}],"type":"mapper_parsing_exception","reason":"Failed >>>>>>>>>>> to parse mapping [_doc]: Root mapping definition has unsupported >>>>>>>>>>> parameters: [accuracy : >>>>>>>>>>> {properties={name={fields={keyword={ignore_above=256, >>>>>>>>>>> type=keyword}}, >>>>>>>>>>> type=text}, >>>>>>>>>>> tmst={type=date}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root >>>>>>>>>>> mapping definition has unsupported parameters: [accuracy : >>>>>>>>>>> {properties={name={fields={keyword={ignore_above=256, >>>>>>>>>>> type=keyword}}, >>>>>>>>>>> type=text}, tmst={type=date}}}]"}},"status":400}* >>>>>>>>>>> >>>>>>>>>>> Seems like the JSON string is missing some values or is >>>>>>>>>>> incorrectly provided. >>>>>>>>>>> >>>>>>>>>>> Would be great if you could please help. >>>>>>>>>>> >>>>>>>>>>> Thanks and Regards, >>>>>>>>>>> Sunil Muniyal >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Sep 7, 2020 at 8:16 PM Sunil Muniyal < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thank you for the response, William. >>>>>>>>>>>> >>>>>>>>>>>> I have started preparing for ES deployment and should attempt >>>>>>>>>>>> the same tomorrow. >>>>>>>>>>>> >>>>>>>>>>>> In the meantime, I will also wait for the Dev team in case they >>>>>>>>>>>> have any additional inputs. >>>>>>>>>>>> >>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>> Sunil Muniyal >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Sep 7, 2020 at 8:06 PM William Guo <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> If dev confirms it to be mandatory, as I understand correct, I >>>>>>>>>>>>> will need to: >>>>>>>>>>>>> 1. Deploy and Configure ES >>>>>>>>>>>>> 2. Update application.properties to include ES details and >>>>>>>>>>>>> create ES index >>>>>>>>>>>>> 3. Rebuild Maven package and rerun the Griffin service >>>>>>>>>>>>> >>>>>>>>>>>>> *Right, you need to package es env configuration into your >>>>>>>>>>>>> jar.* >>>>>>>>>>>>> >>>>>>>>>>>>> There is no need to reload the data into Hadoop (Hive), >>>>>>>>>>>>> correct? >>>>>>>>>>>>> >>>>>>>>>>>>> *No* >>>>>>>>>>>>> >>>>>>>>>>>>> On a side note, is there any other documentation of Griffin >>>>>>>>>>>>> available or underway which would help to get below details while >>>>>>>>>>>>> integrating it with Cloudera Hadoop? >>>>>>>>>>>>> 1. What are the exact ports requirements (internal and >>>>>>>>>>>>> external)? >>>>>>>>>>>>> *check log and make sure all extra connections in properties >>>>>>>>>>>>> can accessible* >>>>>>>>>>>>> 2. Which all packages will be required? >>>>>>>>>>>>> *no* >>>>>>>>>>>>> 3. Any Java dependencies? >>>>>>>>>>>>> *java 1.8* >>>>>>>>>>>>> 4. If we have Cloudera Hadoop cluster kerberized (secured), >>>>>>>>>>>>> what are the dependencies or additional configurations needed? >>>>>>>>>>>>> *Should no extra dependencies, except those transitive >>>>>>>>>>>>> dependencies incurred by spark and hadoop.* >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Sep 7, 2020 at 6:42 PM Sunil Muniyal < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Ohh ok. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If dev confirms it to be mandatory, as I understand correct, >>>>>>>>>>>>>> I will need to: >>>>>>>>>>>>>> 1. Deploy and Configure ES >>>>>>>>>>>>>> 2. Update application.properties to include ES details and >>>>>>>>>>>>>> create ES index >>>>>>>>>>>>>> 3. Rebuild Maven package and rerun the Griffin service >>>>>>>>>>>>>> >>>>>>>>>>>>>> There is no need to reload the data into Hadoop (Hive), >>>>>>>>>>>>>> correct? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On a side note, is there any other documentation of Griffin >>>>>>>>>>>>>> available or underway which would help to get below details while >>>>>>>>>>>>>> integrating it with Cloudera Hadoop? >>>>>>>>>>>>>> 1. What are the exact ports requirements (internal and >>>>>>>>>>>>>> external)? >>>>>>>>>>>>>> 2. Which all packages will be required? >>>>>>>>>>>>>> 3. Any Java dependencies? >>>>>>>>>>>>>> 4. If we have Cloudera Hadoop cluster kerberized (secured), >>>>>>>>>>>>>> what are the dependencies or additional configurations needed? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I know some of the above information can be fetched from the >>>>>>>>>>>>>> deployment guide on Github. However, checking if any other formal >>>>>>>>>>>>>> documentation has been made available for the same? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>> Sunil Muniyal >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Sep 7, 2020 at 4:05 PM William Guo <[email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> cc dev for double checking. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Measure will emit metrics and store them in elastic, UI >>>>>>>>>>>>>>> fetch those metrics from elastic. >>>>>>>>>>>>>>> So elastic should be mandatory. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> William >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Sep 7, 2020 at 6:32 PM Sunil Muniyal < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you for the quick response, William. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have not configured ElasticSearch since it is not >>>>>>>>>>>>>>>> deployed. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In the application.properties, I just added the dummy >>>>>>>>>>>>>>>> information (as below) just to pass the validation test and >>>>>>>>>>>>>>>> get Griffin up >>>>>>>>>>>>>>>> and running. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # elasticsearch >>>>>>>>>>>>>>>> # elasticsearch.host = <IP> >>>>>>>>>>>>>>>> # elasticsearch.port = <elasticsearch rest port> >>>>>>>>>>>>>>>> # elasticsearch.user = user >>>>>>>>>>>>>>>> # elasticsearch.password = password >>>>>>>>>>>>>>>> elasticsearch.host=localhost >>>>>>>>>>>>>>>> elasticsearch.port=9200 >>>>>>>>>>>>>>>> elasticsearch.scheme=http >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is ElasticSearch a mandatory requirement to use Griffin? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>>> Sunil Muniyal >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Sep 7, 2020 at 3:58 PM William Guo < >>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Could you check whether ES has been injected with those >>>>>>>>>>>>>>>>> metrics or not? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Mon, Sep 7, 2020 at 6:23 PM Sunil Muniyal < >>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello William, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I was able to bypass this error by entering the default >>>>>>>>>>>>>>>>>> field values for LDAP, ElasticSearch and Livy in >>>>>>>>>>>>>>>>>> application.properties and >>>>>>>>>>>>>>>>>> successfully get Griffin running. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> By following the below article, I have created a test >>>>>>>>>>>>>>>>>> measure and then a job which triggers that measure. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://github.com/apache/griffin/blob/master/griffin-doc/ui/user-guide.md >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Have allowed the job to get triggered multiple times, >>>>>>>>>>>>>>>>>> however, still i can't see anything in metrics related to >>>>>>>>>>>>>>>>>> the job. Neither >>>>>>>>>>>>>>>>>> I see anything in *health *or *mydashboard* tabs. Also, >>>>>>>>>>>>>>>>>> if you notice in the screenshot below, being in the *DQ >>>>>>>>>>>>>>>>>> Metrics* tab, I still do not see the created measure in >>>>>>>>>>>>>>>>>> the drop down list. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [image: image.png] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Test job executed multiple times:* >>>>>>>>>>>>>>>>>> [image: image.png] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Please advise if anything is mis-configured. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>>>>> Sunil Muniyal >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Mon, Sep 7, 2020 at 12:40 PM Sunil Muniyal < >>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hello William, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thank you for the reply. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> This helped, actually i had missed to add the property >>>>>>>>>>>>>>>>>>> in application.properties. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Now the other challenge is, along with ES and Livy, I am >>>>>>>>>>>>>>>>>>> also not using LDAP and it is hitting the error *unable >>>>>>>>>>>>>>>>>>> to resolve ldap.url property.* Of Course it will, since >>>>>>>>>>>>>>>>>>> the property is not configured. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Please suggest. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>>>>>> Sunil Muniyal >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Sun, Sep 6, 2020 at 7:26 PM William Guo < >>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> hi Sunil Muniyal, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Could you check this property in your griffin >>>>>>>>>>>>>>>>>>>> properties file? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> internal.event.listeners >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> William >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Thu, Sep 3, 2020 at 11:05 PM Sunil Muniyal < >>>>>>>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I am attempting to integrate Griffin with Cloudera >>>>>>>>>>>>>>>>>>>>> Hadoop by following below article: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> https://github.com/apache/griffin/blob/master/griffin-doc/deploy/deploy-guide.md >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> <https://github.com/apache/griffin/blob/master/griffin-doc/deploy/deploy-guide.md>I >>>>>>>>>>>>>>>>>>>>> have followed everything as instructed, apart from below >>>>>>>>>>>>>>>>>>>>> things: >>>>>>>>>>>>>>>>>>>>> 1. Using Cloudera Hadoop 5.15 and relevant >>>>>>>>>>>>>>>>>>>>> configurations instead of Apache Hadoop >>>>>>>>>>>>>>>>>>>>> 2. Not using Elastic search as it is not applicable >>>>>>>>>>>>>>>>>>>>> 3. Did not use Livy as it is not applicable. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Maven build is successful and has got 2 jars at >>>>>>>>>>>>>>>>>>>>> service/target and measure/target which I have uploaded >>>>>>>>>>>>>>>>>>>>> to HDFS. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> However, *starting griffin-service.jar using nohup >>>>>>>>>>>>>>>>>>>>> command* is failing with below error: >>>>>>>>>>>>>>>>>>>>> *Caused by: java.lang.IllegalArgumentException: Could >>>>>>>>>>>>>>>>>>>>> not resolve placeholder 'internal.event.listeners' in >>>>>>>>>>>>>>>>>>>>> string value >>>>>>>>>>>>>>>>>>>>> "#{'${internal.event.listeners}'.split(',')}"* >>>>>>>>>>>>>>>>>>>>> * at >>>>>>>>>>>>>>>>>>>>> org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:174) >>>>>>>>>>>>>>>>>>>>> ~[spring-core-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]* >>>>>>>>>>>>>>>>>>>>> * at >>>>>>>>>>>>>>>>>>>>> org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:126) >>>>>>>>>>>>>>>>>>>>> ~[spring-core-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]* >>>>>>>>>>>>>>>>>>>>> * at >>>>>>>>>>>>>>>>>>>>> org.springframework.core.env.AbstractPropertyResolver.doResolvePlaceholders(AbstractPropertyResolver.java:236) >>>>>>>>>>>>>>>>>>>>> ~[spring-core-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]* >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I have tried to search a lot of articles with no luck. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Would be great if someone could help me to fix this. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Also, attached is the output of nohup command that was >>>>>>>>>>>>>>>>>>>>> written in service.out. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks and Regards, >>>>>>>>>>>>>>>>>>>>> Sunil Muniyal >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: >>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: >>>>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [email protected] >>>>> For additional commands, e-mail: [email protected] >>>> >>>>
