Hi Lionel, Thank you for the response, well this helped.
Cheers !! Thank you, Karan Gupta From: Lionel Liu <[email protected]> Sent: Friday, May 25, 2018 7:07 AM To: Karan Gupta <[email protected]> Cc: [email protected] Subject: Re: Unable to persist profiling results in HDFS Hi Karan, It looks good to me, but maybe a little issue. In the before, we use "evaluateRule" as the field name, but modified to "evaluate.rule" later. You need to check it in the code you are using, as this part: https://github.com/apache/incubator-griffin/blob/master/measure/src/main/scala/org/apache/griffin/measure/config/params/user/UserParam.scala#L30<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-griffin%2Fblob%2Fmaster%2Fmeasure%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fgriffin%2Fmeasure%2Fconfig%2Fparams%2Fuser%2FUserParam.scala%23L30&data=01%7C01%7Ckaran.gupta%40tavant.com%7Ccbb4ec3192ae42d82f1608d5c1e129b5%7Cc6c1e9da5d0c4f8f9a023c67206efbd6%7C0&sdata=Mhq%2Fck9tGeByO2bB7JuAMrPJMd9yz0fkPVWvBYHM%2B2s%3D&reserved=0> If this can not solve your issue, pls show me the application log. Thanks, Lionel On Thu, May 24, 2018 at 2:38 PM, Karan Gupta <[email protected]<mailto:[email protected]>> wrote: Hi Lionel, I created a custom config.json and defined a custom rule to run<dsl.type is spark-sql> and submitted it through spark. The spark submit runs fine without any issues. On HDFS, I could see the directory with the custom rule name but I am unable to see the _METRIC file where the results will be persisted, I only see a _START file. What am I missing here? HDFS -> ../persist/CheckAlphaNumeric/1527143022495/_START Config.json -> { "name": "CheckAlphaNumeric", "process.type": "batch", "data.sources": [ { "name": "src", "connectors": [ { "type": "hive", "version": "1.2", "config": { "database": "griffined", "table.name<https://apac01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ftable.name&data=01%7C01%7Ckaran.gupta%40tavant.com%7Ccbb4ec3192ae42d82f1608d5c1e129b5%7Cc6c1e9da5d0c4f8f9a023c67206efbd6%7C0&sdata=O9mIUQ7xaLRtXJEn56I9GgKtiRU3JghC8pELGQleM7Q%3D&reserved=0>": "check_table" } } ] } ], "evaluateRule": { "rules": [ { "dsl.type": "spark-sql", "dq.type": "profiling", "name": "checkalphnumeric", "rule": "SELECT count(name) FROM src WHERE name REGEXP '^[a-zA-Z0-9]+$'", "metric": { "name": "check_rules" } } ] } } ________________________________ Any comments or statements made in this email are not necessarily those of Tavant Technologies. The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you have received this in error, please contact the sender and delete the material from any computer. All emails sent from or to Tavant Technologies may be subject to our monitoring procedures.
