[jira] [Resolved] (ATLAS-3926) Atlas1.2 with Kerberos problem
[ https://issues.apache.org/jira/browse/ATLAS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuyu resolved ATLAS-3926. - Resolution: Fixed The Kerberos principal name must start with "HTTP/...". For example: "HTTP/localhost@LOCALHOST". There is no default value. This is the cause of my problem. After modifying the principle of Kerberos, this problem was successfully solved. > Atlas1.2 with Kerberos problem > -- > > Key: ATLAS-3926 > URL: https://issues.apache.org/jira/browse/ATLAS-3926 > Project: Atlas > Issue Type: Bug > Components: falcon-integration, hive-integration >Affects Versions: 1.2.0 >Reporter: yuyu >Priority: Major > Fix For: 1.2.0 > > Attachments: image-2020-08-25-16-29-12-906.png, > image-2020-08-25-16-29-29-667.png, image-2020-08-25-16-30-34-221.png, > image-2020-08-25-16-31-23-439.png, image-2020-08-25-16-31-28-330.png, > image-2020-08-25-16-34-41-642.png, image-2020-08-25-16-35-09-836.png, > image-2020-08-25-16-38-45-645.png, image-2020-08-26-16-34-34-457.png > > > Hello, everyone. Recently, I encountered a problem with Kerberos when > deploying atlas.I tried to deploy atlas service on my CDH cluster with CDH > version 5.16 and Kerberos service deployed.My installation method is Atlas > 1.2 integrated CDH HBase and external 5.5.1 Solr > When I start the atlas service, I can successfully access the atlas web page > through my computer： > !image-2020-08-25-16-29-29-667.png! > But when I tried to import hive metadata information, something went wrong，as > shown in the figure below: > !image-2020-08-25-16-30-34-221.png! > !image-2020-08-25-16-38-45-645.png! > !image-2020-08-25-16-31-28-330.png! > It seems to indicate that I don't have a valid Kerberos principal? > Here is part of my atlas configuration file atlas-application.properties： > !image-2020-08-25-16-34-41-642.png! > !image-2020-08-25-16-35-09-836.png! > Can anyone tell me what the problem is,thanks. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (ATLAS-3936) How to use Atlas to manage the "hot" and "cold" degree of Hive table
[ https://issues.apache.org/jira/browse/ATLAS-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuyu updated ATLAS-3936: Component/s: (was: atlas-core) atlas-intg > How to use Atlas to manage the "hot" and "cold" degree of Hive table > > > Key: ATLAS-3936 > URL: https://issues.apache.org/jira/browse/ATLAS-3936 > Project: Atlas > Issue Type: Task > Components: atlas-intg, hive-integration >Affects Versions: 1.2.0 > Environment: Atlas 1.2 + CDH 5.16 (Hive 1.1.0) >Reporter: yuyu >Priority: Major > Fix For: 1.2.0 > > > Hello everyone, I have encountered some problems in using atlas. > I have a requirement: get the number of queries or the query frequency of > each Hive table, so that I can know how "cold" or "hot" the table is.This is > useful because it can help me manage Hive tables better.So my question is, > can Atlas help me with this work? What should I do?When I look at the source > code of this part of Atlas about Hive hook, I find that Atlas does not > operate on the Query statement? > Looking forward to your reply, thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3936) How to use Atlas to manage the "hot" and "cold" degree of Hive table
yuyu created ATLAS-3936: --- Summary: How to use Atlas to manage the "hot" and "cold" degree of Hive table Key: ATLAS-3936 URL: https://issues.apache.org/jira/browse/ATLAS-3936 Project: Atlas Issue Type: Task Components: atlas-core, hive-integration Affects Versions: 1.2.0 Environment: Atlas 1.2 + CDH 5.16 (Hive 1.1.0) Reporter: yuyu Fix For: 1.2.0 Hello everyone, I have encountered some problems in using atlas. I have a requirement: get the number of queries or the query frequency of each Hive table, so that I can know how "cold" or "hot" the table is.This is useful because it can help me manage Hive tables better.So my question is, can Atlas help me with this work? What should I do?When I look at the source code of this part of Atlas about Hive hook, I find that Atlas does not operate on the Query statement? Looking forward to your reply, thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ATLAS-3935) Use Audit framework to capture audit entries for Import/Export operations
Mandar Ambawane created ATLAS-3935: -- Summary: Use Audit framework to capture audit entries for Import/Export operations Key: ATLAS-3935 URL: https://issues.apache.org/jira/browse/ATLAS-3935 Project: Atlas Issue Type: Bug Components: atlas-core Reporter: Mandar Ambawane Assignee: Mandar Ambawane By using Audit Framework, capture audit entries for Import/Export operations. These audit entries can be seen under the "Audits" tab of Admin section. *Add Import and Export specific information in Audit entry* * Total no. of entities imported/ exported * Total no. of entities imported/ exported of specific TypeDef * Export operation Fetch Type -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (ATLAS-3930) Getting Multi exception while updating the dependency version.
[ https://issues.apache.org/jira/browse/ATLAS-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187689#comment-17187689 ] Gaurav Saini commented on ATLAS-3930: - we are trying to fix the variabilities by upgrading the version of the transitive dependency. here is the Github link. ( [https://github.com/crazylab/atlas/tree/cve-fix )|https://github.com/crazylab/atlas/tree/cve-fix)] we are not able to solve many of the variabilities and when we are upgrading the versions, it is breaking in many places. any help will be appreciated. > Getting Multi exception while updating the dependency version. > -- > > Key: ATLAS-3930 > URL: https://issues.apache.org/jira/browse/ATLAS-3930 > Project: Atlas > Issue Type: Bug > Components: atlas-core, atlas-intg, atlas-webui >Affects Versions: 2.1.0 >Reporter: Gaurav Saini >Priority: Blocker > Attachments: dependency-check-report.csv, dependency-check-report.html > > > we are working on apache atlas code and started deploying over > *[https://github.com/apache/atlas/tree/release-2.1.0-rc3]* > Upon scanning using twistlock, we found *180+* vulnerability. > > Out of these, Jackson-databind and netty_netty-all were the most occurring > once. > So, we tried upgrading the versions, but integration tests in atlas-webapp > start failing saying *"org.eclise.jetty, utils: Multi exception".* > The same thing is happening while upgrading versions of any other > dependencies in the atlas module. The application breaks for any other > dependency which we are trying to upgrade. for example, Hadoop_hdfs uses > Jackson-databind as a transitive dependency, hence I am unable to update > version. > _PFA of dependency check for the project._ > *I do not see any open issue on the Github channel too.* > *Have you experienced any such scenario while upgrading earlier?* > *Is there a way for me to move ahead to remove vulnerabilities in the > current version.* -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (ATLAS-3929) Kafka hook for Atlas to capture production/consumption
[ https://issues.apache.org/jira/browse/ATLAS-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viktor Somogyi-Vass reassigned ATLAS-3929: -- Assignee: Viktor Somogyi-Vass > Kafka hook for Atlas to capture production/consumption > -- > > Key: ATLAS-3929 > URL: https://issues.apache.org/jira/browse/ATLAS-3929 > Project: Atlas > Issue Type: Improvement > Components: kafka-integration >Reporter: Dennis Jaheruddin >Assignee: Viktor Somogyi-Vass >Priority: Major > > Currently we already have hooks for many engines (such as Spark, Nifi, > Flink). We also have a hook into the classical storage (HDFS). However, with > streaming becoming more and more dominant, we should also get a hook into the > streaming storage: Kafka. > Currently producers and consumers need to explicitly take atlas into account, > but for stronger governance and auditability, it should be possible to > capture every interaction with Kafka, regardless of how consumption or > production is done. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72815: Deletion of non existing hive entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72815/ --- (Updated Aug. 31, 2020, 8:51 a.m.) Review request for atlas, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Summary (updated) - Deletion of non existing hive entities Repository: atlas Description --- **Problem:** Whenever database or table is dropped in hive, and HiveHook is not enabled, we dont have anyway to get the database and table sync with hive. **Workaround:** Added support to delete hive entities in Atlas which are dropped in hive. **Usage:** ./import-hive.sh -deleteNonExisting Diffs - addons/hive-bridge/src/main/java/org/apache/atlas/hive/bridge/HiveMetaStoreBridge.java e659ca041 client/client-v2/src/main/java/org/apache/atlas/AtlasClientV2.java 6968e8358 Diff: https://reviews.apache.org/r/72815/diff/1/ Testing --- Manually tested Thanks, Pinal Shah
[jira] [Comment Edited] (ATLAS-3906) “NoSuchMethodErrors” due to multiple versions of org.apache.curator:curator-client:jar
[
https://issues.apache.org/jira/browse/ATLAS-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168688#comment-17168688
]
Bing-ok edited comment on ATLAS-3906 at 8/31/20, 8:33 AM:
--
[~madhan] Could you help me review this issue? Thx :P
was (Author: bing-ok):
[~kevalbhatt] Could you help me review this issue? Thx :P
> “NoSuchMethodErrors” due to multiple versions of
> org.apache.curator:curator-client:jar
> --
>
> Key: ATLAS-3906
> URL: https://issues.apache.org/jira/browse/ATLAS-3906
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Bing-ok
>Priority: Major
>
> Hi, there are multiple versions of _*org.apache.curator:curator-client*_ in
> _*atlas/server-api*_. As shown in the following dependency tree, according to
> Maven's “nearest wins” strategy, only
> _*org.apache.curator:curator-client:4.0.1*_ can be loaded,
> _*org.apache.curator:curator-client:2.12.0*_ will be shadowed.
> As _*org.apache.curator:curator-client:2.12.0*_ has not been loaded during
> the building process, several methods are missing. However, the missing
> methods:
> 1. _*org.codelibs.core.lang.StringUtil: java.lang.String
> newStringUnsafe(char[])*_
> {noformat}
> paths--
> getEntity(java.lang.String)> atlas\server-api\target\classes
> java.lang.Object get(java.lang.Object)>
> Repositories\org\apache\curator\curator-client\4.0.1\curator-client-4.0.1.jar
> apply(java.lang.Object)>
> Repositories\org\apache\curator\curator-framework\4.0.1\curator-framework-4.0.1.jar
> apply(org.apache.curator.framework.api.CuratorListener)>
> Repositories\org\apache\curator\curator-framework\4.0.1\curator-framework-4.0.1.jar
> checkInterrupted(java.lang.Throwable)>{noformat}
>
> The above missing methods are actually referenced by _*atlas/server-api*_,
> which will cause “NoSuchMethodErrors” at runtime.
> Suggested fixing solutions:
> 1. Upgrade dependency *_org.apache.hadoop:hadoop-common_* from _*3.1.1*_ to
> *_3.3.0_*. Because one conflicting library version
> _*org.apache.curator:curator-client:2.12.0*_ is transitively introduced by
> *_org.apache.hadoop:hadoop-common:3.1.1_*. Upgrading dependency
> *_org.apache.hadoop:hadoop-common_* from _*3.1.1*_ to *_3.3.0_* can solve
> this dependency conflict.
> 2. Directly declare dependency _*org.apache.curator:curator-client:2.12.0*_.
> 3. Use configuration attribute to unify the version of
> library _*org.apache.curator:curator-client*_ to be _*2.12.0*_ in
> _*atlas/server-api*_'s pom file.
> Please let me know which solution do you prefer? I can submit a PR to fix it.
> Thank you very much for your attention.
> Best regards,
> Dependency tree
> [INFO] org.apache.atlas:atlas-server-api:jar:3.0.0-SNAPSHOT
> [INFO] \- org.apache.hadoop:hadoop-common:jar:3.1.1:compile
> [INFO] +- org.apache.hadoop:hadoop-auth:jar:3.1.1:compile
> [INFO] | \- org.apache.curator:curator-framework:jar:4.0.1:compile (version
> managed from 2.12.0)
> [INFO] | \- _*(org.apache.curator:curator-client:jar:4.0.1:compile - version
> managed from 2.12.0; omitted for duplicate)*_
> [INFO] \- _*org.apache.curator:curator-client:jar:4.0.1:compile*_
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (ATLAS-3934) Dockerfile should build from local repository rather than pulling code from git
[
https://issues.apache.org/jira/browse/ATLAS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Madhan Neethiraj reassigned ATLAS-3934:
---
Assignee: Madhan Neethiraj
> Dockerfile should build from local repository rather than pulling code from
> git
> ---
>
> Key: ATLAS-3934
> URL: https://issues.apache.org/jira/browse/ATLAS-3934
> Project: Atlas
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Syed Atif Akhtar
>Assignee: Madhan Neethiraj
>Priority: Trivial
> Labels: atlas, build, development_environment, docker
>
> *Expected Behaviour:*
> When a developer wants to test out their changes using the dockerfile in dev
> support, they should be able to create the dockerfile with atlas built and
> mount from a local directory with the changes they have done.
> *Current Behaviour:*
> The dockerfile currently defined in dev-support pulls the source code from
> git instead of building the code that the user has defined
>
> {code:java}
> RUN git clone https://github.com/apache/atlas.git -b master{code}
>
> *Suggestion:*
> Maintain separate dockerfiles for publishing an image of Atlas vs development
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
