[GitHub] [atlas] mneethiraj commented on a change in pull request #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
mneethiraj commented on a change in pull request #39: ATLAS-3071:Add
Functionalities to Collect Notification Metrics
URL: https://github.com/apache/atlas/pull/39#discussion_r272449826
##
File path:
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsCounter.java
##
@@ -0,0 +1,255 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.model.metrics;
+
+import com.fasterxml.jackson.annotation.JsonAutoDetect;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility.NONE;
+import static
com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility.PUBLIC_ONLY;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Atlas statistics
+ */
+@JsonAutoDetect(getterVisibility = PUBLIC_ONLY, setterVisibility =
PUBLIC_ONLY, fieldVisibility = NONE)
+@JsonSerialize(include = JsonSerialize.Inclusion.ALWAYS)
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class AtlasMetricsCounter {
+private static final Logger LOG =
LoggerFactory.getLogger(AtlasMetricsCounter.class);
+
+public static final String STAT_SERVER_START_TS=
"serverStartTimeStamp";
+public static final String STAT_SERVER_ACTIVE_TS =
"serverActiveTimeStamp";
+public static final String STAT_SERVER_UP_SINCE=
"serverUpTime";
+public static final String STAT_START_OFFSET =
"KafkaTopic:ATLAS_HOOK:startOffset";
+public static final String STAT_CURRENT_OFFSET =
"KafkaTopic:ATLAS_HOOK:currentOffset";
+public static final String STAT_SOLR_STATUS=
"solrConnectionStatus";
+public static final String STAT_HBASE_STATUS =
"HBaseConnectionStatus";
+public static final String STAT_LAST_MESSAGE_PROCESSED_TIME_TS =
"lastMessageProcessedTimeStamp";
+public static final String STAT_AVG_MESSAGE_PROCESSING_TIME=
"avgMessageProcessingTime";
+
+private static final long HOUR = 1000 * 60 * 60;
+private static final long DAY= 1000 * 60 * 60 * 24;
+private static final long TWO_HOURS = 1000 * 60 * 60 * 2;
+private static final long TWO_DAYS = 1000 * 60 * 60 * 24 * 2;
+private static final int METRIC_WINDOW_SIZE = 48;
+
+private final SimpleDateFormat metricWindowFormat = new
SimpleDateFormat("d MMM, : HH");
+private final SimpleDateFormat simpleDateFormat = new
SimpleDateFormat("d MMM, : hh:mm aaa z");
+
+private List> metricWindow;
+private Map data = new HashMap<>();
+private String[] timestamp;
+
+private int totalSinceLastStart = 0;
+
+public enum MetricsCounterType {
+NOTIFICATION_PROCESSED,
+NOTIFICATION_FAILED,
+ENTITY_CREATED,
+ENTITY_UPDATED,
+ENTITY_DELETE,
+TOTAL_MSG_COUNT
+}
+
+public enum MetricsTimeType {
+THIS_HOUR,
+PAST_HOUR,
+TODAY,
+YESTERDAY,
+TOTAL_MSG_COUNT
+}
+
+public AtlasMetricsCounter() {
+initTotalCount();
+initMetricWindow();
+}
+
+public void setData(Map data) {
+this.data = data;
+}
+
+public Map getData() {
+return data;
+}
+
+@Override
+public String toString() {
+return "AtlasMetricsCounter{" + "data=" + data + '}';
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(data);
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+AtlasMetricsCounter other = (AtlasMetricsCounter) o;
+
+return Objects.equals(this.data, other.data);
+}
+
+public void updateMetricWindow(MetricsCounterType counterType, long
timestamp) {
Review comment:
[GitHub] [atlas] mneethiraj commented on a change in pull request #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
mneethiraj commented on a change in pull request #39: ATLAS-3071:Add
Functionalities to Collect Notification Metrics
URL: https://github.com/apache/atlas/pull/39#discussion_r272451957
##
File path: repository/src/main/java/org/apache/atlas/util/AtlasMetricsUtil.java
##
@@ -251,4 +257,48 @@ private String millisToTimeDiff(long msDiff) {
private String millisToTimeStamp(long ms) {
return simpleDateFormat.format(ms);
}
+
+public void setSuccessfulMetrics(HookNotification.HookNotificationType
type, long timestamp) {
Review comment:
Consider replacing methods setSuccessfulMetrics() and setFailedMetrics()
with the following method:
public void onNotificationProcessingComplete(HookNotificationType type,
boolean isSuccess) {
long startTimeMs = RequestContext.get().getRequestTime();
long timeTakenMs = System.currentTimeMillis() - startTimeMs;
// ..
}
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [atlas] mneethiraj commented on a change in pull request #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
mneethiraj commented on a change in pull request #39: ATLAS-3071:Add
Functionalities to Collect Notification Metrics
URL: https://github.com/apache/atlas/pull/39#discussion_r272452216
##
File path:
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricsCounter.java
##
@@ -0,0 +1,255 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.model.metrics;
+
+import com.fasterxml.jackson.annotation.JsonAutoDetect;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+
+import static com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility.NONE;
+import static
com.fasterxml.jackson.annotation.JsonAutoDetect.Visibility.PUBLIC_ONLY;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Atlas statistics
+ */
+@JsonAutoDetect(getterVisibility = PUBLIC_ONLY, setterVisibility =
PUBLIC_ONLY, fieldVisibility = NONE)
+@JsonSerialize(include = JsonSerialize.Inclusion.ALWAYS)
+@JsonIgnoreProperties(ignoreUnknown = true)
+public class AtlasMetricsCounter {
+private static final Logger LOG =
LoggerFactory.getLogger(AtlasMetricsCounter.class);
+
+public static final String STAT_SERVER_START_TS=
"serverStartTimeStamp";
+public static final String STAT_SERVER_ACTIVE_TS =
"serverActiveTimeStamp";
+public static final String STAT_SERVER_UP_SINCE=
"serverUpTime";
+public static final String STAT_START_OFFSET =
"KafkaTopic:ATLAS_HOOK:startOffset";
+public static final String STAT_CURRENT_OFFSET =
"KafkaTopic:ATLAS_HOOK:currentOffset";
+public static final String STAT_SOLR_STATUS=
"solrConnectionStatus";
+public static final String STAT_HBASE_STATUS =
"HBaseConnectionStatus";
+public static final String STAT_LAST_MESSAGE_PROCESSED_TIME_TS =
"lastMessageProcessedTimeStamp";
+public static final String STAT_AVG_MESSAGE_PROCESSING_TIME=
"avgMessageProcessingTime";
+
+private static final long HOUR = 1000 * 60 * 60;
+private static final long DAY= 1000 * 60 * 60 * 24;
+private static final long TWO_HOURS = 1000 * 60 * 60 * 2;
+private static final long TWO_DAYS = 1000 * 60 * 60 * 24 * 2;
+private static final int METRIC_WINDOW_SIZE = 48;
+
+private final SimpleDateFormat metricWindowFormat = new
SimpleDateFormat("d MMM, : HH");
+private final SimpleDateFormat simpleDateFormat = new
SimpleDateFormat("d MMM, : hh:mm aaa z");
+
+private List> metricWindow;
+private Map data = new HashMap<>();
+private String[] timestamp;
+
+private int totalSinceLastStart = 0;
+
+public enum MetricsCounterType {
+NOTIFICATION_PROCESSED,
+NOTIFICATION_FAILED,
+ENTITY_CREATED,
+ENTITY_UPDATED,
+ENTITY_DELETE,
+TOTAL_MSG_COUNT
+}
+
+public enum MetricsTimeType {
+THIS_HOUR,
+PAST_HOUR,
+TODAY,
+YESTERDAY,
+TOTAL_MSG_COUNT
+}
+
+public AtlasMetricsCounter() {
+initTotalCount();
+initMetricWindow();
+}
+
+public void setData(Map data) {
+this.data = data;
+}
+
+public Map getData() {
+return data;
+}
+
+@Override
+public String toString() {
+return "AtlasMetricsCounter{" + "data=" + data + '}';
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(data);
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) return true;
+if (o == null || getClass() != o.getClass()) return false;
+AtlasMetricsCounter other = (AtlasMetricsCounter) o;
+
+return Objects.equals(this.data, other.data);
+}
+
+public void updateMetricWindow(MetricsCounterType counterType, long
timestamp) {
+if
[GitHub] [atlas] sarathkumarsubramanian edited a comment on issue #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
sarathkumarsubramanian edited a comment on issue #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics URL: https://github.com/apache/atlas/pull/39#issuecomment-480151595 The changes looks good. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [atlas] sarathkumarsubramanian commented on issue #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
sarathkumarsubramanian commented on issue #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics URL: https://github.com/apache/atlas/pull/39#issuecomment-480151595 Other changes looks good. +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [atlas] sarathkumarsubramanian commented on a change in pull request #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
sarathkumarsubramanian commented on a change in pull request #39:
ATLAS-3071:Add Functionalities to Collect Notification Metrics
URL: https://github.com/apache/atlas/pull/39#discussion_r272323199
##
File path:
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricCounter.java
##
@@ -0,0 +1,197 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.model.metrics;
+
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AtlasMetricCounter {
Review comment:
Since this class deals with notification metrics, consider renaming to
AtlasNotificationMetrics
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [atlas] sarathkumarsubramanian commented on a change in pull request #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
sarathkumarsubramanian commented on a change in pull request #39:
ATLAS-3071:Add Functionalities to Collect Notification Metrics
URL: https://github.com/apache/atlas/pull/39#discussion_r272324265
##
File path:
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricCounter.java
##
@@ -0,0 +1,197 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.model.metrics;
+
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AtlasMetricCounter {
+
+private static final Logger LOG =
LoggerFactory.getLogger(AtlasMetricCounter.class);
+
+private static final int METRIC_WINDOW_SIZE = 48;
+private static final long HOUR = 1000 * 60 * 60;
Review comment:
consider suffixing MS (to denote milliseconds), HOUR_MS, DAY_MS,
TWO_HOURS_MS, TWO_DAYS_MS
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [atlas] sarathkumarsubramanian commented on a change in pull request #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
sarathkumarsubramanian commented on a change in pull request #39:
ATLAS-3071:Add Functionalities to Collect Notification Metrics
URL: https://github.com/apache/atlas/pull/39#discussion_r272324531
##
File path:
intg/src/main/java/org/apache/atlas/model/metrics/AtlasMetricCounter.java
##
@@ -0,0 +1,197 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.model.metrics;
+
+import java.text.SimpleDateFormat;
+import java.util.*;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class AtlasMetricCounter {
+
+private static final Logger LOG =
LoggerFactory.getLogger(AtlasMetricCounter.class);
+
+private static final int METRIC_WINDOW_SIZE = 48;
+private static final long HOUR = 1000 * 60 * 60;
+private static final long DAY = 1000 * 60 * 60 * 24;
+private static final long TWO_HOURS = 1000 * 60 * 60 * 2;
+private static final long TWO_DAYS = 1000 * 60 * 60 * 24 * 2;
+
+private final SimpleDateFormat metricWindowFormat = new
SimpleDateFormat("d MMM, : HH");
+private final SimpleDateFormat simpleDateFormat = new
SimpleDateFormat("d MMM, : hh:mm aaa z");
+
+private List> metricWindow;
+private String[] timestamp;
+private int totalSinceLastStart = 0;
+
+public enum MetricCounterType {
Review comment:
change all reference from "metric" to "metrics"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [atlas] sarathkumarsubramanian commented on a change in pull request #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
sarathkumarsubramanian commented on a change in pull request #39:
ATLAS-3071:Add Functionalities to Collect Notification Metrics
URL: https://github.com/apache/atlas/pull/39#discussion_r272321586
##
File path:
repository/src/main/java/org/apache/atlas/util/AtlasMetricCounterUtil.java
##
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.atlas.util;
+
+import org.apache.atlas.model.metrics.AtlasMetricCounter;
+import org.apache.atlas.model.notification.HookNotification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.stereotype.Component;
+
+import java.util.Map;
+
+import javax.inject.Inject;
+
+@Component
+public class AtlasMetricCounterUtil {
+
+private static final Logger LOG =
LoggerFactory.getLogger(AtlasMetricCounterUtil.class);
Review comment:
unused LOG, consider removing it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [atlas] sarathkumarsubramanian commented on a change in pull request #39: ATLAS-3071:Add Functionalities to Collect Notification Metrics
sarathkumarsubramanian commented on a change in pull request #39:
ATLAS-3071:Add Functionalities to Collect Notification Metrics
URL: https://github.com/apache/atlas/pull/39#discussion_r272443590
##
File path:
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
##
@@ -685,6 +686,12 @@ void handleMessage(AtlasKafkaMessage
kafkaMsg) throws AtlasSer
}
}
+if (isFailedMsg) {
Review comment:
689-693 should be moved to finally block below after kafka commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
Re: Review Request 70359: ATLAS-3054: preprocess updated to add container entity reference in moved entities
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70359/ --- (Updated April 5, 2019, 12:52 a.m.) Review request for atlas, Ashutosh Mestry, keval bhatt, Kapildeo Nayak, Nikhil Bonte, Nixon Rodrigues, and Sarath Subramanian. Changes --- updated AtlasArrayType.areEqualValues() to handle the differences in LIST and SET cardinality Bugs: ATLAS-3054 https://issues.apache.org/jira/browse/ATLAS-3054 Repository: atlas Description --- To enable batch-processing of entities in large notification message, ownedRef attributes are removed. Sometimes ownedRef entities may not have attribute that points to container entity (for example hive_column.table attribute). This patch addressed this case by adding the missing attribute that points to the contianer entity. Diffs (updated) - intg/src/main/java/org/apache/atlas/type/AtlasArrayType.java 6147eee91 intg/src/main/java/org/apache/atlas/type/AtlasEntityType.java 1ce776ee3 intg/src/main/java/org/apache/atlas/type/AtlasRelationshipType.java 585d176ca intg/src/main/java/org/apache/atlas/type/AtlasStructType.java fb24df0f3 intg/src/main/java/org/apache/atlas/type/AtlasTypeUtil.java 079a8fc1f intg/src/test/java/org/apache/atlas/type/TestAtlasStructType.java a37dd46b1 repository/src/main/java/org/apache/atlas/repository/converters/AtlasStructFormatConverter.java 173fcee65 repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStream.java d12b03610 webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 48355c941 webapp/src/main/java/org/apache/atlas/notification/preprocessor/EntityPreprocessor.java 085e746b4 webapp/src/main/java/org/apache/atlas/notification/preprocessor/HivePreprocessor.java 0b93658fd webapp/src/main/java/org/apache/atlas/notification/preprocessor/PreprocessorContext.java c85c1b86a webapp/src/main/java/org/apache/atlas/notification/preprocessor/RdbmsPreprocessor.java adc19836a Diff: https://reviews.apache.org/r/70359/diff/5/ Changes: https://reviews.apache.org/r/70359/diff/4-5/ Testing --- - precommit test run: https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1011/ Thanks, Madhan Neethiraj
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: (was: ATLAS-3071.patch)
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
>Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (ATLAS-3071) Add Functionalities to Collect Notification Metrics/Entity Lifecyle
[
https://issues.apache.org/jira/browse/ATLAS-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Le Ma updated ATLAS-3071:
-
Attachment: ATLAS-3071.patch
> Add Functionalities to Collect Notification Metrics/Entity Lifecyle
> ---
>
> Key: ATLAS-3071
> URL: https://issues.apache.org/jira/browse/ATLAS-3071
> Project: Atlas
> Issue Type: New Feature
>Reporter: Le Ma
>Assignee: Le Ma
>Priority: Major
> Attachments: ATLAS-3071.patch, MetricsDataModel.json
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Extend api/atlas/admin/metrics to provide metrics for notifications/entity
> lifycycle:
> # notification
> processed
> # notification
> failed
> today/thisHour/pastHour/total { # entity created
> # entity updated
> # entity deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ATLAS-3080) Impala should send notification for DDL operation to Atlas through Kafka
[ https://issues.apache.org/jira/browse/ATLAS-3080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810321#comment-16810321 ] Dinesh Garg commented on ATLAS-3080: Per our discussion toda,y I assume this wont be a blocker anymore. This is a requirement for the long term not for CDP 1.0 For CDP 1.0, we are going to explore the option of making the DDL operation info available in Lineage file. Some tool in Atlas will pick it from there and put it in Kafka. > Impala should send notification for DDL operation to Atlas through Kafka > > > Key: ATLAS-3080 > URL: https://issues.apache.org/jira/browse/ATLAS-3080 > Project: Atlas > Issue Type: Task > Components: atlas-intg >Affects Versions: 1.1.0 >Reporter: Na Li >Priority: Blocker > > if user needs Tag based security policy for tables in Impala, we need the > table metadata in Atlas, like assigning a PII tag for an Impala table in > atlas and enforce security policy in Ranger for the tag ‘PII’ on who can > access/modify it. > Atlas does not allow user to configure tag on a table if Atlas has not > received notification about that table creation. Right now, Impala does not > send notifications for DDL operation. Therefore, we need to add Atlas code to > integrate with Impala. > Besides this use case, sending notifications to Atlas allow other > functionalities to consume Impala DDL operation activities. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3116) Fix LDAP login, once a sign-on is made through knox SSO.
Nixon Rodrigues created ATLAS-3116: -- Summary: Fix LDAP login, once a sign-on is made through knox SSO. Key: ATLAS-3116 URL: https://issues.apache.org/jira/browse/ATLAS-3116 Project: Atlas Issue Type: Bug Reporter: Nixon Rodrigues Assignee: Nixon Rodrigues -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3115) Enhancement for Basic search API for LIST/SET attribute type
[ https://issues.apache.org/jira/browse/ATLAS-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikhil Bonte updated ATLAS-3115: Issue Type: Improvement (was: Bug) > Enhancement for Basic search API for LIST/SET attribute type > > > Key: ATLAS-3115 > URL: https://issues.apache.org/jira/browse/ATLAS-3115 > Project: Atlas > Issue Type: Improvement >Reporter: Nixon Rodrigues >Assignee: Nikhil Bonte >Priority: Major > > Before ATLAS-3112, classifications list were not part of indexed entity as it > was indexed separately. > ATLAS-3112 indexed entity itself contains list of classifications associated. > Basic search now can be modified to use this while searching by > classification name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ATLAS-3115) Enhancement for Basic search API for LIST/SET attribute type
[ https://issues.apache.org/jira/browse/ATLAS-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikhil Bonte updated ATLAS-3115: Description: Before ATLAS-3112, classifications list were not part of indexed entity as it was indexed separately. ATLAS-3112 indexed entity itself contains list of classifications associated. Basic search now can be modified to use this while searching by classification name. > Enhancement for Basic search API for LIST/SET attribute type > > > Key: ATLAS-3115 > URL: https://issues.apache.org/jira/browse/ATLAS-3115 > Project: Atlas > Issue Type: Bug >Reporter: Nixon Rodrigues >Assignee: Nikhil Bonte >Priority: Major > > Before ATLAS-3112, classifications list were not part of indexed entity as it > was indexed separately. > ATLAS-3112 indexed entity itself contains list of classifications associated. > Basic search now can be modified to use this while searching by > classification name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ATLAS-3115) Enhancement for Basic search API for LIST/SET attribute type
Nixon Rodrigues created ATLAS-3115: -- Summary: Enhancement for Basic search API for LIST/SET attribute type Key: ATLAS-3115 URL: https://issues.apache.org/jira/browse/ATLAS-3115 Project: Atlas Issue Type: Bug Reporter: Nixon Rodrigues Assignee: Nikhil Bonte -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (ATLAS-2960) Custom attribute inside an Attibute Definition
[
https://issues.apache.org/jira/browse/ATLAS-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nixon Rodrigues resolved ATLAS-2960.
Resolution: Information Provided
> Custom attribute inside an Attibute Definition
> ---
>
> Key: ATLAS-2960
> URL: https://issues.apache.org/jira/browse/ATLAS-2960
> Project: Atlas
> Issue Type: Wish
> Components: atlas-core
>Reporter: Mayur Maheshwari
>Priority: Major
>
> How we can add custom attributes while creating a Type using Rest API:
> {
> "enumTypes": [],
> "structTypes": [],
> "traitTypes": [],
> "classTypes": [{
> "superTypes": ["Process"],
> "hierarchicalMetaTypeName":
> "org.apache.atlas.typesystem.types.ClassType",
> "typeName": "ResearchPaperAccessDataset_Process",
> "typeDescription": null,
> "attributeDefinitions": [{
> "name": "resourceSetID",
> "dataTypeName": "int",
> "multiplicity": "required",
> "isComposite": false,
> "isUnique": false,
> "isIndexable": true,
> "reverseAttributeName": null,
> *"sources":[{*
> *"name":"hdfs",*
> *"originalName":"ID"*
> *}*
> *]*
> },
> {
> "name": "researchPaperGroupName",
> "dataTypeName": "string",
> "multiplicity": "required",
> "isComposite": false,
> "isUnique": false,
> "isIndexable": true,
> "reverseAttributeName": null
> }]
> }]
> }
>
> Wanted to add '*sources*' marked in bold while creating a type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
