[GitHub] [hadoop] ndimiduk commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
ndimiduk commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-967711751 Paging @steveloughran @aw-was-here @busbey ... we'd like to ship this opentelemetry-agent.jar in the distribution but we don't want it on the class path. Can you advise on how we might manage the dependency in the pom, but relocate it for distribution? Maybe we add an invocation of the `dependency:copy` goal to somewhere in the dist profile and edit one of the `bin/dist-*-sitiching` scripts to move it into place? I imagine we'd want it in a new directory, maybe `share/hadoop/common/agents` ? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
ndimiduk commented on pull request #3445:
URL: https://github.com/apache/hadoop/pull/3445#issuecomment-966557663
Okay, this patch to your branch,
```
diff --git a/hadoop-client-modules/hadoop-client/pom.xml
b/hadoop-client-modules/hadoop-client/pom.xml
index dced359b286..82f38cc77ec 100644
--- a/hadoop-client-modules/hadoop-client/pom.xml
+++ b/hadoop-client-modules/hadoop-client/pom.xml
@@ -98,6 +98,10 @@
org.slf4j
slf4j-log4j12
+
+ io.opentelemetry.javaagent
+ opentelemetry-javaagent
+
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index 969a9a751d4..2da28614f45 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -380,6 +380,11 @@
opentelemetry-api
${opentelemetry.version}
+
+ io.opentelemetry.javaagent
+ opentelemetry-javaagent
+ all
+
```
Give me this result in the tarball
```
$ ls -1 share/hadoop/common/lib/ | grep opentel
opentelemetry-api-1.3.0.jar
opentelemetry-context-1.3.0.jar
opentelemetry-javaagent-1.3.0-all.jar
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
ndimiduk commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-966536367 Oh I see, you already have the dependency in the `hadoop-project/pom.xml`. I guess it doesn't get picked up from there during packaging. So my patch is not quite right either. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
ndimiduk commented on pull request #3445:
URL: https://github.com/apache/hadoop/pull/3445#issuecomment-966523434
This isn't right, but it's along the path,
```
diff --git a/hadoop-client-modules/hadoop-client/pom.xml
b/hadoop-client-modules/hadoop-client/pom.xml
index dced359b286..82f38cc77ec 100644
--- a/hadoop-client-modules/hadoop-client/pom.xml
+++ b/hadoop-client-modules/hadoop-client/pom.xml
@@ -98,6 +98,10 @@
org.slf4j
slf4j-log4j12
+
+ io.opentelemetry.javaagent
+ opentelemetry-javaagent
+
diff --git a/hadoop-common-project/hadoop-common/pom.xml
b/hadoop-common-project/hadoop-common/pom.xml
index 969a9a751d4..42fe7747bb3 100644
--- a/hadoop-common-project/hadoop-common/pom.xml
+++ b/hadoop-common-project/hadoop-common/pom.xml
@@ -380,6 +380,12 @@
opentelemetry-api
${opentelemetry.version}
+
+ io.opentelemetry.javaagent
+ opentelemetry-javaagent
+ ${opentelemetry-agent.version}
+ all
+
diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
index 76f39bf8590..1486ff8ef33 100644
--- a/hadoop-project/pom.xml
+++ b/hadoop-project/pom.xml
@@ -214,6 +214,7 @@
1.10.11
1.3.0
1.3.0
+1.0.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
ndimiduk commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-966519014 I think you need to add the otel-agent jar as a dependency to hadoop-common and exclude it as a dependency from hadoop-client. then it'll show up in `share/hadoop/common/lib`. That's still not quite right though, because we don't want the agent to be included in any runtime class path, only to be available to inclusion via `-javaagent:...`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
ndimiduk commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-965717655 @ArkenKiran where do you get the opentelemetry-agent jar for your testing? I've built this patch and it appears that the agent jar is not included. ``` $ mvn clean install -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true ... $ cd /tmp $ tar xzf ~/repos/apache/hadoop/hadoop-dist/target/hadoop-3.4.0-SNAPSHOT.tar.gz $ cd hadoop-3.4.0-SNAPSHOT $ ls -1 share/hadoop/common/lib/ | grep opentel opentelemetry-api-1.3.0.jar opentelemetry-context-1.3.0.jar ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ndimiduk commented on pull request #3445: HADOOP-15566 Opentelemetry changes using java agent
ndimiduk commented on pull request #3445: URL: https://github.com/apache/hadoop/pull/3445#issuecomment-964407226 That's great progress, @ArkenKiran ! You got there before I could. On the read path, any block cache miss will go to the hdfs client for a read (can still be local via short-circuit read). On the write path, we should see hdfs interaction via WAL writes and by MemStore flushes. WAL writes are inline with the client activity, MemStore flushes happen in the background. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
