[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/storm/pull/2475


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[hmcc]

Github user hmcc commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162613466
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/utils/DefaultShellLogHandler.java ---
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handle output from non-JVM processes.
+ */
+public class DefaultShellLogHandler implements ShellLogHandler {
+public static final Logger LOG = 
LoggerFactory.getLogger(DefaultShellLogHandler.class);
--- End diff --

Yep, makes sense - thanks!


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162317935
  
--- Diff: storm-client/test/resources/log4j2-test.xml ---
@@ -0,0 +1,32 @@
+
+
+
+
+
+
+
+
+
+
+
--- End diff --

it shouldn't be o.a.s.u.ShellLogHandler, and you would be OK to not add 
this if new tests doesn't produce huge logs.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162313151
  
--- Diff: storm-client/src/jvm/org/apache/storm/task/ShellBolt.java ---
@@ -145,6 +148,8 @@ public void prepare(Map topoConf, 
TopologyContext context,
 workerTimeoutMills = 1000 * 
ObjectReader.getInt(topoConf.get(Config.SUPERVISOR_WORKER_TIMEOUT_SECS));
 }
 
+_logHandler = ShellUtils.getLogHandler(topoConf);
--- End diff --

This should be also placed to spout as I commented above.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162313649
  
--- Diff: storm-client/src/jvm/org/apache/storm/spout/ShellSpout.java ---
@@ -177,6 +179,7 @@ private void handleMetrics(ShellMsg shellMsg) {
 private void querySubprocess() {
 try {
 markWaitingSubprocess();
+_logHandler.setUpContext(_process, _context);
--- End diff --

This can be on-time setup, and maybe better to move this to `open()`, since 
we are more familiar with initializing everything in it.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162314626
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/utils/DefaultShellLogHandler.java ---
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handle output from non-JVM processes.
--- End diff --

1. This explanation looks suitable in ShellLogHandler, not this class. And 
it handles log, not output.
2. Maybe better to describe that this is default implementation.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162317609
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellUtils.java ---
@@ -464,4 +465,18 @@ public void run() {
 }
 }
 
+public static ShellLogHandler getLogHandler(Map 
topoConf) {
+if (topoConf == null) {
+throw new IllegalArgumentException("Config is required");
+}
+
+if (topoConf.containsKey(Config.TOPOLOGY_MULTILANG_LOG_HANDLER)) {
+try {
+return (ShellLogHandler) 
Class.forName(topoConf.get(Config.TOPOLOGY_MULTILANG_LOG_HANDLER).toString()).newInstance();
+} catch (ClassCastException | InstantiationException | 
IllegalAccessException | ClassNotFoundException e) {
+LOG.warn(e.toString());
--- End diff --

And also modify tests as well.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162316967
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellUtils.java ---
@@ -464,4 +465,18 @@ public void run() {
 }
 }
 
+public static ShellLogHandler getLogHandler(Map 
topoConf) {
+if (topoConf == null) {
+throw new IllegalArgumentException("Config is required");
+}
+
+if (topoConf.containsKey(Config.TOPOLOGY_MULTILANG_LOG_HANDLER)) {
+try {
+return (ShellLogHandler) 
Class.forName(topoConf.get(Config.TOPOLOGY_MULTILANG_LOG_HANDLER).toString()).newInstance();
+} catch (ClassCastException | InstantiationException | 
IllegalAccessException | ClassNotFoundException e) {
+LOG.warn(e.toString());
--- End diff --

Let's throw exception (propagation with wrapping RuntimeException) and let 
worker crash instead of leaving warning message. This is easy to be missed, and 
there's a principle of Storm: fail-fast.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162315997
  
--- Diff: 
storm-client/src/jvm/org/apache/storm/utils/DefaultShellLogHandler.java ---
@@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Handle output from non-JVM processes.
+ */
+public class DefaultShellLogHandler implements ShellLogHandler {
+public static final Logger LOG = 
LoggerFactory.getLogger(DefaultShellLogHandler.class);
--- End diff --

Looks like we're losing class information which is logging. (I mean we no 
longer write log with ShellSpout/ShellBolt.) 

Given that we are creating ShellLogHandler instance per task, we can pass 
class name in `setUpContext()` (interface needs to be changed) and initialize 
logger based on the class name. Makes sense?


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162312970
  
--- Diff: storm-client/src/jvm/org/apache/storm/spout/ShellSpout.java ---
@@ -50,6 +51,7 @@
 private SpoutOutputCollector _collector;
 private String[] _command;
 private Map env = new HashMap<>();
+private ShellLogHandler _logHandler;
--- End diff --

You seems missing initialization here on spout.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162314056
  
--- Diff: storm-client/src/jvm/org/apache/storm/task/ShellBolt.java ---
@@ -351,6 +328,7 @@ public void run() {
 
 private class BoltReaderRunnable implements Runnable {
 public void run() {
+_logHandler.setUpContext(_process, _context);
--- End diff --

This is effectively one-time setup, but if we really want to have one-time 
setup, maybe better to initialize it in prepare(), since we are more familiar 
with initializing everything in it.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r162316609
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellLogHandler.java 
---
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+
+/**
+ * Handles output from multilang processes.
--- End diff --

output -> log


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[revans2]

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r159973491
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellLogHandler.java 
---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.logging.log4j.ThreadContext;
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public final class ShellLogHandler {
+public static final Logger LOG = 
LoggerFactory.getLogger(ShellLogHandler.class);
+
+public static final String ID = "id";
+public static final String NAME = "name";
+public static final String PARENT = "parent";
+public static final String PID = "pid";
+public static final String TASK = "task";
+
+private ShellLogHandler() {
+}
+
+private static void putIfNotNull(String key, Object value) {
+if (value != null) {
+ThreadContext.put(key, value.toString());
+}
+}
+
+/**
+ * Update the {@link ThreadContext} with information about the logged
+ * message, including the pid and task.
+ * 
+ * @param shellMsg
+ *- the {@link ShellMsg} containing the ID.
+ * @param process
+ *- the current {@link ShellProcess}.
+ * @param context
+ *- the current {@link TopologyContext}.
+ */
+private static void updateContext(ShellMsg shellMsg, ShellProcess 
process, TopologyContext context) {
+putIfNotNull(ID, shellMsg.getId());
+// Calling this only once allows the same parent ID to be attached 
to
+// all log messages from a tuple tree
+if (!ThreadContext.containsKey(PARENT)) {
+putIfNotNull(PARENT, shellMsg.getId());
+}
--- End diff --

Either is fine with me.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[hmcc]

Github user hmcc commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r159972449
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellLogHandler.java 
---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.logging.log4j.ThreadContext;
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public final class ShellLogHandler {
+public static final Logger LOG = 
LoggerFactory.getLogger(ShellLogHandler.class);
+
+public static final String ID = "id";
+public static final String NAME = "name";
+public static final String PARENT = "parent";
+public static final String PID = "pid";
+public static final String TASK = "task";
+
+private ShellLogHandler() {
+}
+
+private static void putIfNotNull(String key, Object value) {
+if (value != null) {
+ThreadContext.put(key, value.toString());
+}
+}
+
+/**
+ * Update the {@link ThreadContext} with information about the logged
+ * message, including the pid and task.
+ * 
+ * @param shellMsg
+ *- the {@link ShellMsg} containing the ID.
+ * @param process
+ *- the current {@link ShellProcess}.
+ * @param context
+ *- the current {@link TopologyContext}.
+ */
+private static void updateContext(ShellMsg shellMsg, ShellProcess 
process, TopologyContext context) {
+putIfNotNull(ID, shellMsg.getId());
+// Calling this only once allows the same parent ID to be attached 
to
+// all log messages from a tuple tree
+if (!ThreadContext.containsKey(PARENT)) {
+putIfNotNull(PARENT, shellMsg.getId());
+}
--- End diff --

Thanks, that sounds like a plan.

What do you prefer - close this and create a new PR, or leave this open, 
revert the commit and add a new commit with those changes?


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[revans2]

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r159895567
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellLogHandler.java 
---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.logging.log4j.ThreadContext;
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public final class ShellLogHandler {
+public static final Logger LOG = 
LoggerFactory.getLogger(ShellLogHandler.class);
+
+public static final String ID = "id";
+public static final String NAME = "name";
+public static final String PARENT = "parent";
+public static final String PID = "pid";
+public static final String TASK = "task";
+
+private ShellLogHandler() {
+}
+
+private static void putIfNotNull(String key, Object value) {
+if (value != null) {
+ThreadContext.put(key, value.toString());
+}
+}
+
+/**
+ * Update the {@link ThreadContext} with information about the logged
+ * message, including the pid and task.
+ * 
+ * @param shellMsg
+ *- the {@link ShellMsg} containing the ID.
+ * @param process
+ *- the current {@link ShellProcess}.
+ * @param context
+ *- the current {@link TopologyContext}.
+ */
+private static void updateContext(ShellMsg shellMsg, ShellProcess 
process, TopologyContext context) {
+putIfNotNull(ID, shellMsg.getId());
+// Calling this only once allows the same parent ID to be attached 
to
+// all log messages from a tuple tree
+if (!ThreadContext.containsKey(PARENT)) {
+putIfNotNull(PARENT, shellMsg.getId());
+}
--- End diff --

@hmcc 

If parent id is intended to be used to search for all log messages about a 
given tuple tree then there is a bug.  You set parent ID once and only once.  
This means that "parent" will be set to the ID of the first log message with an 
ID ever sent.  Every single tuple is likely to have a different parent so it 
will end up being a totally useless value.


Also I agree with @HeartSaVioR I think there is a much cleaner way to get 
the same results that you want, and not having all of the overhead for those 
that don't want it.  Because the logging is thread specific.

I would propose a ShellLogHandler interface that can be set for 
implementations of ShellBolt and ShellSpout.  That way you can override it, and 
there is no overhead for people who don't want it.  I would propose something 
like

```
interface ShellLogHandler {
  void setupContext(ShellProcess process, TopologyContext context);
  void log(ShellMessage msg);
}
```

`setupContext` would be called from the BoltReaderRunnable thread.  It 
could set the ThreadContext for that thread for things like the PID, etc.  
These are things that will not change during the lifetime of the Bolt.

For the spout it is going to be more overhead because there is no way 
currently to know if you are on a thread by yourself or sharing a thread with 
another spout.  So for each log message the spout would call `setupContext` 
followed by `log`.

You also don't need to modify storm.py as you can just use sendMsgToParent 
yourself directly and have your own custom helper log methods with the ID.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[HeartSaVioR]

Github user HeartSaVioR commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r159863976
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellLogHandler.java 
---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.logging.log4j.ThreadContext;
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public final class ShellLogHandler {
+public static final Logger LOG = 
LoggerFactory.getLogger(ShellLogHandler.class);
+
+public static final String ID = "id";
+public static final String NAME = "name";
+public static final String PARENT = "parent";
+public static final String PID = "pid";
+public static final String TASK = "task";
+
+private ShellLogHandler() {
+}
+
+private static void putIfNotNull(String key, Object value) {
+if (value != null) {
+ThreadContext.put(key, value.toString());
+}
+}
+
+/**
+ * Update the {@link ThreadContext} with information about the logged
+ * message, including the pid and task.
+ * 
+ * @param shellMsg
+ *- the {@link ShellMsg} containing the ID.
+ * @param process
+ *- the current {@link ShellProcess}.
+ * @param context
+ *- the current {@link TopologyContext}.
+ */
+private static void updateContext(ShellMsg shellMsg, ShellProcess 
process, TopologyContext context) {
+putIfNotNull(ID, shellMsg.getId());
+// Calling this only once allows the same parent ID to be attached 
to
+// all log messages from a tuple tree
+if (!ThreadContext.containsKey(PARENT)) {
+putIfNotNull(PARENT, shellMsg.getId());
+}
--- End diff --

I think letting users leave the log message with providing enough 
information is most flexible way. `handleLog` is adding some context which some 
users might not want and in some case could disturb parsing the message.
So the ideal approach from my side is publishing all the necessary context 
to multi-lang running subprocess and let subprocess build the full message 
(without context which log4j2 format provides), but it might bring some 
backward incompatibility like I said, or even some overhead because it should 
be passed to multilang and longer message should be passed to 
ShellSpout/ShellBolt.

I'm OK to follow proposal provided by @hmcc, which may not require huge 
change. ID of the originating tuple is not available (we're sending generated 
new ID to multilang) so need to be added in some way though.


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[revans2]

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r159742244
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellLogHandler.java 
---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.logging.log4j.ThreadContext;
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public final class ShellLogHandler {
+public static final Logger LOG = 
LoggerFactory.getLogger(ShellLogHandler.class);
+
+public static final String ID = "id";
+public static final String NAME = "name";
+public static final String PARENT = "parent";
+public static final String PID = "pid";
+public static final String TASK = "task";
+
+private ShellLogHandler() {
+}
+
+private static void putIfNotNull(String key, Object value) {
+if (value != null) {
+ThreadContext.put(key, value.toString());
+}
+}
+
+/**
+ * Update the {@link ThreadContext} with information about the logged
+ * message, including the pid and task.
+ * 
+ * @param shellMsg
+ *- the {@link ShellMsg} containing the ID.
+ * @param process
+ *- the current {@link ShellProcess}.
+ * @param context
+ *- the current {@link TopologyContext}.
+ */
+private static void updateContext(ShellMsg shellMsg, ShellProcess 
process, TopologyContext context) {
+putIfNotNull(ID, shellMsg.getId());
+// Calling this only once allows the same parent ID to be attached 
to
+// all log messages from a tuple tree
+if (!ThreadContext.containsKey(PARENT)) {
+putIfNotNull(PARENT, shellMsg.getId());
+}
--- End diff --

This really needs some kind of documentation if we are going to start using 
id with logging.  I just don't know how you expect to use this, so I have no 
idea how to say if it works properly or not.

Also you missed javascript support.


https://github.com/apache/storm/blob/a18657d0cd01601dfd193dccade1f601fbef94b4/storm-multilang/javascript/src/main/resources/resources/storm.js#L51-L57
  


---

[GitHub] storm pull request #2475: STORM-2862: More flexible logging in multilang

[revans2]

Github user revans2 commented on a diff in the pull request:

https://github.com/apache/storm/pull/2475#discussion_r159742835
  
--- Diff: storm-client/src/jvm/org/apache/storm/utils/ShellLogHandler.java 
---
@@ -0,0 +1,113 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.storm.utils;
+
+import org.apache.logging.log4j.ThreadContext;
+import org.apache.storm.multilang.ShellMsg;
+import org.apache.storm.task.TopologyContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public final class ShellLogHandler {
+public static final Logger LOG = 
LoggerFactory.getLogger(ShellLogHandler.class);
+
+public static final String ID = "id";
+public static final String NAME = "name";
+public static final String PARENT = "parent";
+public static final String PID = "pid";
+public static final String TASK = "task";
+
+private ShellLogHandler() {
+}
+
+private static void putIfNotNull(String key, Object value) {
+if (value != null) {
+ThreadContext.put(key, value.toString());
+}
+}
+
+/**
+ * Update the {@link ThreadContext} with information about the logged
+ * message, including the pid and task.
+ * 
+ * @param shellMsg
+ *- the {@link ShellMsg} containing the ID.
+ * @param process
+ *- the current {@link ShellProcess}.
+ * @param context
+ *- the current {@link TopologyContext}.
+ */
+private static void updateContext(ShellMsg shellMsg, ShellProcess 
process, TopologyContext context) {
+putIfNotNull(ID, shellMsg.getId());
+// Calling this only once allows the same parent ID to be attached 
to
+// all log messages from a tuple tree
+if (!ThreadContext.containsKey(PARENT)) {
+putIfNotNull(PARENT, shellMsg.getId());
+}
+if (process != null) {
+putIfNotNull(NAME, process.getComponentName());
+putIfNotNull(PID, process.getPid());
+}
+if (context != null) {
+ThreadContext.put(TASK, 
Integer.toString(context.getThisTaskId()));
+}
+}
+
+/**
+ * Log the given message and update the {@link ThreadContext} with some
+ * contextual information, including the pid and task.
+ * 
+ * @param shellMsg
+ *- the {@link ShellMsg} containing the ID. Required.
+ * @param process
+ *- the current {@link ShellProcess}. Optional.
+ * @param context
+ *- the current {@link TopologyContext}. Optional.
+ */
+public static void handleLog(ShellMsg shellMsg, ShellProcess process, 
TopologyContext context) {
+if (shellMsg == null) {
+throw new IllegalArgumentException("shellMsg is required");
+}
+
+updateContext(shellMsg, process, context);
--- End diff --

Could we try and optimize the overhead on this?  Users should not be 
logging much if anything, but it might be nice to check if the current log 
level is enabled before we try to update the context, as I don't know how 
expensive it is.

We can probably make that happen by making some changes to the 
ShellMsg.ShellLogLevel enum to know more about log4j


---