Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-03-05 Thread via GitHub 


markap14 merged PR #8407:
URL: https://github.com/apache/nifi/pull/8407


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-03-05 Thread via GitHub 


markap14 commented on PR #8407:
URL: https://github.com/apache/nifi/pull/8407#issuecomment-1979701945

   Thanks @exceptionfactory all seems to be working well now. This is a huge 
improvement! +1 will merge to main


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-03-05 Thread via GitHub 


exceptionfactory commented on PR #8407:
URL: https://github.com/apache/nifi/pull/8407#issuecomment-1978901124

   Thanks for the feedback @markap14, I rebased to incorporate the recent 
Python framework improvements, and also addressed the comments regarding log 
writing and exception handling.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-03-05 Thread via GitHub 


exceptionfactory commented on code in PR #8407:
URL: https://github.com/apache/nifi/pull/8407#discussion_r1512922173


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcessLogReader.java:
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.py4j;
+
+import org.apache.nifi.py4j.logging.PythonLogLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.stream.Collectors;
+
+/**
+ * Runnable Command for reading a line from Process Output Stream and writing 
to a Logger
+ */
+class PythonProcessLogReader implements Runnable {
+private static final int LOG_LEVEL_BEGIN_INDEX = 0;
+
+private static final int LOG_LEVEL_END_INDEX = 2;
+
+private static final int MESSAGE_BEGIN_INDEX = 3;
+
+private static final char NAME_MESSAGE_SEPARATOR = ':';
+
+private static final int MINIMUM_LOGGER_NAME_INDEX = 3;
+
+private static final String LOG_PREFIX = "PY4JLOG";
+
+private static final int PREFIXED_LOG_LEVEL_BEGIN_INDEX = 8;
+
+private static final String LINE_SEPARATOR = System.lineSeparator();
+
+private static final Map PYTHON_LOG_LEVELS = 
Arrays.stream(PythonLogLevel.values()).collect(
+Collectors.toUnmodifiableMap(
+pythonLogLevel -> 
Integer.toString(pythonLogLevel.getLevel()),
+pythonLogLevel -> pythonLogLevel
+)
+);
+
+private final Logger processLogger = 
LoggerFactory.getLogger("org.apache.nifi.py4j.ProcessLog");
+
+private final BufferedReader processReader;
+
+/**
+ * Standard constructor with Buffered Reader connected to Python Process 
Output Stream
+ *
+ * @param processReader Reader from Process Output Stream
+ */
+PythonProcessLogReader(final BufferedReader processReader) {
+this.processReader = Objects.requireNonNull(processReader, "Reader 
required");
+}
+
+/**
+ * Read lines from Process Reader and write log messages based on parsed 
level and named logger
+ */
+@Override
+public void run() {
+final Queue parsedRecords = new LinkedList<>();
+
+try {
+String line = processReader.readLine();
+while (line != null) {
+processLine(line, parsedRecords);
+
+if (parsedRecords.size() == 2 || !processReader.ready()) {
+final ParsedRecord parsedRecord = parsedRecords.remove();
+log(parsedRecord);
+}

Review Comment:
   Thanks for the thorough feedback and testing. I made a change to check the 
readiness of the Process Reader separately, and then the example `LogContents` 
Processor writes the messages as expected.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-02-28 Thread via GitHub 


markap14 commented on code in PR #8407:
URL: https://github.com/apache/nifi/pull/8407#discussion_r1506303249


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcessLogReader.java:
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.py4j;
+
+import org.apache.nifi.py4j.logging.PythonLogLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.stream.Collectors;
+
+/**
+ * Runnable Command for reading a line from Process Output Stream and writing 
to a Logger
+ */
+class PythonProcessLogReader implements Runnable {
+private static final int LOG_LEVEL_BEGIN_INDEX = 0;
+
+private static final int LOG_LEVEL_END_INDEX = 2;
+
+private static final int MESSAGE_BEGIN_INDEX = 3;
+
+private static final char NAME_MESSAGE_SEPARATOR = ':';
+
+private static final int MINIMUM_LOGGER_NAME_INDEX = 3;
+
+private static final String LOG_PREFIX = "PY4JLOG";
+
+private static final int PREFIXED_LOG_LEVEL_BEGIN_INDEX = 8;
+
+private static final String LINE_SEPARATOR = System.lineSeparator();
+
+private static final Map PYTHON_LOG_LEVELS = 
Arrays.stream(PythonLogLevel.values()).collect(
+Collectors.toUnmodifiableMap(
+pythonLogLevel -> 
Integer.toString(pythonLogLevel.getLevel()),
+pythonLogLevel -> pythonLogLevel
+)
+);
+
+private final Logger processLogger = 
LoggerFactory.getLogger("org.apache.nifi.py4j.ProcessLog");
+
+private final BufferedReader processReader;
+
+/**
+ * Standard constructor with Buffered Reader connected to Python Process 
Output Stream
+ *
+ * @param processReader Reader from Process Output Stream
+ */
+PythonProcessLogReader(final BufferedReader processReader) {
+this.processReader = Objects.requireNonNull(processReader, "Reader 
required");
+}
+
+/**
+ * Read lines from Process Reader and write log messages based on parsed 
level and named logger
+ */
+@Override
+public void run() {
+final Queue parsedRecords = new LinkedList<>();
+
+try {
+String line = processReader.readLine();
+while (line != null) {
+processLine(line, parsedRecords);
+
+if (parsedRecords.size() == 2 || !processReader.ready()) {
+final ParsedRecord parsedRecord = parsedRecords.remove();
+log(parsedRecord);
+}

Review Comment:
   I think the logic here is a bit off.
   I created this Processor for testing:
   ```
   from nifiapi.flowfiletransform import FlowFileTransform, 
FlowFileTransformResult
   import logging
   
   root_logger = logging.getLogger('')
   
   class LogContents(FlowFileTransform):
   class Java:
   implements = ['org.apache.nifi.python.processor.FlowFileTransform']
   class ProcessorDetails:
   version = '0.0.1-SNAPSHOT'
   
   def __init__(self, **kwargs):
   pass
   
   def transform(self, context, flowFile):
   contents = flowFile.getContentsAsBytes().decode("utf-8")
   root_logger.info("Contents:\n1\n2\n3")
   root_logger.info(contents)
   return FlowFileTransformResult(relationship = "success")
   ```
   
   I then sent in some FlowFiles. Each time, I get the log output of:
   ```
   Contents
   1
   2
   3
   ```
   
   but not the output of the actual logger.
   If I change the first log message instead to:
   ```
   root_logger.info("Contents:")
   ```
   
   I get just the line "Contents:" on the first iteration. The second time the 
processor is run, it outputs the contents of the first FlowFile, followed by 
"Contents" again. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-02-28 Thread via GitHub 


markap14 commented on code in PR #8407:
URL: https://github.com/apache/nifi/pull/8407#discussion_r1506300182


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcessLogReader.java:
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.py4j;
+
+import org.apache.nifi.py4j.logging.PythonLogLevel;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.stream.Collectors;
+
+/**
+ * Runnable Command for reading a line from Process Output Stream and writing 
to a Logger
+ */
+class PythonProcessLogReader implements Runnable {
+private static final int LOG_LEVEL_BEGIN_INDEX = 0;
+
+private static final int LOG_LEVEL_END_INDEX = 2;
+
+private static final int MESSAGE_BEGIN_INDEX = 3;
+
+private static final char NAME_MESSAGE_SEPARATOR = ':';
+
+private static final int MINIMUM_LOGGER_NAME_INDEX = 3;
+
+private static final String LOG_PREFIX = "PY4JLOG";
+
+private static final int PREFIXED_LOG_LEVEL_BEGIN_INDEX = 8;
+
+private static final String LINE_SEPARATOR = System.lineSeparator();
+
+private static final Map PYTHON_LOG_LEVELS = 
Arrays.stream(PythonLogLevel.values()).collect(
+Collectors.toUnmodifiableMap(
+pythonLogLevel -> 
Integer.toString(pythonLogLevel.getLevel()),
+pythonLogLevel -> pythonLogLevel
+)
+);
+
+private final Logger processLogger = 
LoggerFactory.getLogger("org.apache.nifi.py4j.ProcessLog");
+
+private final BufferedReader processReader;
+
+/**
+ * Standard constructor with Buffered Reader connected to Python Process 
Output Stream
+ *
+ * @param processReader Reader from Process Output Stream
+ */
+PythonProcessLogReader(final BufferedReader processReader) {
+this.processReader = Objects.requireNonNull(processReader, "Reader 
required");
+}
+
+/**
+ * Read lines from Process Reader and write log messages based on parsed 
level and named logger
+ */
+@Override
+public void run() {
+final Queue parsedRecords = new LinkedList<>();
+
+try {
+String line = processReader.readLine();
+while (line != null) {
+processLine(line, parsedRecords);

Review Comment:
   I think within this `while` loop we should have a try/catch (Exception e)
   Without it, if there's any Exception thrown from parsing, etc. the execution 
will escape the run() method without any sort of warning, and the logging will 
simply stop. Which will eventually cause the Python process' STDOUT to fill up 
and then the Python process will block.
   Need to ensure that this thread never dies unexpectedly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-02-20 Thread via GitHub 


exceptionfactory commented on PR #8407:
URL: https://github.com/apache/nifi/pull/8407#issuecomment-1955870544

   Thanks for the initial feedback @markap14! Based on further discussion, I 
will be following up with additional changes to incorporate log level updates 
from Logback.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-02-14 Thread via GitHub 


exceptionfactory commented on code in PR #8407:
URL: https://github.com/apache/nifi/pull/8407#discussion_r1490140332


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcessLogReader.java:
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.py4j;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Runnable Command for reading a line from Process Output Stream and writing 
to a Logger
+ */
+class PythonProcessLogReader implements Runnable {
+private static final int LOG_LEVEL_BEGIN_INDEX = 0;
+
+private static final int LOG_LEVEL_END_INDEX = 2;
+
+private static final int MESSAGE_BEGIN_INDEX = 3;
+
+private static final int MINIMUM_LINE_LENGTH = 4;
+
+private static final char NAME_MESSAGE_SEPARATOR = ':';
+
+private static final int INDEX_NOT_FOUND = -1;
+
+private final Logger processLogger = 
LoggerFactory.getLogger("org.apache.nifi.py4j.ProcessLog");
+
+private final BufferedReader processReader;
+
+PythonProcessLogReader(final BufferedReader processReader) {
+this.processReader = Objects.requireNonNull(processReader, "Reader 
required");
+}
+
+@Override
+public void run() {
+try {
+String line = processReader.readLine();
+while (line != null) {
+if (line.length() > MINIMUM_LINE_LENGTH) {
+processLine(line);
+}
+line = processReader.readLine();
+}
+} catch (final IOException e) {
+processLogger.error("Read Process output failed", e);

Review Comment:
   The logger name may help, but your suggestion is definitely clearer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-02-14 Thread via GitHub 


markap14 commented on code in PR #8407:
URL: https://github.com/apache/nifi/pull/8407#discussion_r1490101070


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/test/java/org/apache/nifi/py4j/PythonProcessLogReaderTest.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.py4j;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.MockedStatic;
+import org.mockito.junit.jupiter.MockitoExtension;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.StringReader;
+
+import static org.mockito.ArgumentMatchers.eq;
+import static org.mockito.Mockito.mockStatic;
+import static org.mockito.Mockito.verify;
+
+@ExtendWith(MockitoExtension.class)
+class PythonProcessLogReaderTest {
+
+private static final String PROCESS_LOGGER = 
"org.apache.nifi.py4j.ProcessLog";
+
+private static final String CONTROLLER_LOGGER = 
"org.apache.nifi.py4j.Controller";
+
+private static final String LOG_MESSAGE = "Testing Python Processing";
+
+private static final String LINE_FORMAT = "%s %s:%s";
+
+private static final String LINE_UNFORMATTED = "Testing message without 
level or logger";
+
+private static final String LINE_MISSING_SEPARATOR = "%s 
%s".formatted(PythonProcessLogReader.LogLevel.INFO.getLevel(), LOG_MESSAGE);
+
+@Mock
+private Logger logger;
+
+@Test
+void testDebug() {
+runCommand(PythonProcessLogReader.LogLevel.DEBUG);
+
+verify(logger).debug(eq(LOG_MESSAGE));
+}
+
+@Test
+void testInfo() {
+runCommand(PythonProcessLogReader.LogLevel.INFO);
+
+verify(logger).info(eq(LOG_MESSAGE));
+}
+
+@Test
+void testWarning() {
+runCommand(PythonProcessLogReader.LogLevel.WARNING);
+
+verify(logger).warn(eq(LOG_MESSAGE));
+}
+
+@Test
+void testError() {
+runCommand(PythonProcessLogReader.LogLevel.ERROR);
+
+verify(logger).error(eq(LOG_MESSAGE));
+}
+
+@Test
+void testCritical() {
+runCommand(PythonProcessLogReader.LogLevel.CRITICAL);
+
+verify(logger).error(eq(LOG_MESSAGE));
+}
+
+@Test
+void testUnformatted() {
+try (MockedStatic loggerFactory = 
mockStatic(LoggerFactory.class)) {
+loggerFactory.when(() -> 
LoggerFactory.getLogger(eq(PROCESS_LOGGER))).thenReturn(logger);
+
+final BufferedReader reader = new BufferedReader(new 
StringReader(LINE_UNFORMATTED));
+final Runnable command = new PythonProcessLogReader(reader);
+command.run();
+}
+
+verify(logger).info(eq(LINE_UNFORMATTED));
+}
+
+@Test
+void testMissingSeparator() {

Review Comment:
   You mind adding a test for logging an empty string? I was wondering when 
reading the code above, if the `:` separator is the last character in the line, 
is that going to throw some sort of ArrayIndexOutOfBoundsException or something?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-02-14 Thread via GitHub 


markap14 commented on code in PR #8407:
URL: https://github.com/apache/nifi/pull/8407#discussion_r1490098492


##
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcessLogReader.java:
##
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.py4j;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.util.Objects;
+
+/**
+ * Runnable Command for reading a line from Process Output Stream and writing 
to a Logger
+ */
+class PythonProcessLogReader implements Runnable {
+private static final int LOG_LEVEL_BEGIN_INDEX = 0;
+
+private static final int LOG_LEVEL_END_INDEX = 2;
+
+private static final int MESSAGE_BEGIN_INDEX = 3;
+
+private static final int MINIMUM_LINE_LENGTH = 4;
+
+private static final char NAME_MESSAGE_SEPARATOR = ':';
+
+private static final int INDEX_NOT_FOUND = -1;
+
+private final Logger processLogger = 
LoggerFactory.getLogger("org.apache.nifi.py4j.ProcessLog");
+
+private final BufferedReader processReader;
+
+PythonProcessLogReader(final BufferedReader processReader) {
+this.processReader = Objects.requireNonNull(processReader, "Reader 
required");
+}
+
+@Override
+public void run() {
+try {
+String line = processReader.readLine();
+while (line != null) {
+if (line.length() > MINIMUM_LINE_LENGTH) {
+processLine(line);
+}
+line = processReader.readLine();
+}
+} catch (final IOException e) {
+processLogger.error("Read Process output failed", e);

Review Comment:
   The message here is quite vague; perhaps "Failed to read output of Python 
Process"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] NIFI-11443 Route Python Framework Logging to SLF4J [nifi]

2024-02-14 Thread via GitHub 


exceptionfactory opened a new pull request, #8407:
URL: https://github.com/apache/nifi/pull/8407

   # Summary
   
   [NIFI-11443](https://issues.apache.org/jira/browse/NIFI-11443) Updates 
Python Process log handling to route messages to SLF4J so that logging levels 
and appenders can be managed using Logback.
   
   These changes remove the need for separate Python log settings in 
`nifi.properties` and support translating Python log levels and logger names to 
Logback.
   
   The implementation changes the Python logging format to use the Python 
[level numbers](https://docs.python.org/3/library/logging.html#levels) for 
optimized parsing. The implementation changes Python logging to write messages 
to the standard output stream, and the `PythonProcess` builder creates a Java 
Virtual Thread to read lines from the `BufferedReader` containing process 
output stream messages.
   
   These changes eliminate the dedicated `nifi-python.log` and enable Python 
logging to be handled along with other framework logging that defaults to the 
`nifi-app.log` file appender.
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue 
created
   
   ### Pull Request Tracking
   
   - [X] Pull Request title starts with Apache NiFi Jira issue number, such as 
`NIFI-0`
   - [X] Pull Request commit message starts with Apache NiFi Jira issue number, 
as such `NIFI-0`
   
   ### Pull Request Formatting
   
   - [X] Pull Request based on current revision of the `main` branch
   - [X] Pull Request refers to a feature branch with one commit containing 
changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request 
creation.
   
   ### Build
   
   - [X] Build completed using `mvn clean install -P contrib-check`
 - [X] JDK 21
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 
2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License 
Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` 
files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org