rabbah commented on a change in pull request #2878: Streamingly read user-logs.
URL:
https://github.com/apache/incubator-openwhisk/pull/2878#discussion_r152848132
##########
File path:
core/invoker/src/main/scala/whisk/core/containerpool/docker/DockerContainer.scala
##########
@@ -224,31 +230,73 @@ class DockerContainer(protected val id: ContainerId,
*
* @return a vector of Strings with log lines in our own JSON format
*/
- def logs(limit: ByteSize, waitForSentinel: Boolean)(implicit transid:
TransactionId): Future[Vector[String]] = {
-
- def readLogs(retries: Int): Future[Vector[String]] = {
- docker
- .rawContainerLogs(id, logFileOffset)
- .flatMap { rawLogBytes =>
- val rawLog =
- new String(rawLogBytes.array, rawLogBytes.arrayOffset,
rawLogBytes.position, StandardCharsets.UTF_8)
- val (isComplete, isTruncated, formattedLogs) =
processJsonDriverLogContents(rawLog, waitForSentinel, limit)
-
- if (retries > 0 && !isComplete && !isTruncated) {
- logging.info(this, s"log cursor advanced but missing sentinel,
trying $retries more times")
- akka.pattern.after(filePollInterval,
as.scheduler)(readLogs(retries - 1))
- } else {
- logFileOffset += rawLogBytes.position - rawLogBytes.arrayOffset
- Future.successful(formattedLogs)
- }
- }
- .andThen {
- case Failure(e) =>
- logging.error(this, s"Failed to obtain logs of ${id.asString}:
${e.getClass} - ${e.getMessage}")
- }
+ def logs(limit: ByteSize, waitForSentinel: Boolean)(implicit transid:
TransactionId): Source[ByteString, Any] = {
+ val source = docker
+ .rawContainerLogs(id, logFileOffset.get(), if (waitForSentinel)
Some(filePollInterval) else None)
+ .via(Framing.delimiter(delimiter, Int.MaxValue))
+ .limitWeighted(limit.toBytes) { obj =>
Review comment:
thanks for making this change - the Framing does allow for truncated
results, which might be useful; since the format is well know, we could still
parse it. Makes me also wonder if it may be generally better to eschew the more
JSON parser
```
Flow[ByteString].map(_.utf8String.parseJson.convertTo[LogLine].toFormattedString)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
