[ https://issues.apache.org/jira/browse/SPARK-39722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572165#comment-17572165 ]
Erik Krogen commented on SPARK-39722: ------------------------------------- General +1 from me. We have some internal code that does exactly the {{Console.out}} redirection hack you described. > Make Dataset.showString() public > -------------------------------- > > Key: SPARK-39722 > URL: https://issues.apache.org/jira/browse/SPARK-39722 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 2.4.8, 3.3.0 > Reporter: Jatin Sharma > Priority: Trivial > > Currently, we have {{.show}} APIs on a Dataset, but they print directly to > stdout. > But there are a lot of cases where we might need to get a String > representation of the show output. For example > * We have a logging framework to which we need to push the representation of > a df > * We have to send the string over a REST call from the driver > * We want to send the string to stderr instead of stdout > For such cases, currently one needs to do a hack by changing the Console.out > temporarily and catching the representation in a ByteArrayOutputStream or > similar, then extracting the string from it. > Strictly only printing to stdout seems like a limiting choice. > > Solution: > We expose APIs to return the String representation back. We already have the > .{{{}showString{}}} method internally. > > We could mirror the current {{.show}} APIS with a corresponding > {{.showString}} (and rename the internal private function to something else > if required) -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org