Hello Cheng. I don't quite understand, why can't you configure Log4j bundled with spark to write logs in whatever format you need, then use something like promtail to export the logs to whatever log service you want to use?
BR, Martin ________________________________ From: Cheng Pan <cheng...@apache.org> Sent: Tuesday, November 1, 2022 17:03 To: spark-dev <dev@spark.apache.org> Subject: Allow Spark on K8s to integrate w/ External Log Service EXTERNAL SENDER. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username or password. Hi Spark community, I’m recently working on migrating Spark workloads from Yarn to K8s, and one of the issue I found is that there is no out-of-box log solution for Spark applications on K8s as Yarn has, and Spark even does not provide a way to integrate w/ external log service. Based on the above situation, I propose a way[1][2] to allow Spark to integrate w/ external log service when running on K8s, and PR[3] is available for reviewing and testing. I also found there was a PR[4] wants to the similar thing. Please let me know if you have any feedback on this idea, and other log solutions/ideas/suggestions for Spark on K8s are welcome as well. [1] https://docs.google.com/document/d/1MfB39LD4B4Rp7MDRxZbMKMbdNSe6V6mBmMQ-gkCnM-0/edit?usp=sharing<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F1MfB39LD4B4Rp7MDRxZbMKMbdNSe6V6mBmMQ-gkCnM-0%2Fedit%3Fusp%3Dsharing&data=05%7C01%7Cmartin.andersson%40kambi.com%7C3d098567685c47a3c1de08dabc22bdc0%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638029154642900019%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=G7qXFK2msMyB5OI3xUwQ4LqkNnlMnc5SpzmMHApLE1o%3D&reserved=0> [2] https://issues.apache.org/jira/browse/SPARK-40887<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSPARK-40887&data=05%7C01%7Cmartin.andersson%40kambi.com%7C3d098567685c47a3c1de08dabc22bdc0%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638029154642900019%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7ssnO23zcDErOcZsi9I09aMNIxq5xrV0AIVQA8386VA%3D&reserved=0> [3] https://github.com/apache/spark/pull/38357<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fpull%2F38357&data=05%7C01%7Cmartin.andersson%40kambi.com%7C3d098567685c47a3c1de08dabc22bdc0%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638029154643056253%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=UcOJUhZSKlL30QiTUWgGBIMiHuLrPSrBHa5sBf4vpVw%3D&reserved=0> [4] https://github.com/apache/spark/pull/32456<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fpull%2F32456&data=05%7C01%7Cmartin.andersson%40kambi.com%7C3d098567685c47a3c1de08dabc22bdc0%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638029154643056253%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OHNZqnEjwAH%2F2aEQTCxbqHJEwgt6hWc8cVSDahoIBkQ%3D&reserved=0> Thanks, Cheng Pan
