Ruifeng Zheng created SPARK-44564:
-------------------------------------
Summary: Refine the documents with LLM
Key: SPARK-44564
URL: https://issues.apache.org/jira/browse/SPARK-44564
Project: Spark
Issue Type: Umbrella
Components: Documentation
Affects Versions: 4.0.0
Reporter: Ruifeng Zheng
Let's first focus on the Documents of PySpark DataFrame APIs.
1, Chose a subset of DF APIs
Since the review bandwidth is limited, we recommend each PR contains at least 5
APIs;
2, For each API, copy-paste the function (including function signature, doc
string) to a LLM Model, and ask it to refine the document with prompts like:
* please improve the docstring of the 'unionByName' function
* please refine the comments of the 'unionByName' function
* please refine the documents of the 'unionByName' function, and add more
examples
* please provide more example for function 'unionByName'
It is highly recommended to leverage *GPT-4* instead of GPT-3.5, since the
former generate better results.
3, The generated doc string may contain some bugs, e.g.
* The description use wrong version, add a 'Raise' selection;
* The example code doesn't reflect the example title;
* The example results are incorrect
we need to fix them before sending a PR.
We can generate the docs with different prompts, choose the good parts and
combine them to the new doc sting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
