coderfender commented on code in PR #1883:
URL:
https://github.com/apache/datafusion-ballista/pull/1883#discussion_r3575799133
##########
python/python/ballista/jupyter.py:
##########
@@ -221,6 +232,45 @@ def register(self, line: str) -> Optional[str]:
"Currently not supporting the inserted file format"
)
+ @staticmethod
+ def _parse_cell_magic_args(line: str):
+ """Parse the argument line of a ``%%sql`` cell magic.
+
+ Recognises the ``--no-display`` and ``--limit N`` flags (space form,
+ e.g. ``--limit 5``, consistent with the other magics in this module).
+ The first non-flag token, if any, is treated as the variable name to
+ store the result in.
+
+ Returns a ``(var_name, no_display, limit)`` tuple where ``limit`` is
+ ``None`` when ``--limit`` was not supplied. Raises ``ValueError`` for a
+ missing or invalid ``--limit`` value.
+ """
+ tokens = line.strip().split()
+ var_name = None
+ no_display = False
+ limit = None
+
+ i = 0
+ while i < len(tokens):
Review Comment:
Could be a good practice to store comstant ipython flags separately?
##########
python/python/ballista/jupyter.py:
##########
@@ -108,8 +108,19 @@ def decorator(func):
return decorator
+from datafusion.dataframe_formatter import (
+ configure_formatter,
+ get_formatter,
+ set_formatter,
+)
+
from .extension import BallistaSessionContext, DistributedDataFrame
+# Default number of rows rendered for a ``%%sql`` cell when ``--limit`` is not
+# given. This caps only the display (via datafusion's HTML formatter); the
+# underlying result keeps all of its rows.
+DEFAULT_DISPLAY_LIMIT = 50
Review Comment:
Could we probably push this as a limit option to datafusion ? That might
help with probing high volume data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
