Fernando Pereira created SPARK-23997:

             Summary: Configurable max number of buckets
                 Key: SPARK-23997
                 URL: https://issues.apache.org/jira/browse/SPARK-23997
             Project: Spark
          Issue Type: Bug
          Components: Input/Output, SQL
    Affects Versions: 2.3.0, 2.2.1
            Reporter: Fernando Pereira

When exporting data as a table the user can choose to split data in buckets by 
choosing the columns and the number of buckets. Currently there is a hard-coded 
limit of 99'999 buckets.

However, for heavy workloads this limit might be too restrictive, a situation 
that will eventually become more common as workloads grow.

As per the comments in SPARK-19618 this limit could be made configurable.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to