[jira] [Commented] (SPARK-25150) Joining DataFrames derived from the same source yields confusing/incorrect results

2021-12-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459494#comment-17459494 ] Nicholas Chammas commented on SPARK-25150: -- I re-ran my test (described in the issue

[jira] [Commented] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-12-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459467#comment-17459467 ] Nicholas Chammas commented on SPARK-24853: -- [~hyukjin.kwon] - Are you still opposed to this

[jira] [Resolved] (SPARK-26589) proper `median` method for spark dataframe

2021-12-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-26589. -- Resolution: Won't Fix Marking this as "Won't Fix", but I suppose if someone really

[jira] [Commented] (SPARK-26589) proper `median` method for spark dataframe

2021-12-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17459455#comment-17459455 ] Nicholas Chammas commented on SPARK-26589: -- It looks like making a distributed,

[jira] [Created] (SPARK-37647) Expose percentile function in Scala/Python APIs

2021-12-14 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-37647: Summary: Expose percentile function in Scala/Python APIs Key: SPARK-37647 URL: https://issues.apache.org/jira/browse/SPARK-37647 Project: Spark

[jira] [Commented] (SPARK-26589) proper `median` method for spark dataframe

2021-12-09 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17456860#comment-17456860 ] Nicholas Chammas commented on SPARK-26589: -- That makes sense to me. I've been struggling with

[jira] [Commented] (SPARK-26589) proper `median` method for spark dataframe

2021-12-01 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452081#comment-17452081 ] Nicholas Chammas commented on SPARK-26589: -- [~srowen] - I'll ask for help on the dev list if

[jira] [Commented] (SPARK-26589) proper `median` method for spark dataframe

2021-12-01 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451936#comment-17451936 ] Nicholas Chammas commented on SPARK-26589: -- Just for reference, Stack Overflow provides

[jira] [Comment Edited] (SPARK-26589) proper `median` method for spark dataframe

2021-11-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451283#comment-17451283 ] Nicholas Chammas edited comment on SPARK-26589 at 11/30/21, 6:17 PM: -

[jira] [Commented] (SPARK-26589) proper `median` method for spark dataframe

2021-11-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451283#comment-17451283 ] Nicholas Chammas commented on SPARK-26589: -- I'm going to try to implement this using the

[jira] [Updated] (SPARK-12185) Add Histogram support to Spark SQL/DataFrames

2021-11-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-12185: - Labels: (was: bulk-closed) > Add Histogram support to Spark SQL/DataFrames >

[jira] [Reopened] (SPARK-12185) Add Histogram support to Spark SQL/DataFrames

2021-11-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-12185: -- Reopening this because I think it's a valid improvement that mirrors the existing

[jira] [Resolved] (SPARK-37393) Inline annotations for {ml, mllib}/common.py

2021-11-22 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-37393. -- Resolution: Duplicate > Inline annotations for {ml, mllib}/common.py >

[jira] [Updated] (SPARK-37393) Inline annotations for {ml, mllib}/common.py

2021-11-19 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-37393: - Description: This will allow us to run type checks against those files themselves. >

[jira] [Created] (SPARK-37393) Inline annotations for {ml, mllib}/common.py

2021-11-19 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-37393: Summary: Inline annotations for {ml, mllib}/common.py Key: SPARK-37393 URL: https://issues.apache.org/jira/browse/SPARK-37393 Project: Spark Issue

[jira] [Created] (SPARK-37380) Miscellaneous Python lint infra cleanup

2021-11-18 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-37380: Summary: Miscellaneous Python lint infra cleanup Key: SPARK-37380 URL: https://issues.apache.org/jira/browse/SPARK-37380 Project: Spark Issue Type:

[jira] [Updated] (SPARK-37336) Migrate _java2py to SparkSession

2021-11-15 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-37336: - Summary: Migrate _java2py to SparkSession (was: Migrate common ML utils to

[jira] [Created] (SPARK-37336) Migrate common ML utils to SparkSession

2021-11-15 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-37336: Summary: Migrate common ML utils to SparkSession Key: SPARK-37336 URL: https://issues.apache.org/jira/browse/SPARK-37336 Project: Spark Issue Type:

[jira] [Updated] (SPARK-37335) Clarify output of FPGrowth

2021-11-15 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-37335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-37335: - Description: The association rules returned by FPGrow include more columns than are

[jira] [Created] (SPARK-37335) Clarify output of FPGrowth

2021-11-15 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-37335: Summary: Clarify output of FPGrowth Key: SPARK-37335 URL: https://issues.apache.org/jira/browse/SPARK-37335 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437394#comment-17437394 ] Nicholas Chammas edited comment on SPARK-24853 at 11/2/21, 2:41 PM:

[jira] [Commented] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437396#comment-17437396 ] Nicholas Chammas commented on SPARK-24853: -- The [contributing

[jira] [Updated] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-24853: - Priority: Minor (was: Major) > Support Column type for withColumn and

[jira] [Reopened] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-24853: -- > Support Column type for withColumn and withColumnRenamed apis >

[jira] [Commented] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17437394#comment-17437394 ] Nicholas Chammas commented on SPARK-24853: -- [~hyukjin.kwon] - It's not just for consistency. As

[jira] [Updated] (SPARK-24853) Support Column type for withColumn and withColumnRenamed apis

2021-11-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-24853: - Affects Version/s: 3.2.0 > Support Column type for withColumn and withColumnRenamed

[jira] [Commented] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2021-04-15 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322333#comment-17322333 ] Nicholas Chammas commented on SPARK-33000: -- Per the discussion [on the dev

[jira] [Commented] (SPARK-33436) PySpark equivalent of SparkContext.hadoopConfiguration

2021-03-10 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17299283#comment-17299283 ] Nicholas Chammas commented on SPARK-33436: -- [~hyukjin.kwon] - Can you clarify please why this

[jira] [Updated] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2021-03-10 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-33000: - Description: Maybe it's just that the documentation needs to be updated, but I found

[jira] [Commented] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2021-03-04 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295531#comment-17295531 ] Nicholas Chammas commented on SPARK-33000: -- [~caowang888] - If you're still interested in this

[jira] [Resolved] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

2021-02-08 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-34194. -- Resolution: Won't Fix > Queries that only touch partition columns shouldn't scan

[jira] [Commented] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

2021-02-08 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281269#comment-17281269 ] Nicholas Chammas commented on SPARK-34194: -- It's not clear to me whether SPARK-26709 is

[jira] [Comment Edited] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

2021-02-01 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276869#comment-17276869 ] Nicholas Chammas edited comment on SPARK-34194 at 2/2/21, 5:56 AM: ---

[jira] [Commented] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

2021-02-01 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276869#comment-17276869 ] Nicholas Chammas commented on SPARK-34194: -- Interesting reference, [~attilapiros]. It looks

[jira] [Commented] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2021-01-21 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269571#comment-17269571 ] Nicholas Chammas commented on SPARK-12890: -- I've created SPARK-34194 and fleshed out the

[jira] [Created] (SPARK-34194) Queries that only touch partition columns shouldn't scan through all files

2021-01-21 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-34194: Summary: Queries that only touch partition columns shouldn't scan through all files Key: SPARK-34194 URL: https://issues.apache.org/jira/browse/SPARK-34194

[jira] [Commented] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2021-01-18 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267657#comment-17267657 ] Nicholas Chammas commented on SPARK-12890: -- Sure, will do. > Spark SQL query related to only

[jira] [Comment Edited] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2021-01-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253044#comment-17253044 ] Nicholas Chammas edited comment on SPARK-12890 at 1/14/21, 5:41 PM:

[jira] [Reopened] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2020-12-21 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas reopened SPARK-12890: -- Reopening because I think there is a valid potential improvement to be made here. If

[jira] [Updated] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2020-12-21 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-12890: - Labels: (was: bulk-closed) > Spark SQL query related to only partition fields should

[jira] [Updated] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2020-12-21 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-12890: - Priority: Minor (was: Major) > Spark SQL query related to only partition fields should

[jira] [Commented] (SPARK-12890) Spark SQL query related to only partition fields should not scan the whole data.

2020-12-21 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253044#comment-17253044 ] Nicholas Chammas commented on SPARK-12890: -- I think this is still an open issue. On Spark

[jira] [Updated] (SPARK-33436) PySpark equivalent of SparkContext.hadoopConfiguration

2020-11-12 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-33436: - Description: PySpark should offer an API to {{hadoopConfiguration}} to [match

[jira] [Updated] (SPARK-33436) PySpark equivalent of SparkContext.hadoopConfiguration

2020-11-12 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-33436: - Description: PySpark should offer an API to {{hadoopConfiguration}} to [match

[jira] [Created] (SPARK-33436) PySpark equivalent of SparkContext.hadoopConfiguration

2020-11-12 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-33436: Summary: PySpark equivalent of SparkContext.hadoopConfiguration Key: SPARK-33436 URL: https://issues.apache.org/jira/browse/SPARK-33436 Project: Spark

[jira] [Updated] (SPARK-33434) Document spark.conf.isModifiable()

2020-11-12 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-33434: - Affects Version/s: (was: 2.4.7) (was: 3.0.1)

[jira] [Updated] (SPARK-33434) Document spark.conf.isModifiable()

2020-11-12 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-33434: - Affects Version/s: 3.0.1 > Document spark.conf.isModifiable() >

[jira] [Updated] (SPARK-33434) Document spark.conf.isModifiable()

2020-11-12 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-33434: - Description: PySpark's docs make no mention of {{conf.isModifiable()}}, though it

[jira] [Created] (SPARK-33434) Document spark.conf.isModifiable()

2020-11-12 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-33434: Summary: Document spark.conf.isModifiable() Key: SPARK-33434 URL: https://issues.apache.org/jira/browse/SPARK-33434 Project: Spark Issue Type:

[jira] [Commented] (SPARK-26764) [SPIP] Spark Relational Cache

2020-10-21 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218374#comment-17218374 ] Nicholas Chammas commented on SPARK-26764: -- The SPIP PDF references a design doc, but I'm not

[jira] [Commented] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2020-10-16 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215577#comment-17215577 ] Nicholas Chammas commented on SPARK-33000: -- Ctrl-D gracefully shuts down the Python REPL, so

[jira] [Commented] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2020-10-16 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215469#comment-17215469 ] Nicholas Chammas commented on SPARK-33000: -- I've tested this out a bit more, and I think the

[jira] [Commented] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2020-10-15 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215085#comment-17215085 ] Nicholas Chammas commented on SPARK-33000: -- Thanks for the explanation! I'm happy to leave this

[jira] [Commented] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2020-10-15 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214904#comment-17214904 ] Nicholas Chammas commented on SPARK-33000: -- Thanks for the pointer! No need for a new ticket.

[jira] [Created] (SPARK-33017) PySpark Context should have getCheckpointDir() method

2020-09-28 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-33017: Summary: PySpark Context should have getCheckpointDir() method Key: SPARK-33017 URL: https://issues.apache.org/jira/browse/SPARK-33017 Project: Spark

[jira] [Updated] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2020-09-28 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-33000: - Description: Maybe it's just that the documentation needs to be updated, but I found

[jira] [Created] (SPARK-33000) cleanCheckpoints config does not clean all checkpointed RDDs on shutdown

2020-09-25 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-33000: Summary: cleanCheckpoints config does not clean all checkpointed RDDs on shutdown Key: SPARK-33000 URL: https://issues.apache.org/jira/browse/SPARK-33000

[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py

2020-08-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187364#comment-17187364 ] Nicholas Chammas commented on SPARK-32084: -- Can you share a couple of examples of what you are

[jira] [Updated] (SPARK-31167) Refactor how we track Python test/build dependencies

2020-08-25 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31167: - Description: Ideally, we should have a single place to track Python development

[jira] [Updated] (SPARK-31167) Refactor how we track Python test/build dependencies

2020-08-25 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31167: - Description: Ideally, we should have a single place to track Python development

[jira] [Created] (SPARK-32686) Un-deprecate inferring DataFrame schema from list of dictionaries

2020-08-21 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-32686: Summary: Un-deprecate inferring DataFrame schema from list of dictionaries Key: SPARK-32686 URL: https://issues.apache.org/jira/browse/SPARK-32686 Project:

[jira] [Commented] (SPARK-32686) Un-deprecate inferring DataFrame schema from list of dictionaries

2020-08-21 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17182040#comment-17182040 ] Nicholas Chammas commented on SPARK-32686: -- Not sure if I have the "Affects Version" field set

[jira] [Commented] (SPARK-27623) Provider org.apache.spark.sql.avro.AvroFileFormat could not be instantiated

2020-04-22 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089917#comment-17089917 ] Nicholas Chammas commented on SPARK-27623: -- Is this perhaps just a documentation issue? i.e.

[jira] [Commented] (SPARK-31170) Spark Cli does not respect hive-site.xml and spark.sql.warehouse.dir

2020-04-21 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088997#comment-17088997 ] Nicholas Chammas commented on SPARK-31170: -- Isn't this also an issue in Spark 2.4.5?

[jira] [Commented] (SPARK-31330) Automatically label PRs based on the paths they touch

2020-04-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074217#comment-17074217 ] Nicholas Chammas commented on SPARK-31330: -- Hmm, I didn't see anything from you on the mailing

[jira] [Commented] (SPARK-31330) Automatically label PRs based on the paths they touch

2020-04-02 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074124#comment-17074124 ] Nicholas Chammas commented on SPARK-31330: -- Unfortunately, it seems I jumped the gun on sending

[jira] [Created] (SPARK-31330) Automatically label PRs based on the paths they touch

2020-04-02 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31330: Summary: Automatically label PRs based on the paths they touch Key: SPARK-31330 URL: https://issues.apache.org/jira/browse/SPARK-31330 Project: Spark

[jira] [Updated] (SPARK-31167) Refactor how we track Python test/build dependencies

2020-03-16 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31167: - Summary: Refactor how we track Python test/build dependencies (was: Refactor how we

[jira] [Updated] (SPARK-31167) Refactor how we track Python test dependencies

2020-03-16 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31167: - Summary: Refactor how we track Python test dependencies (was: Specify missing test

[jira] [Created] (SPARK-31167) Specify missing test dependencies

2020-03-16 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31167: Summary: Specify missing test dependencies Key: SPARK-31167 URL: https://issues.apache.org/jira/browse/SPARK-31167 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31155) Remove pydocstyle tests

2020-03-16 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31155: - Summary: Remove pydocstyle tests (was: Enable pydocstyle tests) > Remove pydocstyle

[jira] [Updated] (SPARK-31155) Remove pydocstyle tests

2020-03-16 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31155: - Description: pydocstyle tests have been running neither on Jenkins nor on Github. We

[jira] [Updated] (SPARK-29280) DataFrameReader should support a compression option

2020-03-15 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-29280: - Affects Version/s: (was: 3.0.0) 3.1.0 > DataFrameReader

[jira] [Created] (SPARK-31155) Enable pydocstyle tests

2020-03-14 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31155: Summary: Enable pydocstyle tests Key: SPARK-31155 URL: https://issues.apache.org/jira/browse/SPARK-31155 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-31153) Cleanup several failures in lint-python

2020-03-14 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31153: Summary: Cleanup several failures in lint-python Key: SPARK-31153 URL: https://issues.apache.org/jira/browse/SPARK-31153 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-31075) Add documentation for ALTER TABLE ... ADD PARTITION

2020-03-09 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-31075. -- Resolution: Duplicate > Add documentation for ALTER TABLE ... ADD PARTITION >

[jira] [Commented] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

2020-03-08 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054676#comment-17054676 ] Nicholas Chammas commented on SPARK-31043: -- It's working for me now (per my comment), but when

[jira] [Commented] (SPARK-31065) Empty string values cause schema_of_json() to return a schema not usable by from_json()

2020-03-08 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17054674#comment-17054674 ] Nicholas Chammas commented on SPARK-31065: -- Thanks for looking into it. I have a silly

[jira] [Created] (SPARK-31075) Add documentation for ALTER TABLE ... ADD PARTITION

2020-03-06 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31075: Summary: Add documentation for ALTER TABLE ... ADD PARTITION Key: SPARK-31075 URL: https://issues.apache.org/jira/browse/SPARK-31075 Project: Spark

[jira] [Updated] (SPARK-31041) Show Maven errors from within make-distribution.sh

2020-03-06 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31041: - Description: This works: {code:java} ./dev/make-distribution.sh \ --pip \

[jira] [Updated] (SPARK-31041) Show Maven errors from within make-distribution.sh

2020-03-06 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31041: - Summary: Show Maven errors from within make-distribution.sh (was: Make arguments to

[jira] [Commented] (SPARK-31043) Spark 3.0 built against hadoop2.7 can't start standalone master

2020-03-05 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053080#comment-17053080 ] Nicholas Chammas commented on SPARK-31043: -- FWIW I was seeing the same

[jira] [Commented] (SPARK-31065) Empty string values cause schema_of_json() to return a schema not usable by from_json()

2020-03-05 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053079#comment-17053079 ] Nicholas Chammas commented on SPARK-31065: -- Confirmed this issue is also present on

[jira] [Updated] (SPARK-31065) Empty string values cause schema_of_json() to return a schema not usable by from_json()

2020-03-05 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31065: - Affects Version/s: 3.0.0 > Empty string values cause schema_of_json() to return a

[jira] [Commented] (SPARK-31065) Empty string values cause schema_of_json() to return a schema not usable by from_json()

2020-03-05 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053052#comment-17053052 ] Nicholas Chammas commented on SPARK-31065: -- cc [~hyukjin.kwon] > Empty string values cause

[jira] [Created] (SPARK-31065) Empty string values cause schema_of_json() to return a schema not usable by from_json()

2020-03-05 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31065: Summary: Empty string values cause schema_of_json() to return a schema not usable by from_json() Key: SPARK-31065 URL: https://issues.apache.org/jira/browse/SPARK-31065

[jira] [Updated] (SPARK-31041) Make arguments to make-distribution.sh position-independent

2020-03-04 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31041: - Description: This works: {code:java} ./dev/make-distribution.sh \ --pip \

[jira] [Updated] (SPARK-31041) Make arguments to make-distribution.sh position-independent

2020-03-04 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31041: - Description: This works: ``` ./dev/make-distribution.sh \ --pip \ -Phadoop-2.7

[jira] [Updated] (SPARK-31041) Make arguments to make-distribution.sh position-independent

2020-03-04 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31041: - Description: This works:   ``` ./dev/make-distribution.sh \ --pip \ -Phadoop-2.7

[jira] [Updated] (SPARK-31041) Make arguments to make-distribution.sh position-independent

2020-03-04 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-31041: - Summary: Make arguments to make-distribution.sh position-independent (was: Make

[jira] [Created] (SPARK-31041) Make argument to make-distribution position-independent

2020-03-04 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31041: Summary: Make argument to make-distribution position-independent Key: SPARK-31041 URL: https://issues.apache.org/jira/browse/SPARK-31041 Project: Spark

[jira] [Created] (SPARK-31001) Add ability to create a partitioned table via catalog.createTable()

2020-03-01 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31001: Summary: Add ability to create a partitioned table via catalog.createTable() Key: SPARK-31001 URL: https://issues.apache.org/jira/browse/SPARK-31001 Project:

[jira] [Created] (SPARK-31000) Add ability to set table description in the catalog

2020-03-01 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-31000: Summary: Add ability to set table description in the catalog Key: SPARK-31000 URL: https://issues.apache.org/jira/browse/SPARK-31000 Project: Spark

[jira] [Resolved] (SPARK-30838) Add missing pages to documentation index

2020-02-18 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas resolved SPARK-30838. -- Resolution: Won't Fix > Add missing pages to documentation index >

[jira] [Commented] (SPARK-30838) Add missing pages to documentation index

2020-02-18 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039351#comment-17039351 ] Nicholas Chammas commented on SPARK-30838: -- Actually, it looks like the pages I wanted to add

[jira] [Updated] (SPARK-30838) Add missing pages to documentation index

2020-02-18 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-30838: - Summary: Add missing pages to documentation index (was: Add missing pages to

[jira] [Updated] (SPARK-30838) Add missing pages to documentation top navigation menu

2020-02-18 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-30838: - Description: There are a few pages tracked in `docs/` that are not linked to from the

[jira] [Updated] (SPARK-30838) Add missing pages to documentation top navigation menu

2020-02-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-30838: - Description: There are a few pages tracked in `docs/` that are not linked to from the

[jira] [Updated] (SPARK-30838) Add missing pages to documentation top navigation menu

2020-02-14 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-30838: - Description: There are a few pages tracked in `docs/` that are not linked to from the

[jira] [Created] (SPARK-30838) Add missing pages to documentation top navigation menu

2020-02-14 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-30838: Summary: Add missing pages to documentation top navigation menu Key: SPARK-30838 URL: https://issues.apache.org/jira/browse/SPARK-30838 Project: Spark

<    1   2   3   4   5   6   7   8   9   10   >