[jira] [Resolved] (SPARK-33790) Reduce the rpc call of getFileStatus in SingleFileEventLogFileReader

[ https://issues.apache.org/jira/browse/SPARK-33790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33790. -- Fix Version/s: 3.2.0 Assignee: dzcxzl Resolution: Fixed Issue resolved via

[jira] [Resolved] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

[ https://issues.apache.org/jira/browse/SPARK-32910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-32910. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 30668

[jira] [Assigned] (SPARK-32910) Remove UninterruptibleThread usage from KafkaOffsetReader

[ https://issues.apache.org/jira/browse/SPARK-32910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-32910: Assignee: Gabor Somogyi > Remove UninterruptibleThread usage from KafkaOffsetReader >

[jira] [Resolved] (SPARK-30946) FileStreamSourceLog/FileStreamSinkLog: leverage UnsafeRow type to serialize/deserialize entry

[ https://issues.apache.org/jira/browse/SPARK-30946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30946. -- Resolution: Won't Fix Closing as I don't see any support on this and I'm now in favor of one

[jira] [Closed] (SPARK-30946) FileStreamSourceLog/FileStreamSinkLog: leverage UnsafeRow type to serialize/deserialize entry

[ https://issues.apache.org/jira/browse/SPARK-30946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim closed SPARK-30946. > FileStreamSourceLog/FileStreamSinkLog: leverage UnsafeRow type to > serialize/deserialize entry >

[jira] [Resolved] (SPARK-33660) Update Kafka Headers Documentation in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-33660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33660. -- Fix Version/s: 3.2.0 3.0.2 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-33660) Update Kafka Headers Documentation in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-33660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33660: Assignee: German Schiavon Matteo > Update Kafka Headers Documentation in Structured

[jira] [Comment Edited] (SPARK-33638) Full support of V2 table creation in Structured Streaming writer path

[ https://issues.apache.org/jira/browse/SPARK-33638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243803#comment-17243803 ] Jungtaek Lim edited comment on SPARK-33638 at 12/4/20, 7:56 AM: I don't

[jira] [Commented] (SPARK-33638) Full support of V2 table creation in Structured Streaming writer path

[ https://issues.apache.org/jira/browse/SPARK-33638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243803#comment-17243803 ] Jungtaek Lim commented on SPARK-33638: -- I don't agree with handling this in DataStreamWriter, hence

[jira] [Updated] (SPARK-33638) Full support of V2 table creation in Structured Streaming writer path

[ https://issues.apache.org/jira/browse/SPARK-33638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-33638: - Summary: Full support of V2 table creation in Structured Streaming writer path (was: Full

[jira] [Resolved] (SPARK-33577) Add support for V1Table in stream writer table API

[ https://issues.apache.org/jira/browse/SPARK-33577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33577. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30521

[jira] [Assigned] (SPARK-33577) Add support for V1Table in stream writer table API

[ https://issues.apache.org/jira/browse/SPARK-33577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33577: Assignee: Yuanjian Li > Add support for V1Table in stream writer table API >

[jira] [Assigned] (SPARK-32863) Full outer stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-32863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-32863: Assignee: Cheng Su > Full outer stream-stream join > - > >

[jira] [Resolved] (SPARK-32863) Full outer stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-32863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-32863. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30395

[jira] [Resolved] (SPARK-32032) Eliminate deprecated poll(long) API calls to avoid infinite wait in driver

[ https://issues.apache.org/jira/browse/SPARK-32032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-32032. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29729

[jira] [Assigned] (SPARK-32032) Eliminate deprecated poll(long) API calls to avoid infinite wait in driver

[ https://issues.apache.org/jira/browse/SPARK-32032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-32032: Assignee: Gabor Somogyi > Eliminate deprecated poll(long) API calls to avoid infinite

[jira] [Resolved] (SPARK-27188) FileStreamSink: provide a new option to have retention on output files

[ https://issues.apache.org/jira/browse/SPARK-27188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-27188. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28363

[jira] [Assigned] (SPARK-27188) FileStreamSink: provide a new option to have retention on output files

[ https://issues.apache.org/jira/browse/SPARK-27188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-27188: Assignee: Jungtaek Lim > FileStreamSink: provide a new option to have retention on

[jira] [Resolved] (SPARK-30900) FileStreamSource: Avoid reading compact metadata log twice if the query stops from compact batch and restarts

[ https://issues.apache.org/jira/browse/SPARK-30900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30900. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27649

[jira] [Assigned] (SPARK-30900) FileStreamSource: Avoid reading compact metadata log twice if the query stops from compact batch and restarts

[ https://issues.apache.org/jira/browse/SPARK-30900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-30900: Assignee: Jungtaek Lim > FileStreamSource: Avoid reading compact metadata log twice if

[jira] [Resolved] (SPARK-33607) Input Rate timeline/histogram aren't rendered if built with Scala 2.13

[ https://issues.apache.org/jira/browse/SPARK-33607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33607. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30546

[jira] [Assigned] (SPARK-33440) Spark schedules on updating delegation token with 0 interval under some token provider implementation

[ https://issues.apache.org/jira/browse/SPARK-33440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33440: Assignee: Jungtaek Lim > Spark schedules on updating delegation token with 0 interval

[jira] [Resolved] (SPARK-33440) Spark schedules on updating delegation token with 0 interval under some token provider implementation

[ https://issues.apache.org/jira/browse/SPARK-33440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33440. -- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-33224) Expose watermark information on SS UI

[ https://issues.apache.org/jira/browse/SPARK-33224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33224: Assignee: Jungtaek Lim > Expose watermark information on SS UI >

[jira] [Resolved] (SPARK-33224) Expose watermark information on SS UI

[ https://issues.apache.org/jira/browse/SPARK-33224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33224. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30427

[jira] [Assigned] (SPARK-33287) Expose state custom metrics information on SS UI

[ https://issues.apache.org/jira/browse/SPARK-33287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33287: Assignee: Gabor Somogyi > Expose state custom metrics information on SS UI >

[jira] [Resolved] (SPARK-33287) Expose state custom metrics information on SS UI

[ https://issues.apache.org/jira/browse/SPARK-33287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33287. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30336

[jira] [Assigned] (SPARK-31962) Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

[ https://issues.apache.org/jira/browse/SPARK-31962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-31962: Assignee: Christopher Highman > Provide modifiedAfter and modifiedBefore options when

[jira] [Resolved] (SPARK-31962) Provide modifiedAfter and modifiedBefore options when filtering from a batch-based file data source

[ https://issues.apache.org/jira/browse/SPARK-31962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-31962. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30411

[jira] [Assigned] (SPARK-33209) Clean up unit test file UnsupportedOperationsSuite.scala

[ https://issues.apache.org/jira/browse/SPARK-33209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33209: Assignee: Cheng Su > Clean up unit test file UnsupportedOperationsSuite.scala >

[jira] [Resolved] (SPARK-33209) Clean up unit test file UnsupportedOperationsSuite.scala

[ https://issues.apache.org/jira/browse/SPARK-33209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33209. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30347

[jira] [Commented] (SPARK-33440) Spark schedules on updating delegation token with 0 interval under some token provider implementation

[ https://issues.apache.org/jira/browse/SPARK-33440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17231147#comment-17231147 ] Jungtaek Lim commented on SPARK-33440: -- I have a fix and now refining a bit. Will raise a PR soon.

[jira] [Created] (SPARK-33440) Spark schedules on updating delegation token with 0 interval under some token provider implementation

Jungtaek Lim created SPARK-33440: Summary: Spark schedules on updating delegation token with 0 interval under some token provider implementation Key: SPARK-33440 URL:

[jira] [Resolved] (SPARK-33223) Expose state information on SS UI

[ https://issues.apache.org/jira/browse/SPARK-33223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33223. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30151

[jira] [Assigned] (SPARK-33223) Expose state information on SS UI

[ https://issues.apache.org/jira/browse/SPARK-33223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33223: Assignee: Gabor Somogyi > Expose state information on SS UI >

[jira] [Resolved] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

[ https://issues.apache.org/jira/browse/SPARK-33347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33347. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30251

[jira] [Assigned] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

[ https://issues.apache.org/jira/browse/SPARK-33347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33347: Assignee: Yang Jie > Clean up useless variables in MutableApplicationInfo >

[jira] [Assigned] (SPARK-23432) Expose executor memory metrics in the web UI for executors

[ https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-23432: Assignee: Zhongwei Zhu > Expose executor memory metrics in the web UI for executors >

[jira] [Resolved] (SPARK-23432) Expose executor memory metrics in the web UI for executors

[ https://issues.apache.org/jira/browse/SPARK-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-23432. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30186

[jira] [Commented] (SPARK-33361) Dataset.observe() functionality does not work with structured streaming

[ https://issues.apache.org/jira/browse/SPARK-33361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227078#comment-17227078 ] Jungtaek Lim commented on SPARK-33361: -- cc. [~hvanhovell] > Dataset.observe() functionality does

[jira] [Resolved] (SPARK-33359) foreachBatch sink outputs wrong metrics

[ https://issues.apache.org/jira/browse/SPARK-33359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33359. -- Resolution: Not A Problem > foreachBatch sink outputs wrong metrics >

[jira] [Commented] (SPARK-33359) foreachBatch sink outputs wrong metrics

[ https://issues.apache.org/jira/browse/SPARK-33359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227029#comment-17227029 ] Jungtaek Lim commented on SPARK-33359: -- That's by design. The metric is available only for V2 sink.

[jira] [Resolved] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30294. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 26935

[jira] [Assigned] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-30294: Assignee: Jungtaek Lim > Read-only state store unnecessarily creates and deletes the

[jira] [Commented] (SPARK-33259) Joining 3 streams results in incorrect output

[ https://issues.apache.org/jira/browse/SPARK-33259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17224446#comment-17224446 ] Jungtaek Lim commented on SPARK-33259: -- I'll also link to the first JIRA issue which the problem

[jira] [Updated] (SPARK-33314) Avro reader drops rows

[ https://issues.apache.org/jira/browse/SPARK-33314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-33314: - Priority: Blocker (was: Major) > Avro reader drops rows > -- > >

[jira] [Commented] (SPARK-33280) Spark 3.0 serialization issue

[ https://issues.apache.org/jira/browse/SPARK-33280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17223274#comment-17223274 ] Jungtaek Lim commented on SPARK-33280: -- Probably better to suspect Scala 2.12 as the common thing

[jira] [Commented] (SPARK-33267) Query with having null in "in" condition against data source V2 source table supporting push down filter fails with NPE

[ https://issues.apache.org/jira/browse/SPARK-33267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17222122#comment-17222122 ] Jungtaek Lim commented on SPARK-33267: -- Will submit a PR soon. > Query with having null in "in"

[jira] [Updated] (SPARK-33267) Query with having null in "in" condition against data source V2 source table supporting push down filter fails with NPE

[ https://issues.apache.org/jira/browse/SPARK-33267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-33267: - Description: The query with having null in "in" condition against data source V2 source table

[jira] [Updated] (SPARK-33267) Query with having null in "in" condition against data source V2 source table supporting push down filter fails with NPE

[ https://issues.apache.org/jira/browse/SPARK-33267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-33267: - Summary: Query with having null in "in" condition against data source V2 source table

[jira] [Created] (SPARK-33267) Query with having null in "in" condition against data source V2 source table fails with NPE

Jungtaek Lim created SPARK-33267: Summary: Query with having null in "in" condition against data source V2 source table fails with NPE Key: SPARK-33267 URL: https://issues.apache.org/jira/browse/SPARK-33267

[jira] [Commented] (SPARK-33259) Joining 3 streams results in incorrect output

[ https://issues.apache.org/jira/browse/SPARK-33259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17221409#comment-17221409 ] Jungtaek Lim commented on SPARK-33259: -- As you already figured out, this is a known limitation, and

[jira] [Resolved] (SPARK-33215) Speed up event log download by skipping UI rebuild

[ https://issues.apache.org/jira/browse/SPARK-33215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33215. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30126

[jira] [Assigned] (SPARK-33215) Speed up event log download by skipping UI rebuild

[ https://issues.apache.org/jira/browse/SPARK-33215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33215: Assignee: Baohe Zhang > Speed up event log download by skipping UI rebuild >

[jira] [Commented] (SPARK-33240) Fail fast when fails to instantiate configured v2 session catalog

[ https://issues.apache.org/jira/browse/SPARK-33240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220487#comment-17220487 ] Jungtaek Lim commented on SPARK-33240: -- Will work on it. > Fail fast when fails to instantiate

[jira] [Created] (SPARK-33240) Fail fast when fails to instantiate configured v2 session catalog

Jungtaek Lim created SPARK-33240: Summary: Fail fast when fails to instantiate configured v2 session catalog Key: SPARK-33240 URL: https://issues.apache.org/jira/browse/SPARK-33240 Project: Spark

[jira] [Assigned] (SPARK-32862) Left semi stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-32862: Assignee: Cheng Su > Left semi stream-stream join > > >

[jira] [Resolved] (SPARK-32862) Left semi stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-32862. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30076

[jira] [Resolved] (SPARK-33232) ConcurrentAppendException while updating delta lake table

[ https://issues.apache.org/jira/browse/SPARK-33232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33232. -- Resolution: Invalid > ConcurrentAppendException while updating delta lake table >

[jira] [Commented] (SPARK-33232) ConcurrentAppendException while updating delta lake table

[ https://issues.apache.org/jira/browse/SPARK-33232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219997#comment-17219997 ] Jungtaek Lim commented on SPARK-33232: -- The issue doesn't look to be specific to "Apache Spark".

[jira] [Updated] (SPARK-32863) Full outer stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-32863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32863: - Priority: Major (was: Trivial) > Full outer stream-stream join > -

[jira] [Updated] (SPARK-32883) Stream-stream join improvement

[ https://issues.apache.org/jira/browse/SPARK-32883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32883: - Priority: Major (was: Minor) > Stream-stream join improvement > --

[jira] [Updated] (SPARK-32862) Left semi stream-stream join

[ https://issues.apache.org/jira/browse/SPARK-32862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32862: - Priority: Major (was: Minor) > Left semi stream-stream join > > >

[jira] [Updated] (SPARK-32557) Logging and Swallowing the Exception Per Entry in History Server

[ https://issues.apache.org/jira/browse/SPARK-32557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32557: - Fix Version/s: 3.0.2 > Logging and Swallowing the Exception Per Entry in History Server >

[jira] [Updated] (SPARK-33146) Encountering an invalid rolling event log folder prevents loading other applications in SHS

[ https://issues.apache.org/jira/browse/SPARK-33146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-33146: - Fix Version/s: 3.0.2 > Encountering an invalid rolling event log folder prevents loading other

[jira] [Assigned] (SPARK-33146) Encountering an invalid rolling event log folder prevents loading other applications in SHS

[ https://issues.apache.org/jira/browse/SPARK-33146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-33146: Assignee: Adam Binford > Encountering an invalid rolling event log folder prevents

[jira] [Resolved] (SPARK-33146) Encountering an invalid rolling event log folder prevents loading other applications in SHS

[ https://issues.apache.org/jira/browse/SPARK-33146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-33146. -- Fix Version/s: 3.0.2 3.1.0 Resolution: Fixed Issue resolved by pull

[jira] [Commented] (SPARK-32342) Kafka events are missing magic byte

[ https://issues.apache.org/jira/browse/SPARK-32342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213900#comment-17213900 ] Jungtaek Lim commented on SPARK-32342: -- I'm not sure we want to support Confluent SR natively. If

[jira] [Commented] (SPARK-33133) History server fails when loading invalid rolling event logs

[ https://issues.apache.org/jira/browse/SPARK-33133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213689#comment-17213689 ] Jungtaek Lim commented on SPARK-33133: -- [~Kimahriman] Thanks for the report! Let's fix the two

[jira] [Commented] (SPARK-33136) Handling nullability for complex types is broken during resolution of V2 write command

[ https://issues.apache.org/jira/browse/SPARK-33136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213612#comment-17213612 ] Jungtaek Lim commented on SPARK-33136: -- Note that AppendData in branch-2.4 is also broken as same,

[jira] [Commented] (SPARK-33136) Handling nullability for complex types is broken during resolution of V2 write command

[ https://issues.apache.org/jira/browse/SPARK-33136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213448#comment-17213448 ] Jungtaek Lim commented on SPARK-33136: -- will submit a PR soon. > Handling nullability for complex

[jira] [Created] (SPARK-33136) Handling nullability for complex types is broken during resolution of V2 write command

Jungtaek Lim created SPARK-33136: Summary: Handling nullability for complex types is broken during resolution of V2 write command Key: SPARK-33136 URL: https://issues.apache.org/jira/browse/SPARK-33136

[jira] [Commented] (SPARK-28025) HDFSBackedStateStoreProvider should not leak .crc files

[ https://issues.apache.org/jira/browse/SPARK-28025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210072#comment-17210072 ] Jungtaek Lim commented on SPARK-28025: -- [~ste...@apache.org] Sorry to ping you on the old issue.

[jira] [Resolved] (SPARK-32960) Provide better exception on temporary view against DataFrameWriterV2

[ https://issues.apache.org/jira/browse/SPARK-32960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-32960. -- Resolution: Won't Fix Superceded by SPARK-33087 > Provide better exception on temporary view

[jira] [Commented] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path

[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209310#comment-17209310 ] Jungtaek Lim commented on SPARK-30542: -- This is a limitation, not a bug. There're known 3rd party

[jira] [Created] (SPARK-33011) Promote the stability annotation to Evolving for MLEvent traits/classes

Jungtaek Lim created SPARK-33011: Summary: Promote the stability annotation to Evolving for MLEvent traits/classes Key: SPARK-33011 URL: https://issues.apache.org/jira/browse/SPARK-33011 Project:

[jira] [Commented] (SPARK-32306) `approx_percentile` in Spark SQL gives incorrect results

[ https://issues.apache.org/jira/browse/SPARK-32306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17199825#comment-17199825 ] Jungtaek Lim commented on SPARK-32306: -- 50% percentile should give you median, not average. So that

[jira] [Created] (SPARK-32960) Provide better exception on temporary view against DataFrameWriterV2

Jungtaek Lim created SPARK-32960: Summary: Provide better exception on temporary view against DataFrameWriterV2 Key: SPARK-32960 URL: https://issues.apache.org/jira/browse/SPARK-32960 Project: Spark

[jira] [Updated] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30294: - Affects Version/s: (was: 3.0.0) 3.1.0 > Read-only state store

[jira] [Updated] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30294: - Issue Type: Improvement (was: Bug) > Read-only state store unnecessarily creates and deletes

[jira] [Commented] (SPARK-30294) Read-only state store unnecessarily creates and deletes the temp file for delta file every batch

[ https://issues.apache.org/jira/browse/SPARK-30294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17198682#comment-17198682 ] Jungtaek Lim commented on SPARK-30294: -- Agree. Let me update the type. > Read-only state store

[jira] [Assigned] (SPARK-26425) Add more constraint checks in file streaming source to avoid checkpoint corruption

[ https://issues.apache.org/jira/browse/SPARK-26425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-26425: Assignee: Jungtaek Lim (was: Tathagata Das) > Add more constraint checks in file

[jira] [Resolved] (SPARK-26425) Add more constraint checks in file streaming source to avoid checkpoint corruption

[ https://issues.apache.org/jira/browse/SPARK-26425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-26425. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 25965

[jira] [Created] (SPARK-32896) Add DataStreamWriter.table API

Jungtaek Lim created SPARK-32896: Summary: Add DataStreamWriter.table API Key: SPARK-32896 URL: https://issues.apache.org/jira/browse/SPARK-32896 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-32847) Add DataStreamWriterV2 API

Jungtaek Lim created SPARK-32847: Summary: Add DataStreamWriterV2 API Key: SPARK-32847 URL: https://issues.apache.org/jira/browse/SPARK-32847 Project: Spark Issue Type: New Feature

[jira] [Assigned] (SPARK-32831) Refactor SupportsStreamingUpdate to represent actual meaning of the behavior

[ https://issues.apache.org/jira/browse/SPARK-32831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-32831: Assignee: Jungtaek Lim > Refactor SupportsStreamingUpdate to represent actual meaning of

[jira] [Resolved] (SPARK-32831) Refactor SupportsStreamingUpdate to represent actual meaning of the behavior

[ https://issues.apache.org/jira/browse/SPARK-32831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-32831. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29693

[jira] [Created] (SPARK-32831) Refactor SupportsStreamingUpdate to represent actual meaning of the behavior

Jungtaek Lim created SPARK-32831: Summary: Refactor SupportsStreamingUpdate to represent actual meaning of the behavior Key: SPARK-32831 URL: https://issues.apache.org/jira/browse/SPARK-32831

[jira] [Commented] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192574#comment-17192574 ] Jungtaek Lim commented on SPARK-24295: -- [~sta...@gmail.com] Thanks for sharing the workaround.

[jira] [Commented] (SPARK-32821) cannot group by with window in sql sentence for structured streaming with watermark

[ https://issues.apache.org/jira/browse/SPARK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192138#comment-17192138 ] Jungtaek Lim commented on SPARK-32821: -- Let's leave the fix version field be empty - the field will

[jira] [Updated] (SPARK-32821) cannot group by with window in sql sentence for structured streaming with watermark

[ https://issues.apache.org/jira/browse/SPARK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32821: - Fix Version/s: (was: 3.0.1) > cannot group by with window in sql sentence for structured

[jira] [Updated] (SPARK-32821) cannot group by with window in sql sentence for structured streaming with watermark

[ https://issues.apache.org/jira/browse/SPARK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32821: - Labels: (was: 2.1.0) > cannot group by with window in sql sentence for structured streaming

[jira] [Commented] (SPARK-27483) move the data source v2 fallback to v1 logic to an analyzer rule

[ https://issues.apache.org/jira/browse/SPARK-27483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191392#comment-17191392 ] Jungtaek Lim commented on SPARK-27483: -- Shall we elaborate more on the description if you don't

[jira] [Commented] (SPARK-27484) create the streaming writing logical plan node before query is analyzed

[ https://issues.apache.org/jira/browse/SPARK-27484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17191393#comment-17191393 ] Jungtaek Lim commented on SPARK-27484: -- Shall we elaborate more on the description if you don't

[jira] [Commented] (SPARK-32776) Limit in streaming should not be optimized away by PropagateEmptyRelation

[ https://issues.apache.org/jira/browse/SPARK-32776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17189125#comment-17189125 ] Jungtaek Lim commented on SPARK-32776: -- It sounds to be safer to mark 3.0.x to 3.0.2 until the vote

[jira] [Updated] (SPARK-32776) Limit in streaming should not be optimized away by PropagateEmptyRelation

[ https://issues.apache.org/jira/browse/SPARK-32776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32776: - Fix Version/s: (was: 3.0.1) 3.0.2 > Limit in streaming should not be

[jira] [Commented] (SPARK-32530) SPIP: Kotlin support for Apache Spark

[ https://issues.apache.org/jira/browse/SPARK-32530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188879#comment-17188879 ] Jungtaek Lim commented on SPARK-32530: -- I'm not sure the relation between supports for vendors and

[jira] [Commented] (SPARK-32672) Data corruption in some cached compressed boolean columns

[ https://issues.apache.org/jira/browse/SPARK-32672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181632#comment-17181632 ] Jungtaek Lim commented on SPARK-32672: -- Just FYI, he's a PMC member. And correctness issue goes

[jira] [Updated] (SPARK-32672) Data corruption in some cached compressed boolean columns

[ https://issues.apache.org/jira/browse/SPARK-32672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-32672: - Priority: Blocker (was: Critical) > Data corruption in some cached compressed boolean columns

[jira] [Created] (SPARK-32648) Remove unused DELETE_ACTION in FileStreamSinkLog

Jungtaek Lim created SPARK-32648: Summary: Remove unused DELETE_ACTION in FileStreamSinkLog Key: SPARK-32648 URL: https://issues.apache.org/jira/browse/SPARK-32648 Project: Spark Issue Type:

<    5   6   7   8   9   10   11   12   13   14   >