[jira] [Comment Edited] (SPARK-34427) Session window support in SS
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284362#comment-17284362 ] Jungtaek Lim edited comment on SPARK-34427 at 2/14/21, 7:28 AM: I'd say SPARK-10816 has no active progress so far because of lack of interest. Once you found the necessity then SPARK-10816 can be unblocked instead of spending another non-trivial time to reinvent the wheel, no? See SPARK-10816, there're SPIP docs, discussions, ideas around them, even perf tests on WIP PRs. These efforts were actually more than a month. What this issue provides? It's just a simple link Flink provides. No design doc, no implementation, no test. This JIRA issue basically does nothing yet. That said, it's far behind than the existing effort. You can't simply say existing PR is complicated than you have in mind unless you can prove it via similar sort of effort, SPIP doc with design doc. I pointed out JIRA issue assignee issue because you're trying to take up multiple major efforts where there're folks in community want to take anything up. You're also assigning yourself on SPARK-34198 which isn't even having a PR up for reviewing, right? Don't try to occupy major efforts in parallel and block others to even try out. was (Author: kabhwan): I'd say SPARK-10816 has no active progress so far because of lack of interest. Once you found the necessity then SPARK-10816 can be unblocked instead of spending another non-trivial time to reinvent the wheel, no? See SPARK-10816, there're SPIP docs, discussions, ideas around them, even perf tests on WIP PRs. These efforts were actually more than a month. What this issue provides? It's just a simple link Flink provides. No design doc, no implementation, no test. This JIRA issue basically does nothing yet. That said, it's far behind than the existing effort. You can't simply say existing PR is complicated than you have in mind unless you can prove it via similar sort of effort, SPIP doc with design doc. I pointed out JIRA issue assignee issue because you're trying to take up multiple major efforts where there're folks in community want to take anything up. You're also assigning yourself on SPARK-34198 which isn't even having a PR up for reviewing, right? Don't try to occupy major efforts in parallel. > Session window support in SS > > > Key: SPARK-34427 > URL: https://issues.apache.org/jira/browse/SPARK-34427 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Priority: Major > > Currently structured streaming supports two kinds of windows: tumbling window > and sliding window. Another useful window function is session window. Which > is not supported by SS. We have user requirement to use session window. We'd > like to have this support in the upstream. > About session window, there is some info: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34427) Session window support in SS
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284363#comment-17284363 ] Jungtaek Lim commented on SPARK-34427: -- If you'd like to say assignee is to avoid stepping on others toes, what if I assign myself in SPARK-10816 and claim I own the issue and I have more progress then this? Once I claim then my claim is quite true, right? > Session window support in SS > > > Key: SPARK-34427 > URL: https://issues.apache.org/jira/browse/SPARK-34427 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Priority: Major > > Currently structured streaming supports two kinds of windows: tumbling window > and sliding window. Another useful window function is session window. Which > is not supported by SS. We have user requirement to use session window. We'd > like to have this support in the upstream. > About session window, there is some info: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34427) Session window support in SS
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284362#comment-17284362 ] Jungtaek Lim commented on SPARK-34427: -- I'd say SPARK-10816 has no active progress so far because of lack of interest. Once you found the necessity then SPARK-10816 can be unblocked instead of spending another non-trivial time to reinvent the wheel, no? See SPARK-10816, there're SPIP docs, discussions, ideas around them, even perf tests on WIP PRs. These efforts were actually more than a month. What this issue provides? It's just a simple link Flink provides. No design doc, no implementation, no test. This JIRA issue basically does nothing yet. That said, it's far behind than the existing effort. You can't simply say existing PR is complicated than you have in mind unless you can prove it via similar sort of effort, SPIP doc with design doc. I pointed out JIRA issue assignee issue because you're trying to take up multiple major efforts where there're folks in community want to take anything up. You're also assigning yourself on SPARK-34198 which isn't even having a PR up for reviewing, right? Don't try to occupy major efforts in parallel. > Session window support in SS > > > Key: SPARK-34427 > URL: https://issues.apache.org/jira/browse/SPARK-34427 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Priority: Major > > Currently structured streaming supports two kinds of windows: tumbling window > and sliding window. Another useful window function is session window. Which > is not supported by SS. We have user requirement to use session window. We'd > like to have this support in the upstream. > About session window, there is some info: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34427) Session window support in SS
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284348#comment-17284348 ] L. C. Hsieh commented on SPARK-34427: - If I don't miss anything, SPARK-10816 has no active progress in more than two years. I don't know about your intention here, but I don't really see that it was actively pushed during such long time. Seems to me the effort was abandoned there and it looks totally okay to me that others can work on it, isn't? About assigning JIRA issue, I'm not sure if you really do not know but basically I remember the committers can assign JIRA issues to themselves if they are working on it. We don't assign the JIRA issues created by contributors to ourselves, because it is really unfair. For JIRA issues created for our ongoing work, this is like a convention by other committers too. It is for not step on others toes. If you are really against it in general, maybe you can raise a discussion to formally disallow it in Spark community. I'm happy to follow it if we finally have a consensus about it. Thanks. > Session window support in SS > > > Key: SPARK-34427 > URL: https://issues.apache.org/jira/browse/SPARK-34427 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Priority: Major > > Currently structured streaming supports two kinds of windows: tumbling window > and sliding window. Another useful window function is session window. Which > is not supported by SS. We have user requirement to use session window. We'd > like to have this support in the upstream. > About session window, there is some info: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34426) Add driver and executors POD logs to integration tests log when the test fails
[ https://issues.apache.org/jira/browse/SPARK-34426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284346#comment-17284346 ] Apache Spark commented on SPARK-34426: -- User 'attilapiros' has created a pull request for this issue: https://github.com/apache/spark/pull/31561 > Add driver and executors POD logs to integration tests log when the test fails > -- > > Key: SPARK-34426 > URL: https://issues.apache.org/jira/browse/SPARK-34426 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Tests >Affects Versions: 3.2.0 >Reporter: Attila Zsolt Piros >Priority: Major > > Now both the driver and executors logs are lost. > In https://spark.apache.org/developer-tools.html there is a hint: > "Getting logs from the pods and containers directly is an exercise left to > the reader." > But when the test is executed by Jenkins and there is failure we really need > the POD logs to analyze problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34426) Add driver and executors POD logs to integration tests log when the test fails
[ https://issues.apache.org/jira/browse/SPARK-34426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34426: Assignee: (was: Apache Spark) > Add driver and executors POD logs to integration tests log when the test fails > -- > > Key: SPARK-34426 > URL: https://issues.apache.org/jira/browse/SPARK-34426 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Tests >Affects Versions: 3.2.0 >Reporter: Attila Zsolt Piros >Priority: Major > > Now both the driver and executors logs are lost. > In https://spark.apache.org/developer-tools.html there is a hint: > "Getting logs from the pods and containers directly is an exercise left to > the reader." > But when the test is executed by Jenkins and there is failure we really need > the POD logs to analyze problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34426) Add driver and executors POD logs to integration tests log when the test fails
[ https://issues.apache.org/jira/browse/SPARK-34426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284345#comment-17284345 ] Apache Spark commented on SPARK-34426: -- User 'attilapiros' has created a pull request for this issue: https://github.com/apache/spark/pull/31561 > Add driver and executors POD logs to integration tests log when the test fails > -- > > Key: SPARK-34426 > URL: https://issues.apache.org/jira/browse/SPARK-34426 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Tests >Affects Versions: 3.2.0 >Reporter: Attila Zsolt Piros >Priority: Major > > Now both the driver and executors logs are lost. > In https://spark.apache.org/developer-tools.html there is a hint: > "Getting logs from the pods and containers directly is an exercise left to > the reader." > But when the test is executed by Jenkins and there is failure we really need > the POD logs to analyze problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34426) Add driver and executors POD logs to integration tests log when the test fails
[ https://issues.apache.org/jira/browse/SPARK-34426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34426: Assignee: Apache Spark > Add driver and executors POD logs to integration tests log when the test fails > -- > > Key: SPARK-34426 > URL: https://issues.apache.org/jira/browse/SPARK-34426 > Project: Spark > Issue Type: Improvement > Components: Kubernetes, Tests >Affects Versions: 3.2.0 >Reporter: Attila Zsolt Piros >Assignee: Apache Spark >Priority: Major > > Now both the driver and executors logs are lost. > In https://spark.apache.org/developer-tools.html there is a hint: > "Getting logs from the pods and containers directly is an exercise left to > the reader." > But when the test is executed by Jenkins and there is failure we really need > the POD logs to analyze problem. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284343#comment-17284343 ] Dongjoon Hyun commented on SPARK-25075: --- Thank you for sharing the progress, [~tisue]. We will try Scala 2.13.5 definitely. > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Priority: Major > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34427) Session window support in SS
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284342#comment-17284342 ] Jungtaek Lim commented on SPARK-34427: -- Just to make clear, SPARK-10816 has SPIP docs from two different groups and details/comparison docs as well. > Session window support in SS > > > Key: SPARK-34427 > URL: https://issues.apache.org/jira/browse/SPARK-34427 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Priority: Major > > Currently structured streaming supports two kinds of windows: tumbling window > and sliding window. Another useful window function is session window. Which > is not supported by SS. We have user requirement to use session window. We'd > like to have this support in the upstream. > About session window, there is some info: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34198) Add RocksDB StateStore as external module
[ https://issues.apache.org/jira/browse/SPARK-34198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284336#comment-17284336 ] Jungtaek Lim commented on SPARK-34198: -- Now we have consensus on having this, which is great. Thanks for raising the discussion and gathering consensus. Which implementation we should take as baseline remains the question. Two implementations what I'm aware of are: 1. https://github.com/chermenin/spark-states 2. https://github.com/apache/spark/pull/24922 Ideally it'd be nice to take the first one as it has been known to be the way to use RocksDB state store with SS for years. Probably we need to pursue the maintainer of the project, but in most cases they tend to be happy with donating the code (with simple PR, as it doesn't seem to be tied with specific employer) so worth trying out. Second case is probably easier to continue, as we can leverage existing PR without explicit approval from original author. What we should keep in mind is "retaining" main authorship, otherwise it wouldn't be a problem to take it over. Another crazy idea (but probably most stable among all implementations) is asking Databricks to donate the commercial implementation of RocksDB state store. I guess it should have been used by their customers for years, so proven to be stable relatively. [~rxin] Given you'd also give +1 on adding RocksDB state store to Spark codebase, any chance Databricks donate the existing implementation to Spark? Is it just me and would it be crazy idea asking this? > Add RocksDB StateStore as external module > - > > Key: SPARK-34198 > URL: https://issues.apache.org/jira/browse/SPARK-34198 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > > Currently Spark SS only has one built-in StateStore implementation > HDFSBackedStateStore. Actually it uses in-memory map to store state rows. As > there are more and more streaming applications, some of them requires to use > large state in stateful operations such as streaming aggregation and join. > Several other major streaming frameworks already use RocksDB for state > management. So it is proven to be good choice for large state usage. But > Spark SS still lacks of a built-in state store for the requirement. > We would like to explore the possibility to add RocksDB-based StateStore into > Spark SS. For the concern about adding RocksDB as a direct dependency, our > plan is to add this StateStore as an external module first. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34427) Session window support in SS
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reassigned SPARK-34427: Assignee: (was: L. C. Hsieh) > Session window support in SS > > > Key: SPARK-34427 > URL: https://issues.apache.org/jira/browse/SPARK-34427 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Priority: Major > > Currently structured streaming supports two kinds of windows: tumbling window > and sliding window. Another useful window function is session window. Which > is not supported by SS. We have user requirement to use session window. We'd > like to have this support in the upstream. > About session window, there is some info: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-34427) Session window support in SS
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-34427. -- Resolution: Duplicate > Session window support in SS > > > Key: SPARK-34427 > URL: https://issues.apache.org/jira/browse/SPARK-34427 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > > Currently structured streaming supports two kinds of windows: tumbling window > and sliding window. Another useful window function is session window. Which > is not supported by SS. We have user requirement to use session window. We'd > like to have this support in the upstream. > About session window, there is some info: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34427) Session window support in SS
[ https://issues.apache.org/jira/browse/SPARK-34427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284334#comment-17284334 ] Jungtaek Lim commented on SPARK-34427: -- We can close this one and continue from SPARK-10816. This JIRA issue loses all inputs and efforts in SPARK-10816 which were worth a month. For complexity, I can simply push back linked-list version and it'll reduce 1000+ lines. That was to address one of new requirements in SPARK-10816 and I don't think it should be addressed. I'm also OK to revisit [~XuanYuan] and decide to take one of twos. Both I and [~XuanYuan] are active in the community, so any minor issues could be handled without taking over or new implementation. One thing I would like to say is, let's not assign the JIRA issue - that is against what we do with most JIRA issues, and simply "unfair" to contributors. I'd like to see major efforts be well distributed across community. > Session window support in SS > > > Key: SPARK-34427 > URL: https://issues.apache.org/jira/browse/SPARK-34427 > Project: Spark > Issue Type: New Feature > Components: Structured Streaming >Affects Versions: 3.2.0 >Reporter: L. C. Hsieh >Assignee: L. C. Hsieh >Priority: Major > > Currently structured streaming supports two kinds of windows: tumbling window > and sliding window. Another useful window function is session window. Which > is not supported by SS. We have user requirement to use session window. We'd > like to have this support in the upstream. > About session window, there is some info: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#session-windows. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34432) add a java implementation for the simple writable data source
[ https://issues.apache.org/jira/browse/SPARK-34432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284315#comment-17284315 ] Apache Spark commented on SPARK-34432: -- User 'kevincmchen' has created a pull request for this issue: https://github.com/apache/spark/pull/31560 > add a java implementation for the simple writable data source > - > > Key: SPARK-34432 > URL: https://issues.apache.org/jira/browse/SPARK-34432 > Project: Spark > Issue Type: Test > Components: SQL, Tests >Affects Versions: 3.1.1 >Reporter: Kevin Pis >Priority: Minor > > This is a followup of https://github.com/apache/spark/pull/19269 > In #19269 , there is only a scala implementation of simple writable data > source in `DataSourceV2Suite`. > This PR adds a java implementation of it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34433) Lock jekyll version by Gemfile and Bundler
[ https://issues.apache.org/jira/browse/SPARK-34433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34433: Assignee: (was: Apache Spark) > Lock jekyll version by Gemfile and Bundler > -- > > Key: SPARK-34433 > URL: https://issues.apache.org/jira/browse/SPARK-34433 > Project: Spark > Issue Type: Improvement > Components: Build, Deploy, Documentation >Affects Versions: 3.2.0 >Reporter: Attila Zsolt Piros >Priority: Major > > The Jekyll version can be pinned to specific 4.2.0 with Gemfile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34433) Lock jekyll version by Gemfile and Bundler
[ https://issues.apache.org/jira/browse/SPARK-34433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284299#comment-17284299 ] Apache Spark commented on SPARK-34433: -- User 'attilapiros' has created a pull request for this issue: https://github.com/apache/spark/pull/31559 > Lock jekyll version by Gemfile and Bundler > -- > > Key: SPARK-34433 > URL: https://issues.apache.org/jira/browse/SPARK-34433 > Project: Spark > Issue Type: Improvement > Components: Build, Deploy, Documentation >Affects Versions: 3.2.0 >Reporter: Attila Zsolt Piros >Priority: Major > > The Jekyll version can be pinned to specific 4.2.0 with Gemfile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34433) Lock jekyll version by Gemfile and Bundler
[ https://issues.apache.org/jira/browse/SPARK-34433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34433: Assignee: Apache Spark > Lock jekyll version by Gemfile and Bundler > -- > > Key: SPARK-34433 > URL: https://issues.apache.org/jira/browse/SPARK-34433 > Project: Spark > Issue Type: Improvement > Components: Build, Deploy, Documentation >Affects Versions: 3.2.0 >Reporter: Attila Zsolt Piros >Assignee: Apache Spark >Priority: Major > > The Jekyll version can be pinned to specific 4.2.0 with Gemfile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284288#comment-17284288 ] Seth Tisue edited comment on SPARK-25075 at 2/13/21, 8:56 PM: -- We at Lightbend will be releasing Scala 2.13.5 soon (thread [https://contributors.scala-lang.org/t/scala-2-13-5-planning/4825/10),] hopefully within a week or two, with a fix for [https://github.com/scala/bug/issues/12038] was (Author: tisue): We at Lightbend will be releasing Scala 2.13.5 soon (thread [https://contributors.scala-lang.org/t/scala-2-13-5-planning/4825/10),] hopefully within a week or two, with a fix [https://github.com/scala/bug/issues/12038] > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Priority: Major > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284288#comment-17284288 ] Seth Tisue edited comment on SPARK-25075 at 2/13/21, 8:54 PM: -- We at Lightbend will be releasing Scala 2.13.5 soon (thread [https://contributors.scala-lang.org/t/scala-2-13-5-planning/4825/10),] hopefully within a week or two, with a fix [https://github.com/scala/bug/issues/12038] was (Author: tisue): We'll be releasing Scala 2.13.5 soon (thread [https://contributors.scala-lang.org/t/scala-2-13-5-planning/4825/10),] hopefully within a week or two, with a fix [https://github.com/scala/bug/issues/12038] > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Priority: Major > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25075) Build and test Spark against Scala 2.13
[ https://issues.apache.org/jira/browse/SPARK-25075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284288#comment-17284288 ] Seth Tisue commented on SPARK-25075: We'll be releasing Scala 2.13.5 soon (thread [https://contributors.scala-lang.org/t/scala-2-13-5-planning/4825/10),] hopefully within a week or two, with a fix [https://github.com/scala/bug/issues/12038] > Build and test Spark against Scala 2.13 > --- > > Key: SPARK-25075 > URL: https://issues.apache.org/jira/browse/SPARK-25075 > Project: Spark > Issue Type: Umbrella > Components: Build, MLlib, Project Infra, Spark Core, SQL >Affects Versions: 3.0.0 >Reporter: Guillaume Massé >Priority: Major > > This umbrella JIRA tracks the requirements for building and testing Spark > against the current Scala 2.13 milestone. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-34433) Lock jekyll version by Gemfile and Bundler
Attila Zsolt Piros created SPARK-34433: -- Summary: Lock jekyll version by Gemfile and Bundler Key: SPARK-34433 URL: https://issues.apache.org/jira/browse/SPARK-34433 Project: Spark Issue Type: Improvement Components: Build, Deploy, Documentation Affects Versions: 3.2.0 Reporter: Attila Zsolt Piros The Jekyll version can be pinned to specific 4.2.0 with Gemfile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34433) Lock jekyll version by Gemfile and Bundler
[ https://issues.apache.org/jira/browse/SPARK-34433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284287#comment-17284287 ] Attila Zsolt Piros commented on SPARK-34433: I am working on this. > Lock jekyll version by Gemfile and Bundler > -- > > Key: SPARK-34433 > URL: https://issues.apache.org/jira/browse/SPARK-34433 > Project: Spark > Issue Type: Improvement > Components: Build, Deploy, Documentation >Affects Versions: 3.2.0 >Reporter: Attila Zsolt Piros >Priority: Major > > The Jekyll version can be pinned to specific 4.2.0 with Gemfile. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34432) add a java implementation for the simple writable data source
[ https://issues.apache.org/jira/browse/SPARK-34432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34432: Assignee: Apache Spark > add a java implementation for the simple writable data source > - > > Key: SPARK-34432 > URL: https://issues.apache.org/jira/browse/SPARK-34432 > Project: Spark > Issue Type: Test > Components: SQL, Tests >Affects Versions: 3.1.1 >Reporter: Kevin Pis >Assignee: Apache Spark >Priority: Minor > > This is a followup of https://github.com/apache/spark/pull/19269 > In #19269 , there is only a scala implementation of simple writable data > source in `DataSourceV2Suite`. > This PR adds a java implementation of it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34432) add a java implementation for the simple writable data source
[ https://issues.apache.org/jira/browse/SPARK-34432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284264#comment-17284264 ] Apache Spark commented on SPARK-34432: -- User 'kevincmchen' has created a pull request for this issue: https://github.com/apache/spark/pull/31558 > add a java implementation for the simple writable data source > - > > Key: SPARK-34432 > URL: https://issues.apache.org/jira/browse/SPARK-34432 > Project: Spark > Issue Type: Test > Components: SQL, Tests >Affects Versions: 3.1.1 >Reporter: Kevin Pis >Priority: Minor > > This is a followup of https://github.com/apache/spark/pull/19269 > In #19269 , there is only a scala implementation of simple writable data > source in `DataSourceV2Suite`. > This PR adds a java implementation of it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34432) add a java implementation for the simple writable data source
[ https://issues.apache.org/jira/browse/SPARK-34432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34432: Assignee: (was: Apache Spark) > add a java implementation for the simple writable data source > - > > Key: SPARK-34432 > URL: https://issues.apache.org/jira/browse/SPARK-34432 > Project: Spark > Issue Type: Test > Components: SQL, Tests >Affects Versions: 3.1.1 >Reporter: Kevin Pis >Priority: Minor > > This is a followup of https://github.com/apache/spark/pull/19269 > In #19269 , there is only a scala implementation of simple writable data > source in `DataSourceV2Suite`. > This PR adds a java implementation of it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33210) Set the rebasing mode for parquet INT96 type to `EXCEPTION` by default
[ https://issues.apache.org/jira/browse/SPARK-33210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284245#comment-17284245 ] Apache Spark commented on SPARK-33210: -- User 'MaxGekk' has created a pull request for this issue: https://github.com/apache/spark/pull/31557 > Set the rebasing mode for parquet INT96 type to `EXCEPTION` by default > -- > > Key: SPARK-33210 > URL: https://issues.apache.org/jira/browse/SPARK-33210 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Maxim Gekk >Assignee: Maxim Gekk >Priority: Major > Fix For: 3.1.0 > > > The ticket aims to set the following SQL configs: > - spark.sql.legacy.parquet.int96RebaseModeInWrite > - spark.sql.legacy.parquet.int96RebaseModeInRead > to EXCEPTION by default. > The reason is let users to decide should Spark modify loaded/saved timestamps > instead of silently shifting timestamps while rebasing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-34432) add a java implementation for the simple writable data source
Kevin Pis created SPARK-34432: - Summary: add a java implementation for the simple writable data source Key: SPARK-34432 URL: https://issues.apache.org/jira/browse/SPARK-34432 Project: Spark Issue Type: Test Components: SQL, Tests Affects Versions: 3.1.1 Reporter: Kevin Pis This is a followup of https://github.com/apache/spark/pull/19269 In #19269 , there is only a scala implementation of simple writable data source in `DataSourceV2Suite`. This PR adds a java implementation of it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33210) Set the rebasing mode for parquet INT96 type to `EXCEPTION` by default
[ https://issues.apache.org/jira/browse/SPARK-33210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284244#comment-17284244 ] Apache Spark commented on SPARK-33210: -- User 'MaxGekk' has created a pull request for this issue: https://github.com/apache/spark/pull/31557 > Set the rebasing mode for parquet INT96 type to `EXCEPTION` by default > -- > > Key: SPARK-33210 > URL: https://issues.apache.org/jira/browse/SPARK-33210 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.1.0 >Reporter: Maxim Gekk >Assignee: Maxim Gekk >Priority: Major > Fix For: 3.1.0 > > > The ticket aims to set the following SQL configs: > - spark.sql.legacy.parquet.int96RebaseModeInWrite > - spark.sql.legacy.parquet.int96RebaseModeInRead > to EXCEPTION by default. > The reason is let users to decide should Spark modify loaded/saved timestamps > instead of silently shifting timestamps while rebasing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-33434) Document spark.conf.isModifiable()
[
https://issues.apache.org/jira/browse/SPARK-33434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen resolved SPARK-33434.
--
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 31483
[https://github.com/apache/spark/pull/31483]
> Document spark.conf.isModifiable()
> --
>
> Key: SPARK-33434
> URL: https://issues.apache.org/jira/browse/SPARK-33434
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, PySpark
>Affects Versions: 3.1.0
>Reporter: Nicholas Chammas
>Priority: Minor
> Fix For: 3.2.0
>
>
> PySpark's docs make no mention of {{conf.isModifiable()}}, though it appears
> to be a public method introduced in SPARK-24761.
> http://spark.apache.org/docs/3.0.1/api/python/pyspark.sql.html#pyspark.sql.SparkSession.conf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33434) Document spark.conf.isModifiable()
[
https://issues.apache.org/jira/browse/SPARK-33434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen reassigned SPARK-33434:
Assignee: Eric Lemmon
> Document spark.conf.isModifiable()
> --
>
> Key: SPARK-33434
> URL: https://issues.apache.org/jira/browse/SPARK-33434
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, PySpark
>Affects Versions: 3.1.0
>Reporter: Nicholas Chammas
>Assignee: Eric Lemmon
>Priority: Minor
> Fix For: 3.2.0
>
>
> PySpark's docs make no mention of {{conf.isModifiable()}}, though it appears
> to be a public method introduced in SPARK-24761.
> http://spark.apache.org/docs/3.0.1/api/python/pyspark.sql.html#pyspark.sql.SparkSession.conf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33434) Document spark.conf.isModifiable()
[
https://issues.apache.org/jira/browse/SPARK-33434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean R. Owen updated SPARK-33434:
-
Priority: Trivial (was: Minor)
> Document spark.conf.isModifiable()
> --
>
> Key: SPARK-33434
> URL: https://issues.apache.org/jira/browse/SPARK-33434
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, PySpark
>Affects Versions: 3.1.0
>Reporter: Nicholas Chammas
>Assignee: Eric Lemmon
>Priority: Trivial
> Fix For: 3.2.0
>
>
> PySpark's docs make no mention of {{conf.isModifiable()}}, though it appears
> to be a public method introduced in SPARK-24761.
> http://spark.apache.org/docs/3.0.1/api/python/pyspark.sql.html#pyspark.sql.SparkSession.conf
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34431) Only load hive-site.xml once
[ https://issues.apache.org/jira/browse/SPARK-34431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284183#comment-17284183 ] Apache Spark commented on SPARK-34431: -- User 'MaxGekk' has created a pull request for this issue: https://github.com/apache/spark/pull/31556 > Only load hive-site.xml once > > > Key: SPARK-34431 > URL: https://issues.apache.org/jira/browse/SPARK-34431 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.2.0 >Reporter: Maxim Gekk >Priority: Major > > Hive configs from hive-site.xml are parsed over and over again. We can > optimize this, and parse it only once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34431) Only load hive-site.xml once
[ https://issues.apache.org/jira/browse/SPARK-34431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34431: Assignee: (was: Apache Spark) > Only load hive-site.xml once > > > Key: SPARK-34431 > URL: https://issues.apache.org/jira/browse/SPARK-34431 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.2.0 >Reporter: Maxim Gekk >Priority: Major > > Hive configs from hive-site.xml are parsed over and over again. We can > optimize this, and parse it only once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34431) Only load hive-site.xml once
[ https://issues.apache.org/jira/browse/SPARK-34431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34431: Assignee: Apache Spark > Only load hive-site.xml once > > > Key: SPARK-34431 > URL: https://issues.apache.org/jira/browse/SPARK-34431 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.2.0 >Reporter: Maxim Gekk >Assignee: Apache Spark >Priority: Major > > Hive configs from hive-site.xml are parsed over and over again. We can > optimize this, and parse it only once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-34431) Only load hive-site.xml once
Maxim Gekk created SPARK-34431: -- Summary: Only load hive-site.xml once Key: SPARK-34431 URL: https://issues.apache.org/jira/browse/SPARK-34431 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.2.0 Reporter: Maxim Gekk Hive configs from hive-site.xml are parsed over and over again. We can optimize this, and parse it only once. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34430) Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available
[ https://issues.apache.org/jira/browse/SPARK-34430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34430: Assignee: Apache Spark > Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, > int) not available > > > Key: SPARK-34430 > URL: https://issues.apache.org/jira/browse/SPARK-34430 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Marco van der Linden >Assignee: Apache Spark >Priority: Trivial > Labels: pull-request-available > Fix For: 3.0.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Took us a while to figure out how to fix this with pyspark this might save a > few people a few hours... > > The documentation describes vaguely how to fix the issue, by setting a > parameter but without an actual working example. > With the given PySpark example it should hold enough information to set this > in other scenarios as well. > > > Kept the change to the docs as small as possible. > h3. What changes were proposed in this pull request? > doc update, see title > h3. Why are the changes needed? > save people time figuring out how to resolve it > h3. Does this PR introduce _any_ user-facing change? > no > h3. How was this patch tested? > no code changes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34430) Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available
[ https://issues.apache.org/jira/browse/SPARK-34430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34430: Assignee: (was: Apache Spark) > Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, > int) not available > > > Key: SPARK-34430 > URL: https://issues.apache.org/jira/browse/SPARK-34430 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Marco van der Linden >Priority: Trivial > Labels: pull-request-available > Fix For: 3.0.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Took us a while to figure out how to fix this with pyspark this might save a > few people a few hours... > > The documentation describes vaguely how to fix the issue, by setting a > parameter but without an actual working example. > With the given PySpark example it should hold enough information to set this > in other scenarios as well. > > > Kept the change to the docs as small as possible. > h3. What changes were proposed in this pull request? > doc update, see title > h3. Why are the changes needed? > save people time figuring out how to resolve it > h3. Does this PR introduce _any_ user-facing change? > no > h3. How was this patch tested? > no code changes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34430) Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available
[ https://issues.apache.org/jira/browse/SPARK-34430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284161#comment-17284161 ] Apache Spark commented on SPARK-34430: -- User 'Nozziel' has created a pull request for this issue: https://github.com/apache/spark/pull/31551 > Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, > int) not available > > > Key: SPARK-34430 > URL: https://issues.apache.org/jira/browse/SPARK-34430 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Marco van der Linden >Priority: Trivial > Labels: pull-request-available > Fix For: 3.0.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Took us a while to figure out how to fix this with pyspark this might save a > few people a few hours... > > The documentation describes vaguely how to fix the issue, by setting a > parameter but without an actual working example. > With the given PySpark example it should hold enough information to set this > in other scenarios as well. > > > Kept the change to the docs as small as possible. > h3. What changes were proposed in this pull request? > doc update, see title > h3. Why are the changes needed? > save people time figuring out how to resolve it > h3. Does this PR introduce _any_ user-facing change? > no > h3. How was this patch tested? > no code changes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34430) Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available
[ https://issues.apache.org/jira/browse/SPARK-34430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284160#comment-17284160 ] Apache Spark commented on SPARK-34430: -- User 'Nozziel' has created a pull request for this issue: https://github.com/apache/spark/pull/31551 > Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, > int) not available > > > Key: SPARK-34430 > URL: https://issues.apache.org/jira/browse/SPARK-34430 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Marco van der Linden >Priority: Trivial > Labels: pull-request-available > Fix For: 3.0.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Took us a while to figure out how to fix this with pyspark this might save a > few people a few hours... > > The documentation describes vaguely how to fix the issue, by setting a > parameter but without an actual working example. > With the given PySpark example it should hold enough information to set this > in other scenarios as well. > > > Kept the change to the docs as small as possible. > h3. What changes were proposed in this pull request? > doc update, see title > h3. Why are the changes needed? > save people time figuring out how to resolve it > h3. Does this PR introduce _any_ user-facing change? > no > h3. How was this patch tested? > no code changes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-34430) Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available
Marco van der Linden created SPARK-34430: Summary: Update index.md with a pyspark hint to avoid java.nio.DirectByteBuffer.(long, int) not available Key: SPARK-34430 URL: https://issues.apache.org/jira/browse/SPARK-34430 Project: Spark Issue Type: Documentation Components: Documentation Affects Versions: 3.0.0 Reporter: Marco van der Linden Fix For: 3.0.0 Took us a while to figure out how to fix this with pyspark this might save a few people a few hours... The documentation describes vaguely how to fix the issue, by setting a parameter but without an actual working example. With the given PySpark example it should hold enough information to set this in other scenarios as well. Kept the change to the docs as small as possible. h3. What changes were proposed in this pull request? doc update, see title h3. Why are the changes needed? save people time figuring out how to resolve it h3. Does this PR introduce _any_ user-facing change? no h3. How was this patch tested? no code changes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
