[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
Step to reproduce:
1)Use SPak Streaming application.
2) inside foreachRDD of
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev resolved SPARK-31427.
Resolution: Invalid
> Spark Structure streaming read data twice per every micro-batch.
>
[
https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev resolved SPARK-34204.
Fix Version/s: 3.1.1
Resolution: Fixed
> When use input_file_name() func all
[
https://issues.apache.org/jira/browse/SPARK-35094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751630#comment-17751630
]
Nick Hryhoriev commented on SPARK-35094:
I miss understand, do the issue resolved?
> Spark
[
https://issues.apache.org/jira/browse/SPARK-42034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686914#comment-17686914
]
Nick Hryhoriev edited comment on SPARK-42034 at 2/10/23 7:36 AM:
-
I can
[
https://issues.apache.org/jira/browse/SPARK-42034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686914#comment-17686914
]
Nick Hryhoriev commented on SPARK-42034:
I can do PR, but I need someone to target me to some
Nick Hryhoriev created SPARK-42034:
--
Summary: QueryExecutionListener and Observation API, df.observe do
not work with `foreach` action.
Key: SPARK-42034
URL: https://issues.apache.org/jira/browse/SPARK-42034
[
https://issues.apache.org/jira/browse/SPARK-36263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676265#comment-17676265
]
Nick Hryhoriev commented on SPARK-36263:
Hi, I find this feature does not work for `foreach` or
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344312#comment-17344312
]
Nick Hryhoriev commented on SPARK-11844:
[~sungwy92]
spark.hadoop.fs.s3a.fast.upload": "true",
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342447#comment-17342447
]
Nick Hryhoriev commented on SPARK-11844:
I turn off `fast.data.upload` in spark Hadoop config
[
https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340038#comment-17340038
]
Nick Hryhoriev commented on SPARK-34204:
I write every simple code to avoid it, it little bit
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17334577#comment-17334577
]
Nick Hryhoriev commented on SPARK-11844:
[~Xu_Guang_Lv] My issue is also always reproducible
[
https://issues.apache.org/jira/browse/SPARK-35094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-35094:
---
Description:
I use spark 3.1.1 and 3.0.2.
Function `from_json` return wrong result with
[
https://issues.apache.org/jira/browse/SPARK-35094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-35094:
---
Description:
I use spark 3.1.1 and 3.0.2.
Function `from_json` return wrong result with
Nick Hryhoriev created SPARK-35094:
--
Summary: Spark from_json(JsonToStruct) function return wrong
value in permissive mode in case best effort
Key: SPARK-35094
URL:
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317066#comment-17317066
]
Nick Hryhoriev edited comment on SPARK-11844 at 4/12/21, 3:05 PM:
--
I
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317874#comment-17317874
]
Nick Hryhoriev commented on SPARK-31427:
thx [~kabhwan], yes my only concern read data twice.
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317066#comment-17317066
]
Nick Hryhoriev edited comment on SPARK-11844 at 4/8/21, 2:22 PM:
-
I have
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317066#comment-17317066
]
Nick Hryhoriev edited comment on SPARK-11844 at 4/8/21, 11:50 AM:
--
I
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317066#comment-17317066
]
Nick Hryhoriev edited comment on SPARK-11844 at 4/8/21, 10:51 AM:
--
I
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317066#comment-17317066
]
Nick Hryhoriev edited comment on SPARK-11844 at 4/8/21, 10:49 AM:
--
I
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317066#comment-17317066
]
Nick Hryhoriev edited comment on SPARK-11844 at 4/8/21, 10:35 AM:
--
I
[
https://issues.apache.org/jira/browse/SPARK-11844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317066#comment-17317066
]
Nick Hryhoriev commented on SPARK-11844:
I have experienced the same issue with spark 2.4.7 and
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301281#comment-17301281
]
Nick Hryhoriev edited comment on SPARK-31427 at 3/14/21, 9:26 PM:
--
hi
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17301281#comment-17301281
]
Nick Hryhoriev commented on SPARK-31427:
hi [~xccui] , after some deep investigation, I can add:
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-31427:
---
Affects Version/s: 2.4.7
3.0.1
3.0.2
> Spark
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300777#comment-17300777
]
Nick Hryhoriev edited comment on SPARK-31427 at 3/13/21, 10:38 AM:
---
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300777#comment-17300777
]
Nick Hryhoriev edited comment on SPARK-31427 at 3/13/21, 10:21 AM:
---
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300777#comment-17300777
]
Nick Hryhoriev edited comment on SPARK-31427 at 3/13/21, 10:20 AM:
---
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300777#comment-17300777
]
Nick Hryhoriev edited comment on SPARK-31427 at 3/13/21, 10:16 AM:
---
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300777#comment-17300777
]
Nick Hryhoriev edited comment on SPARK-31427 at 3/13/21, 9:41 AM:
--
And
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300777#comment-17300777
]
Nick Hryhoriev commented on SPARK-31427:
And even more strange, when I use only `repartition`,
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300775#comment-17300775
]
Nick Hryhoriev commented on SPARK-31427:
In spark 3 on GUI, I even can see that every batch
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300773#comment-17300773
]
Nick Hryhoriev edited comment on SPARK-31427 at 3/13/21, 9:24 AM:
--
I
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300773#comment-17300773
]
Nick Hryhoriev edited comment on SPARK-31427 at 3/13/21, 9:22 AM:
--
I
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17300773#comment-17300773
]
Nick Hryhoriev commented on SPARK-31427:
I check spark 3, it's the same.
Maybe in the case of
[
https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-34204:
---
Description:
input_file_name() function damage applying projection to the physical plan of
[
https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-34204:
---
Description:
input_file_name() function damage applying projection to the physical plan of
[
https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270326#comment-17270326
]
Nick Hryhoriev commented on SPARK-34204:
Look like `spark_partition_id` have same issue.
> When
[
https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-34204:
---
Summary: When use input_file_name() func all column from file appeared in
physical plan of
[
https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-34204:
---
Description:
input_file_name() function damage applying projection to the physical plan of
[
https://issues.apache.org/jira/browse/SPARK-34204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-34204:
---
Description:
input_file_name() function damage applying projection to the physical plan of
Nick Hryhoriev created SPARK-34204:
--
Summary: input_file_name() func damage applying projection to
physical plan of query.
Key: SPARK-34204
URL: https://issues.apache.org/jira/browse/SPARK-34204
[
https://issues.apache.org/jira/browse/SPARK-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188342#comment-17188342
]
Nick Hryhoriev edited comment on SPARK-12837 at 9/1/20, 10:12 AM:
--
I
[
https://issues.apache.org/jira/browse/SPARK-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188342#comment-17188342
]
Nick Hryhoriev commented on SPARK-12837:
I have the same issue with the dataset.count. in 2.4.3
[
https://issues.apache.org/jira/browse/SPARK-30364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084033#comment-17084033
]
Nick Hryhoriev commented on SPARK-30364:
I have the same issue with the spark app on Mac OS with
[
https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082163#comment-17082163
]
Nick Hryhoriev commented on SPARK-31427:
[~kabhwan] I will try to do it, but do not expect to
Nick Hryhoriev created SPARK-31427:
--
Summary: Spark Structure streaming read data twice per every
micro-batch.
Key: SPARK-31427
URL: https://issues.apache.org/jira/browse/SPARK-31427
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968226#comment-16968226
]
Nick Hryhoriev commented on SPARK-19407:
[~uncleGen]
> defaultFS is used FileSystem.get instead
[
https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968221#comment-16968221
]
Nick Hryhoriev edited comment on SPARK-19407 at 11/6/19 9:40 AM:
-
I have
[
https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968221#comment-16968221
]
Nick Hryhoriev edited comment on SPARK-19407 at 11/6/19 9:39 AM:
-
I have
[
https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968221#comment-16968221
]
Nick Hryhoriev edited comment on SPARK-19407 at 11/6/19 9:37 AM:
-
I have
[
https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16968221#comment-16968221
]
Nick Hryhoriev commented on SPARK-19407:
I have the same issue with spark 2.4.4.
When I Use
[
https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004733#comment-16004733
]
Nick Hryhoriev edited comment on SPARK-5594 at 5/12/17 6:32 AM:
Hi,
i have
[
https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16004733#comment-16004733
]
Nick Hryhoriev commented on SPARK-5594:
---
Hi,
i have the same issue.
But in spark 2.1. But i can't
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
Step to reproduce:
1)Use SPak Streaming application.
2) inside foreachRDD of
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
Step to reproduce:
1)Use SPak Streaming application.
2) inside foreachRDD of
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
Step to reproduce:
1)Use SPak Streaming application.
2) inside foreachRDD of
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
Step to reproduce:
1)Use SPak Streaming application.
2) inside foreachRDD of
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
Step to reproduce:
1)Use SPak Streaming application.
2) inside foreachRDD of
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940113#comment-15940113
]
Nick Hryhoriev commented on SPARK-20080:
It's clear issue. Even with a link to code.
You have
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Summary: Spak streaming application do not throw serialisation exception in
foreachRDD
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev reopened SPARK-20080:
Problem with loosing exception still exist.
> Spak streaming up do not throw serialisation
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15940069#comment-15940069
]
Nick Hryhoriev commented on SPARK-20080:
[~sowen] You don't understand problem.
The problem that
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
When i try use or init org.slf4j.Logger inside foreachPartition, that extracted
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
up vote
0
down vote
favorite
When i try use or init org.slf4j.Logger inside
[
https://issues.apache.org/jira/browse/SPARK-20080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Hryhoriev updated SPARK-20080:
---
Description:
up vote
0
down vote
favorite
When i try use or init org.slf4j.Logger inside
Nick Hryhoriev created SPARK-20080:
--
Summary: Spak streaming up do not throw serialisation exception in
foreachRDD
Key: SPARK-20080
URL: https://issues.apache.org/jira/browse/SPARK-20080
Project:
68 matches
Mail list logo