Re: Spark (K8S) IPv6 support

[Sean Owen]

I don't know about the state of IPv6 support, but yes you're right in
guessing that 3.4.0 might be released perhaps early next year.
You can always clone the source repo and build it!

On Thu, Jul 14, 2022 at 2:19 PM Valer  wrote:

> Hi,
>
> We're starting to use IPv6-only K8S cluster (EKS) which currently breaks
> spark. I've noticed SPARK-39457
> 
>  contains
> a lot of focus on this, where all the sub-tasks seem to be done and
> indicates this should come in 3.4.0, so I'd like to ask a couple of
> questions:
>
>
>- Is 3.4.0 supposed to fully support IPv6 ?
>- When should I roughly expect it to be released? I've noticed that
>3.2 released in October and 3.3 this June. Is this a somewhat stable
>release frequency (half-yearly)?
>- Is there any way currently to download a tarball with the "master" /
>"latest" version that we could run before releasing ? The apache archive
>only has actual semver'd releases.
>
>
> Thanks in advance :)
>
> Regards,
> *Valér*
>

Spark (K8S) IPv6 support

[Valer]

Hi,

We're starting to use IPv6-only K8S cluster (EKS) which currently breaks spark. 
I've noticed 
[SPARK-39457](https://issues.apache.org/jira/browse/SPARK-39457?jql=project%20%3D%20SPARK%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22IPv6%22%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC)
 contains a lot of focus on this, where all the sub-tasks seem to be done and 
indicates this should come in 3.4.0, so I'd like to ask a couple of questions:

- Is 3.4.0 supposed to fully support IPv6 ?
- When should I roughly expect it to be released? I've noticed that 3.2 
released in October and 3.3 this June. Is this a somewhat stable release 
frequency (half-yearly)?
- Is there any way currently to download a tarball with the "master" / "latest" 
version that we could run before releasing ? The apache archive only has actual 
semver'd releases.

Thanks in advance :)
Regards,
Valér

unsubscribe

[randy clinton]

-- 
I appreciate your time,

~Randy

Re: How use pattern matching in spark

[Bjørn Jørgensen]

Use
from datetime import date

today = date.today()

day = today.strftime("%d/%m/%Y")
print(day)
to get today's date.
cast it to sting testday = str(day)

Compare ==
day == df_date
True or False

use loc to get row text

test_str = test.loc[1][0]

String = list in python soo

test_str[2]

'1'




ons. 13. jul. 2022 kl. 08:25 skrev Sid :

> Hi Team,
>
> I have a dataset like the below one in .dat file:
>
> 13/07/2022abc
> PWJ   PWJABC 513213217ABC GM20 05. 6/20/39
> #01000count
>
> Now I want to extract the header and tail records which I was able to do
> it. Now, from the header, I need to extract the date and match it with the
> current system date. Also, for the tail records, I need to match the number
> of actual rows i.e 1 in my case with the values mentioned in the last row.
> That is a kind of pattern matching so that I can find '1' in the last row
> and say that the actual records and the value in the tail record matches
> with each other.
>
> How can I do this? Any links would be helpful. I think regex pattern
> matching should help.
>
> Also, I will be getting 3 formats for now i.e CSV, .DAT file and .TXT
> file.
>
> So, as per me I could do validation for all these 3 file formats using
> spark.read.text().rdd and performing intended operations on Rdds. Just the
> validation part.
>
> Therefore, wanted to understand is there any better way to achieve this?
>
> Thanks,
> Sid
>


-- 
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge

+47 480 94 297


Untitled2.pdf
Description: Adobe PDF document

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org