Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/22575
Nice! I am looking forward to it.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/22575
How should we do if we wanna join two kafka stream and sink the result to
another stream?
---
-
To unsubscribe, e-mail
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/22575
Is this still a WIP?
Using isStreaming tag in DDL to mark if a table is streaming or not is
brilliant. It keeps compatible with batch queries sql.
If possible, I think
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/22575
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user WangTaoTheTonic closed the pull request at:
https://github.com/apache/spark/pull/16331
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/16331
[SPARK-18920][HISTORYSERVER]Update outdated date formatting
## What changes were proposed in this pull request?
Before we show "-" while the timestamp is less than 0,
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/16031
I agree with you @ajbozarth. Since only one column uses `replace` function
we can keep it same as now. If there is more data using this function in the
future we will extract it to a simple
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/16031
@srowen @ajbozarth Sorry for the delay.
I've tried the solution, but found it didn't work, as we already defined
the type(`appid-numeric`) which will override value of `sType
Github user WangTaoTheTonic commented on a diff in the pull request:
https://github.com/apache/spark/pull/16031#discussion_r89758734
--- Diff: core/src/main/resources/org/apache/spark/ui/static/historypage.js
---
@@ -78,6 +78,12 @@ jQuery.extend( jQuery.fn.dataTableExt.oSort
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/16031
[SPARK-18606][HISTORYSERVER]remove useless elements while searching
## What changes were proposed in this pull request?
When we search applications in HistoryServer
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
It is slways showing UTC in the main page, but with server timezone in
other pages like last page I've pasted. Not sure if it's true. Any way we'll
waiting for the results from @windpiger
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
It's always better to show a timezone in table header, i think, no matter
what the timezone it really uses.
But changing to show GMT/UTC always? I have to say it's a bold move even
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
I would like solution 1, as other time string in Spark UI shows, like
JobPage:
---
If your project is set up for it, you can reply to this email and have your
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
@srowen Before the code changes, browser get date string from server side,
now instead it get Date(this conclusion comes from codes
debugging(https://github.com/apache/spark/blob/master
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
@ajbozarth In browser side the timezone used to build Date from epoch time
is the one **at browser side**, not that one in **History Server side**. These
two are different in many cases. So
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
I think the problem is that Date tranfered in REST ways take no timezone,
one possible reason is :
http://stackoverflow.com/questions/23730062/use-iso-8601-dates-in-jax-rs-responses
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
Like what showed in image, user can get app infos in two ways
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
I'll post how UI works and what changes it did to be different before later
:)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15838
Is it good to go? @srowen @vanzin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/15838
[SPARK-18396]"Duration" column makes search result confused, maybe we
should make it unsearchable
## What changes were proposed in this pull request?
When we s
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
Nah i think we have a misunderstand here. @tgravescs
If understand right, what you mean is that most companies run their server
in UTC timezone. That's OK. Under that condition
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
I agree with showing the timezone with date string.
But always using GMT/UTC time is not a good choice, logs of
application(using log4j) usually are printed using local timezone
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15803
thanks for the fix.
This patch parse the timestamp instead of the Date String returned. The
REST api still return the GMT time, which is insistent with UI showing.
I've
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15176
I will close this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user WangTaoTheTonic closed the pull request at:
https://github.com/apache/spark/pull/15176
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/15176
after second glimpse, I found this current behavior will not cause problems
because `dagScheduler.handletaskCompletion` is triggerred by `CompletionEvent`.
Dagscheduler handle events one
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/15176
[SPARK-17610][CORE][SCHEDULER]The failed stage caused by FetchFailed may
never be resubmitted
## What changes were proposed in this pull request?
The improper time order
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/14605
[SPARK-17022][YARN]Handle potential deadlock in driver handling messages
## What changes were proposed in this pull request?
We directly send RequestExecutors to AM instead
Github user WangTaoTheTonic commented on the issue:
https://github.com/apache/spark/pull/14591
@andrewor14
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/14591
[SPARK-17010][MINOR][DOC]Wrong description in memory management document
## What changes were proposed in this pull request?
change the remain percent to right one
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/13409
cc @liancheng @andrewor14 Could you please review this? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/13409
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/13409
[SPARK-15667][SQL]Throw exception if columns number of outputs mismatch the
inputs
## What changes were proposed in this pull request?
We will throw exception if the columns
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/12053
[SPARK-14258]change scope of functions in KafkaCluster
## What changes were proposed in this pull request?
changing scopes of some functions to minus.
## How
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-200374692
It will not impact on actuall result, but a error stacktrace and log
showing failure will make user confused and easy to believe that the
application is failed
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-198427974
so, how about it guys?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-198742134
have you tried to reproduce the scenaro and see what happend? The
`UndeclaredThrowableException` will be caught by
`e.getCause.isInstanceOf
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-197188289
`monitorApplication` returns 2-tuple, in which FinalApplicationStatus is
not used when sc stops normally. that's why it doesn't matter what status we
set
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-197135484
the application is finished successfully(RM UI also show success state) but
log shows it failed, that's the problem i think.
yeah you're right sleep
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-197120333
@jerryshao yes, the problem is that client side's log will throw exception
and show app is failed. more details are in log i pasted.
---
If your project
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-197118357
@tgravescs I reproduce it and the error message like:
>> 16/03/16 10:29:33 INFO YarnClientSchedulerBackend: Shutting down all
executors
16/03
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-196908886
I've only observed exception caused by InterruptedException but not itself
directly, thought it should be wrapped internally. The status in RM is ok
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-196898154
for another concern about the final application status returned, we don't
need too much worry as it is barely used by the codes who invoke this.
---
If your
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-196889060
hi @tgravescs , it happened when sc stop normally in client mode. As
sc.stop will stop dagscheduler -> stop taskscheduler -> stop scheduler b
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11692#issuecomment-196325232
@srowen thanks for your comments. I've changed it, please check.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user WangTaoTheTonic commented on a diff in the pull request:
https://github.com/apache/spark/pull/11692#discussion_r55998595
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -976,6 +976,11 @@ private[spark] class Client
Github user WangTaoTheTonic commented on a diff in the pull request:
https://github.com/apache/spark/pull/11692#discussion_r55973772
--- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
@@ -976,6 +976,11 @@ private[spark] class Client
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/11692
[SPARK-13852]handle the InterruptedException caused by YARN HA switch
when sc stops, it will interrupt thread using to monitor app status.
the thread will throw an InterruptedException
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/11358#issuecomment-193776099
hi @vanzin, how about spark sql in this issue, in your view? as in spark
sql it will revoke SessionState.start in which will finally connected to
metastore
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/10238#issuecomment-169207523
It seems conflicted :(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/10238#issuecomment-167507929
Sorry for late. Think you need update codes to the newest and see if the
test cases fine.
Then we do some review and function tests.
---
If your project
Github user WangTaoTheTonic closed the pull request at:
https://github.com/apache/spark/pull/8048
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/8048#issuecomment-167198843
No prob. will close this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5664#issuecomment-156339123
ok once jacky raise the PR, I will close this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/9543#issuecomment-155067126
I think it can be used for who have their custom hive hosted on their own
maven repository.
Though using maven to download hive metastore jars is using
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5664#issuecomment-152442824
@jacek-lewandowski Okay I will rebase this in a week or so.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5664#issuecomment-152686993
@jacek-lewandowski Sure. I'm glad for this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/9113#issuecomment-147977645
with this patch, can we launch multiple thrift server instances and enable
client to submit application without sensing particular thrift server addresses
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/9113#issuecomment-148255083
Yeah I think it is a nice feature which can bring HA into spark Thrift
Server. Have you testet it yet?
---
If your project is set up for it, you can reply
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/8909#issuecomment-146812382
Nice work! As it is a blocker could we merge this into branch-1.5 too?
---
If your project is set up for it, you can reply to this email and have your
reply
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7118#issuecomment-144035049
@marmbrus How do you think this fix? As the issue priority is very high, I
think we better fix it ASAP.
---
If your project is set up for it, you can reply
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5722#issuecomment-143799075
I think we better keep backwards compatibility and treat [min, max] as a
fine grained control on exact port while the "only 8080" with max retries as
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/2994#issuecomment-143799798
Hi @harishreedharan, I think this is a nice feature which is very helpful
for user who tries to write DStream back into Kafka. The implement is very neat
too
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5722#issuecomment-143107363
Hi guys, sorry for seeing your comments so late. I think we already come up
with an agreement with two cases this patch wanna fix:
One is that some
Github user WangTaoTheTonic closed the pull request at:
https://github.com/apache/spark/pull/4505
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/4505#issuecomment-142156332
@JoshRosen @andrewor14 @srowen @pfxuan
As the network related configurations need to be considered globally, I
will close this for now.
---
If your
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7118#issuecomment-139505601
@navis Thanks you for the fix. I have tested "use $database" on local
Thrift Server and the function is ok.
There might still have a test f
Github user WangTaoTheTonic commented on a diff in the pull request:
https://github.com/apache/spark/pull/7118#discussion_r3981
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -57,10 +58,9 @@ import org.apache.spark.util
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7118#issuecomment-139136144
I tested this patch, but got this error when executing "show databases;"
using beeline,
>>15/09/10 15:11:02 INFO SessionState: Create
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/8048
[SPARK-8676][SQL]Lazy start event logger in sql application to avoid TGT
expiring in lâ¦
â¦ong connection
Now in Thrift Server/Spark SQL, it will login first in `Client.scala
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7931#issuecomment-127896125
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7931#issuecomment-128045456
@marmbrus I am not sure why it will leads to a test failed. Could you help
to check this patch and the failed reason? Thanks.
---
If your project is set up
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7931#issuecomment-128232430
@marmbrus Looks like the patch is ok with excluding hive classes. Thanks
for your guide :)
---
If your project is set up for it, you can reply to this email
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/7931
[SPARK-9596][SQL]treat hadoop classes as shared one in IsolatedClientLoader
https://issues.apache.org/jira/browse/SPARK-9596
You can merge this pull request into a Git repository by running
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7931#issuecomment-127658671
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7931#issuecomment-127808147
The error is:
stderr FAILED: SemanticException [Error 10072]: Database does not
exist: hive_test_db
[info] stderr Exception in thread main
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7931#issuecomment-127811490
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/7815#issuecomment-126584002
@rxin
Like javax.jdo.option.ConnectionPassword, until now I only find this one.
---
If your project is set up for it, you can reply to this email and have
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/7815
[SPARK-9496][SQL]do not print the password in config
https://issues.apache.org/jira/browse/SPARK-9496
We better do not print the password in log.
You can merge this pull request
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5722#issuecomment-118211126
@andrewor14 There are two motivations about this, first is mention in
https://github.com/apache/spark/pull/3314, we would like to control the retry
range when
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/4064#issuecomment-112679214
@zhzhan Hey Could you describe the error and your configurations in detail
please? As we now use Hive 13 + Hadoop 2.7 in our product and never ran
Github user WangTaoTheTonic commented on a diff in the pull request:
https://github.com/apache/spark/pull/6839#discussion_r32518242
--- Diff:
core/src/main/scala/org/apache/spark/ui/scope/RDDOperationGraph.scala ---
@@ -70,6 +70,13 @@ private[ui] class RDDOperationCluster(val id
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/6741
[SPARK-8290]spark class command builder need read SPARK_JAVA_OPTS
SPARK_JAVA_OPTS was missed in reconstructing the launcher part, we should
add it back so spark-class could read
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6741#issuecomment-110823733
From the old codes we can see daemons like Master or Worker use
`SPARK_DAEMON_JAVA_OPTS` and the rest use `SPARK_JAVA_OPTS`.
Maybe
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6741#issuecomment-110803555
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6741#issuecomment-110803206
I'm not sure I understood what u mean totally. Before SPARK_JAVA_OPTS will
be injected to any class launched by `spark-class`, but was omitted after
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6741#issuecomment-110829628
@vanzin Thanks for comments.
I already have added the missing SPARK_DRIVER_MEMORY(oh shame for not
finding that) and modified the description
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6741#issuecomment-110816364
Looks like something is wrong with RAT. Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6717#issuecomment-110992692
Oh Thanks andrew.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/6717
[SPARK-8273]Driver hangs up when yarn shutdown in client mode
In client mode, if yarn was shut down with spark application running, the
application will hang up after several retries
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6717#issuecomment-110358988
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6627#issuecomment-109951833
A high level question: how about the hive 1.0/1.1/1.2? It might be hard to
support so many versions if there's no compatibility between them.
---
If your
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/6603
[Minor]make the launcher project name consistent with others
I found this by chance while building spark and think it is better to keep
its name consistent with other sub-projects (Spark
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6545#issuecomment-107430955
Hey Sean, what we wanna fix is the load once, show them forever issue.
That is to say, when user click a link on history page, the provider will load
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6051#issuecomment-107007797
@harishreedharan How about the confirm? Should we do some changes in branh
1.4?
---
If your project is set up for it, you can reply to this email and have your
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6496#issuecomment-106998580
I've done a test with current code, it worked out fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/6496#issuecomment-106787780
What more confusion would this cause? And I cann't think of another
solution to address this perfectly so did same as `Utils.getPropertiesFromFile
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/6496
[SPARK-7945][CORE]Do trim to values in properties file
https://issues.apache.org/jira/browse/SPARK-7945
Now applications submited by org.apache.spark.launcher.Main read properties
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5722#issuecomment-106792667
Looks like @vanzin is okay about this, what more concerns do you have?
@srowen
---
If your project is set up for it, you can reply to this email and have your
1 - 100 of 460 matches
Mail list logo
