[jira] [Commented] (TEZ-4082) Reduce excessive getFileLinkInfo calls in Tez
[ https://issues.apache.org/jira/browse/TEZ-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916820#comment-16916820 ] Jonathan Eagles commented on TEZ-4082: -- Thanks, [~rajesh.balamohan]. Committed this to master and branch-0.9. > Reduce excessive getFileLinkInfo calls in Tez > - > > Key: TEZ-4082 > URL: https://issues.apache.org/jira/browse/TEZ-4082 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-4082.001.patch, TEZ-4082.002.patch, > TEZ-4082.003.patch, TEZ-4082.004.patch > > > Tez benefits from the improvements MAPREDUCE-7232, but also has some > redundancy with fileSystem usage of getFileLinkInfo (resolvePath) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (TEZ-4082) Reduce excessive getFileLinkInfo calls in Tez
[ https://issues.apache.org/jira/browse/TEZ-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4082: - Fix Version/s: 0.9.3 0.10.0 > Reduce excessive getFileLinkInfo calls in Tez > - > > Key: TEZ-4082 > URL: https://issues.apache.org/jira/browse/TEZ-4082 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Fix For: 0.10.0, 0.9.3 > > Attachments: TEZ-4082.001.patch, TEZ-4082.002.patch, > TEZ-4082.003.patch, TEZ-4082.004.patch > > > Tez benefits from the improvements MAPREDUCE-7232, but also has some > redundancy with fileSystem usage of getFileLinkInfo (resolvePath) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (TEZ-4082) Reduce excessive getFileLinkInfo calls in Tez
[
https://issues.apache.org/jira/browse/TEZ-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916039#comment-16916039
]
Jonathan Eagles commented on TEZ-4082:
--
[~rajesh.balamohan], was hoping that since TezCommonUtils is marked private
that it wouldn't be used. Now that I look closely, Hive is using some of the
APIs in TezCommonUtils, but not getTezBaseStagingPath or callers at least that
I can see. As all clients need to call
TezClientUtils::createApplicationSubmissionContext (also private) under the
hood, we can ensure that TezClientUtils::ensureStagingDirExists is called in
every case. Pig and Hive have a separate framework staging dir IIRC.
{code}
@Private
public class TezCommonUtils {
...
}
{code}
> Reduce excessive getFileLinkInfo calls in Tez
> -
>
> Key: TEZ-4082
> URL: https://issues.apache.org/jira/browse/TEZ-4082
> Project: Apache Tez
> Issue Type: Improvement
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4082.001.patch, TEZ-4082.002.patch,
> TEZ-4082.003.patch, TEZ-4082.004.patch
>
>
> Tez benefits from the improvements MAPREDUCE-7232, but also has some
> redundancy with fileSystem usage of getFileLinkInfo (resolvePath)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (TEZ-4082) Reduce excessive getFileLinkInfo calls in Tez
[ https://issues.apache.org/jira/browse/TEZ-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4082: - Attachment: TEZ-4082.004.patch > Reduce excessive getFileLinkInfo calls in Tez > - > > Key: TEZ-4082 > URL: https://issues.apache.org/jira/browse/TEZ-4082 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-4082.001.patch, TEZ-4082.002.patch, > TEZ-4082.003.patch, TEZ-4082.004.patch > > > Tez benefits from the improvements MAPREDUCE-7232, but also has some > redundancy with fileSystem usage of getFileLinkInfo (resolvePath) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (TEZ-4082) Reduce excessive getFileLinkInfo calls in Tez
[ https://issues.apache.org/jira/browse/TEZ-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4082: - Attachment: TEZ-4082.003.patch > Reduce excessive getFileLinkInfo calls in Tez > - > > Key: TEZ-4082 > URL: https://issues.apache.org/jira/browse/TEZ-4082 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-4082.001.patch, TEZ-4082.002.patch, > TEZ-4082.003.patch > > > Tez benefits from the improvements MAPREDUCE-7232, but also has some > redundancy with fileSystem usage of getFileLinkInfo (resolvePath) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (TEZ-4082) Reduce excessive getFileLinkInfo calls in Tez
[ https://issues.apache.org/jira/browse/TEZ-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4082: - Attachment: TEZ-4082.002.patch > Reduce excessive getFileLinkInfo calls in Tez > - > > Key: TEZ-4082 > URL: https://issues.apache.org/jira/browse/TEZ-4082 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-4082.001.patch, TEZ-4082.002.patch > > > Tez benefits from the improvements MAPREDUCE-7232, but also has some > redundancy with fileSystem usage of getFileLinkInfo (resolvePath) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (TEZ-4082) Reduce excessive getFileLinkInfo calls in Tez
[ https://issues.apache.org/jira/browse/TEZ-4082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4082: - Attachment: TEZ-4082.001.patch > Reduce excessive getFileLinkInfo calls in Tez > - > > Key: TEZ-4082 > URL: https://issues.apache.org/jira/browse/TEZ-4082 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-4082.001.patch > > > Tez benefits from the improvements MAPREDUCE-7232, but also has some > redundancy with fileSystem usage of getFileLinkInfo (resolvePath) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (TEZ-4082) Reduce excessive getFileLinkInfo calls in Tez
Jonathan Eagles created TEZ-4082: Summary: Reduce excessive getFileLinkInfo calls in Tez Key: TEZ-4082 URL: https://issues.apache.org/jira/browse/TEZ-4082 Project: Apache Tez Issue Type: Improvement Reporter: Jonathan Eagles Assignee: Jonathan Eagles Tez benefits from the improvements MAPREDUCE-7232, but also has some redundancy with fileSystem usage of getFileLinkInfo (resolvePath) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (TEZ-4081) Container release idle timeout exception for equal min and max values
Jonathan Eagles created TEZ-4081:
Summary: Container release idle timeout exception for equal min
and max values
Key: TEZ-4081
URL: https://issues.apache.org/jira/browse/TEZ-4081
Project: Apache Tez
Issue Type: Improvement
Reporter: Jonathan Eagles
Documentation is that Max must be greater than or equal to the minimum values.
{code}
/**
* Int value. The maximum amount of time to hold on to a container if no task
can be
* assigned to it immediately. Only active when reuse is enabled. The value
* must be +ve and >=
* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS.
* Containers will have an expire time set to a random value between
* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MIN_MILLIS &&
* TezConfiguration#TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS. This
* creates a graceful reduction in the amount of idle resources held
*/
@ConfigurationScope(Scope.AM)
@ConfigurationProperty(type="long")
public static final String TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS =
TEZ_AM_PREFIX + "container.idle.release-timeout-max.millis";
public static final long
TEZ_AM_CONTAINER_IDLE_RELEASE_TIMEOUT_MAX_MILLIS_DEFAULT = 1l;
{code}
Yet equal values of min and max cause this exception.
{code}
Service Error: Error in TaskScheduler for handling Task De-allocation,
eventType=S_TA_ENDED, scheduler=[0:TezYarn],
taskAttemptId=attempt_1564035733375_382652_1_00_51_0,
eventType=TASK_SCHEDULER_SERVICE_FATAL_ERROR,
exception=org.apache.commons.math3.exception.NumberIsTooLargeException: lower
bound (60,000) must be strictly less than upper bound (60,000)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (TEZ-4080) TezClient should close FileSystem objects to prevent leak
[
https://issues.apache.org/jira/browse/TEZ-4080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892082#comment-16892082
]
Jonathan Eagles commented on TEZ-4080:
--
[~kgyrtkirk], took a look at the patch and description to understand the
problem better. The design of the FileSystem cache is that there are three
parts to the key in getting a cache hit. 1) Each new FileSystem instance has a
unique URI, 2) is given an immutable globally unique ID, and 3) a UGI
(UserGroupInformation) that created the filesystem. This leads me to believe
that there are multiple UGIs at play in this scenario that prevent the cache
from working properly.
Quick glance at createRemoteUser calls, noting that each createRemoteUser call
is considered unique regardless of whether the user is the same as previous
users.
{code}
// get a list of non-test createRemoteUser call
// $ git grep createRemoteUser | grep -v test
tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java:
UserGroupInformation userUgi =
UserGroupInformation.createRemoteUser(UserGroupInformation
tez-api/src/main/java/org/apache/tez/common/security/JobTokenIdentifier.java:
return UserGroupInformation.createRemoteUser(jobid.toString());
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
.createRemoteUser(jobUserName);
tez-dag/src/main/java/org/apache/tez/dag/app/dag/impl/DAGImpl.java: dagUGI
= UserGroupInformation.createRemoteUser(this.userName);
tez-dag/src/main/java/org/apache/tez/dag/app/web/AMWebController.java:
callerUGI = UserGroupInformation.createRemoteUser(remoteUser);
tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java:
UserGroupInformation taskOwner =
UserGroupInformation.createRemoteUser(tokenIdentifier);
tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java:
childUGI = UserGroupInformation.createRemoteUser(user);
{code}
Best practices for FileSystem cleanup per UGI is to use the
FileSystem.closeAllForUGI.
{code}
// get a list of file systems that were closed per UGI, excluding tests
$ git grep FileSystem.closeAllForUGI | grep -v test
tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java:
FileSystem.closeAllForUGI(context.getCurrentDAG().getDagUGI());
tez-runtime-internals/src/main/java/org/apache/tez/runtime/task/TezChild.java:
FileSystem.closeAllForUGI(childUGI);
{code}
>From this, we can conclude that there is potentially some file systems left in
>the cache for each of the UGI's that were left open (assuming some filesystems
>were opened using this UGI). Your analysis shows the same.
Could you change from closing individual file systems to closing all file
systems for the UGI?
> TezClient should close FileSystem objects to prevent leak
> -
>
> Key: TEZ-4080
> URL: https://issues.apache.org/jira/browse/TEZ-4080
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: Zoltan Haindrich
>Priority: Major
> Attachments: TEZ-4080.branch-0.9.1.patch
>
>
> When opening/closing a lot of tez clients; some FileSystem object references
> are retained even after the client is closed - due to the fact the
> FileSystem has a "cache" which collects all open FileSystem objects - to be
> able to close all of them from a single shutdownhook.
> Not closing these FileSystem objects causes them to pile up in the "cache"
> which has hard references to them
> In a simple hive test which was run with 150M of memory; these "lost"
> filesystem objects could result in an OOM after ~170 sessions.
> A sample creation stack trace of a FileSystem object:
> {code}
> at
> org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem.(ProxyLocalFileSystem.java:49)
> at sun.reflect.GeneratedConstructorAccessor83.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3353)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:502)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
[jira] [Commented] (TEZ-4079) Limit the size of broadcast data
[ https://issues.apache.org/jira/browse/TEZ-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891322#comment-16891322 ] Jonathan Eagles commented on TEZ-4079: -- Perhaps logic could be added to BroadcastEdgeManager to fail if a max threshold is breached. Broadcast size can only be know during runtime currently. > Limit the size of broadcast data > > > Key: TEZ-4079 > URL: https://issues.apache.org/jira/browse/TEZ-4079 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Priority: Major > > Since broadcast data is fetched all from a single node currently. Large > broadcast data can DDOS shuffle handler with sufficiently high number of > downstream tasks. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (TEZ-4079) Limit the size of broadcast data
[ https://issues.apache.org/jira/browse/TEZ-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4079: - Description: Since broadcast data is fetched all from a single node currently. Large broadcast data can DDOS shuffle handler with sufficiently high number of downstream tasks. > Limit the size of broadcast data > > > Key: TEZ-4079 > URL: https://issues.apache.org/jira/browse/TEZ-4079 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Priority: Major > > Since broadcast data is fetched all from a single node currently. Large > broadcast data can DDOS shuffle handler with sufficiently high number of > downstream tasks. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (TEZ-4079) Limit the size of broadcast data
Jonathan Eagles created TEZ-4079: Summary: Limit the size of broadcast data Key: TEZ-4079 URL: https://issues.apache.org/jira/browse/TEZ-4079 Project: Apache Tez Issue Type: Improvement Reporter: Jonathan Eagles -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (TEZ-4077) Exclude hadoop mapreduce jars from tez minimal distribution
[ https://issues.apache.org/jira/browse/TEZ-4077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884189#comment-16884189 ] Jonathan Eagles commented on TEZ-4077: -- [~aditya-shah], there is some more discussions on the mapreduce jar removal in TEZ-3463. I have run with the patch in that JIRA for over 2 years with good results. > Exclude hadoop mapreduce jars from tez minimal distribution > --- > > Key: TEZ-4077 > URL: https://issues.apache.org/jira/browse/TEZ-4077 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 >Reporter: Aditya Shah >Priority: Major > Attachments: TEZ-4077.branch-0.9.1.patch > > > There are hadoop-mapreduce-client-common-2.7.0.jar and > hadoop-mapreduce-client-core-2.7.0.jar for tez version 0.9.1 in > /tez-dist/target/tez-0.9.1-qds-1.0-SNAPSHOT-minimal/lib/ which is redundant > and not required. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (TEZ-4078) Issue with Tez execution engine when the default FS is referred to S3
[ https://issues.apache.org/jira/browse/TEZ-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884099#comment-16884099 ] Jonathan Eagles commented on TEZ-4078: -- [~lkammili], could you post a stack trace or task log that shows the failure. That will help to quickly debug. > Issue with Tez execution engine when the default FS is referred to S3 > - > > Key: TEZ-4078 > URL: https://issues.apache.org/jira/browse/TEZ-4078 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.9.1 > Environment: AWS >Reporter: Laks >Priority: Major > > In AWS when we have changed default FS of hive-site.xml file from HDFS to S3 > and executing hive "INSERT" statement we are receiving the issue. > > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.tez.TezTask > > Incomplete HDFS URI > > But when i updated the hive execution engine to MR i am able to execute the > INSERT statement and this looks to me as bug that needs to be fixed. > > Please suggest. > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (TEZ-4069) Avoid repeated computation of preferred locations in split grouping.
[ https://issues.apache.org/jira/browse/TEZ-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842796#comment-16842796 ] Jonathan Eagles commented on TEZ-4069: -- [~odraese], I don't have enough knowledge about the HostAffinitySplitLocationProvider to make a decision about this. Could you help me understand what product and usage? In your experience, how much CPU savings (versus memory cost) are we see. How many splits? > Avoid repeated computation of preferred locations in split grouping. > > > Key: TEZ-4069 > URL: https://issues.apache.org/jira/browse/TEZ-4069 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.9.2 >Reporter: Oliver Draese >Priority: Major > Attachments: TEZ-4069.1.patch, TEZ-4069.patch > > > The TezSplitGrouper iterates through the list of splits multiple times, when > trying to group the splits (see getGroupedSplits). Each time, it asks the > locationProvider to return the array of preferred locations for the splits. > This has two side effects: > * generating the list of preferred locations can cause some CPU overhead > (i.e. calculating the consistent hash in HostAffinitySplitLocationProvider), > which can be avoided > * if the list of preferred location is changing between the different loops > of getGroupedSplits, we might encounter a NullPointerException. This happens > if a new location appears, that was not part of the initial set of locations > when populating the distinctLocations map. > The getGroupedSplits should query the preferred locations only once (for each > split) via the location provider and then memorize these instead of asking > the location provider repeatedly. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3894) Tez intermediate outputs implicitly rely on permissive umask for shuffle
[ https://issues.apache.org/jira/browse/TEZ-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842776#comment-16842776 ] Jonathan Eagles commented on TEZ-3894: -- [~tarekabouzeid91], as this jira is fixed in 0.9.2 (you state you are running a vender version based on 0.9.1), I would consider upgrading versions of the tez to 0.9.2. Otherwise, it will be best to get debugging results through the vendor or through the tez user group (u...@tez.apache.org) > Tez intermediate outputs implicitly rely on permissive umask for shuffle > > > Key: TEZ-3894 > URL: https://issues.apache.org/jira/browse/TEZ-3894 > Project: Apache Tez > Issue Type: Bug >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Major > Fix For: 0.9.2 > > Attachments: TEZ-3894.001.patch > > > Tez does not explicitly set the permissions of intermediate output files for > shuffle. In a secure cluster the shuffle service is running as a different > user than the task, so the output files require group readability in order to > serve up the data during the shuffle phase. If the umask is too restrictive > (e.g.: 077) then the task's file.out and file.out.index permissions can be > too restrictive to allow the shuffle handler to access them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4070) SSLFactory not closed in DAGClientTimelineImpl caused native memory issues
[
https://issues.apache.org/jira/browse/TEZ-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16842767#comment-16842767
]
Jonathan Eagles commented on TEZ-4070:
--
Thanks for filing this jira, [~renxunsaky] I need to look into this further to
see what the right fix is. It may be right to shutdown the SSLFactory, but
perhaps it could be reused.
> SSLFactory not closed in DAGClientTimelineImpl caused native memory issues
> --
>
> Key: TEZ-4070
> URL: https://issues.apache.org/jira/browse/TEZ-4070
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Xun REN
>Priority: Major
>
> Hi,
> Recently, we're facing native memory issues on Redhat servers. It crashed
> completely our servers.
> *Context:*
> - HDP-2.6.5
> - Redhat 7.4
> *Problem:*
> After upgrading from HDP-2.6.2 to HDP-2.6.5, after several days running, our
> HiveServer2 can eat up to more than 100GB memory. However, we have configured
> Xmx20G and MaxMetaspace to 10GB.
> After searching, we have found the similar issue here:
> https://issues.apache.org/jira/browse/YARN-5309
> This is fixed in the hadoop-common module. Our version includes already this
> issue, however, we still have the problem.
> After searching, I have found that in the class
> org.apache.tez.dag.api.client.TimelineReaderFactory of Tez, if HTTPS is used
> for YARN, it will create SSLFactory which is not destroyed after utilization.
> TimelineReaderFactory is referenced in the class DAGClientTimelineImpl.
> If ATS is used and DAG is completed, the method switchToTimelineClient in the
> class DAGClientImpl will be called. It will close the previous HTTPClient,
> but not the SSLFactory inside. And the SSLFactory will create a thread for
> each connection. Finally, we will get thousands of threads consuming a lot
> native memories.
> {code:java}
> private void switchToTimelineClient() throws IOException, TezException {
> realClient.close();
> realClient = new DAGClientTimelineImpl(appId, dagId, conf, frameworkClient,
> (int) (2 * PRINT_STATUS_INTERVAL_MILLIS));
> if (LOG.isDebugEnabled()) {
> LOG.debug("dag completed switching to DAGClientTimelineImpl");
> }
> }{code}
> I have checked on another environment which is still on HDP-2.6.2, we also
> have a lot of running threads holding by SSLFactory. That means the problem
> is zoomed in the version HDP-2.6.5
>
> *How to reproduce the problem:*
> 1. Use Tez as Hive execution engine
> 2. Launch a Beeline session for Hive
> 3. Do a select with a simple where clause on a table
> 4. Repeat steps 2-3 in order to open different connections (it is the case
> for a shared cluster with multiple clients).
> Finally, you can check in the thread dump file, that a lot of threads are
> named "Truststore reloader thread". And the native memory usage is very high
> by doing the command "top" or "ps".
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TEZ-4068) Prevent new speculative attempt after task has issued canCommit to an attempt
[ https://issues.apache.org/jira/browse/TEZ-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-4068. -- Resolution: Fixed Fix Version/s: 0.9.3 0.10.1 Thanks [~Chyler] for patch and [~yingdachen] for review. Committed patch to master and branch-0.9. > Prevent new speculative attempt after task has issued canCommit to an attempt > - > > Key: TEZ-4068 > URL: https://issues.apache.org/jira/browse/TEZ-4068 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Assignee: Ying Han >Priority: Major > Fix For: 0.10.1, 0.9.3 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > When a running attempt calls TaskImpl#canCommit through the taskUmbilical, > the TaskImpl will issue a "go" if it is the first attempt to do so. Otherwise > it will issue a "no-go". After commitAttempt is assigned is TaskImpl, no > other attempt is allowed to succeed at that point. So a speculative attempt > that is launched after commitAttempt is assigned can never finished before > the original since is will allows be given a "no-go" in the canCommit > response. In this jira, I propose to discuss disabling speculative attempts > after commitAttempt has been assigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (TEZ-2249) Wait for all task attempt finished before moving Task to finished state
[ https://issues.apache.org/jira/browse/TEZ-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835901#comment-16835901 ] Jonathan Eagles edited comment on TEZ-2249 at 5/8/19 9:38 PM: -- TEZ-4068 may be a way to prevent the likelihood of temporary directories being created after task has committed and before vertex has committed. This JIRA, on the other hand, would permanently prevent that case. was (Author: jeagles): TEZ-4068 may be a way to prevent the likelihood temporary directories being created after task has committed and before vertex has committed. > Wait for all task attempt finished before moving Task to finished state > --- > > Key: TEZ-2249 > URL: https://issues.apache.org/jira/browse/TEZ-2249 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Attachments: TEZ-2249-1.patch > > > 2 cases: > * If Task needs to move the SUCCEEDED, then committing may happens while > there's still task attempt running. > * If Tasks needs to move to FAILED/KILLED/ERROD, then aborting may happens > while there's still task attempt running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-2249) Wait for all task attempt finished before moving Task to finished state
[ https://issues.apache.org/jira/browse/TEZ-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835901#comment-16835901 ] Jonathan Eagles commented on TEZ-2249: -- TEZ-4068 may be a way to prevent the likelihood temporary directories being created after task has committed and before vertex has committed. > Wait for all task attempt finished before moving Task to finished state > --- > > Key: TEZ-2249 > URL: https://issues.apache.org/jira/browse/TEZ-2249 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Attachments: TEZ-2249-1.patch > > > 2 cases: > * If Task needs to move the SUCCEEDED, then committing may happens while > there's still task attempt running. > * If Tasks needs to move to FAILED/KILLED/ERROD, then aborting may happens > while there's still task attempt running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4068) Prevent new speculative attempt after task has issued canCommit to an attempt
[ https://issues.apache.org/jira/browse/TEZ-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835897#comment-16835897 ] Jonathan Eagles commented on TEZ-4068: -- [~Chyler], This change in behavior is similar to the TaskImpl state machine change made in TEZ-4062. I would like to hear your thoughts on this jira and whether it is a good change or not. > Prevent new speculative attempt after task has issued canCommit to an attempt > - > > Key: TEZ-4068 > URL: https://issues.apache.org/jira/browse/TEZ-4068 > Project: Apache Tez > Issue Type: Improvement >Reporter: Jonathan Eagles >Priority: Major > > When a running attempt calls TaskImpl#canCommit through the taskUmbilical, > the TaskImpl will issue a "go" if it is the first attempt to do so. Otherwise > it will issue a "no-go". After commitAttempt is assigned is TaskImpl, no > other attempt is allowed to succeed at that point. So a speculative attempt > that is launched after commitAttempt is assigned can never finished before > the original since is will allows be given a "no-go" in the canCommit > response. In this jira, I propose to discuss disabling speculative attempts > after commitAttempt has been assigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4068) Prevent new speculative attempt after task has issued canCommit to an attempt
Jonathan Eagles created TEZ-4068: Summary: Prevent new speculative attempt after task has issued canCommit to an attempt Key: TEZ-4068 URL: https://issues.apache.org/jira/browse/TEZ-4068 Project: Apache Tez Issue Type: Improvement Reporter: Jonathan Eagles When a running attempt calls TaskImpl#canCommit through the taskUmbilical, the TaskImpl will issue a "go" if it is the first attempt to do so. Otherwise it will issue a "no-go". After commitAttempt is assigned is TaskImpl, no other attempt is allowed to succeed at that point. So a speculative attempt that is launched after commitAttempt is assigned can never finished before the original since is will allows be given a "no-go" in the canCommit response. In this jira, I propose to discuss disabling speculative attempts after commitAttempt has been assigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-2249) Wait for all task attempt finished before moving Task to finished state
[ https://issues.apache.org/jira/browse/TEZ-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835691#comment-16835691 ] Jonathan Eagles commented on TEZ-2249: -- Looked at MapReduce for a similar feature, but there is none. It is susceptible to the same race condition. I have seen this occur recently and the outcome can be bad since temporary directory (and presumably files) can show up after the vertex stage commits. If subsequent stages are triggered based on a SUCCESS file being written, this can cause issues and contents change after the SUCCESS marker is created (a '_SUCCESS' file). If there is still interest, I could help work on this patch (giving [~zjffdu] proper credit) as assignee isn't able to work on this. > Wait for all task attempt finished before moving Task to finished state > --- > > Key: TEZ-2249 > URL: https://issues.apache.org/jira/browse/TEZ-2249 > Project: Apache Tez > Issue Type: Bug >Reporter: Jeff Zhang >Assignee: Jeff Zhang >Priority: Major > Attachments: TEZ-2249-1.patch > > > 2 cases: > * If Task needs to move the SUCCEEDED, then committing may happens while > there's still task attempt running. > * If Tasks needs to move to FAILED/KILLED/ERROD, then aborting may happens > while there's still task attempt running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-746) DAG and Vertex commit should ensure that all tasks have completed before committing
[ https://issues.apache.org/jira/browse/TEZ-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-746. - Resolution: Duplicate > DAG and Vertex commit should ensure that all tasks have completed before > committing > --- > > Key: TEZ-746 > URL: https://issues.apache.org/jira/browse/TEZ-746 > Project: Apache Tez > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Major > > There may be a race when a task (say a speculative task) is still running. > Then the committer commits (say lists & cleans some files in the output dir). > Then that task writes some temporary files in the output dir and gets killed. > Those temp files may remain behind. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4066) Upgrade servlet-api from 2.5 to 3.1.0
[
https://issues.apache.org/jira/browse/TEZ-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4066:
-
Attachment: TEZ-4066.002.patch
> Upgrade servlet-api from 2.5 to 3.1.0
> -
>
> Key: TEZ-4066
> URL: https://issues.apache.org/jira/browse/TEZ-4066
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4066.001.patch, TEZ-4066.002.patch
>
>
> Oozie launcher jobs trying to launch Tez jobs now fail to render Oozie
> Launcher Job AM due to both 2.5 (from tez) and 3.1.0 (from hadoop)
> servlet-api both being in the classpath. Tez should sync with servlet api
> version from tez master branch that only supports hadoop 3+
> {code}
> 2019-04-30 14:53:02,747 WARN [qtp1213419524-119]
> org.eclipse.jetty.server.HttpChannel:
> java.lang.NoSuchMethodError:
> javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:688)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4066) Upgrade servlet-api from 2.5 to 3.1.0
[
https://issues.apache.org/jira/browse/TEZ-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834235#comment-16834235
]
Jonathan Eagles commented on TEZ-4066:
--
Thanks, [~kshukla]. Change the jar name in the license files in patch 002
> Upgrade servlet-api from 2.5 to 3.1.0
> -
>
> Key: TEZ-4066
> URL: https://issues.apache.org/jira/browse/TEZ-4066
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4066.001.patch, TEZ-4066.002.patch
>
>
> Oozie launcher jobs trying to launch Tez jobs now fail to render Oozie
> Launcher Job AM due to both 2.5 (from tez) and 3.1.0 (from hadoop)
> servlet-api both being in the classpath. Tez should sync with servlet api
> version from tez master branch that only supports hadoop 3+
> {code}
> 2019-04-30 14:53:02,747 WARN [qtp1213419524-119]
> org.eclipse.jetty.server.HttpChannel:
> java.lang.NoSuchMethodError:
> javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:688)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4066) Upgrade servlet-api from 2.5 to 3.1.0
[
https://issues.apache.org/jira/browse/TEZ-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833934#comment-16833934
]
Jonathan Eagles commented on TEZ-4066:
--
This should only be committed to master branch as branch-0.9 has a different
requirement for hadoop dependency
> Upgrade servlet-api from 2.5 to 3.1.0
> -
>
> Key: TEZ-4066
> URL: https://issues.apache.org/jira/browse/TEZ-4066
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4066.001.patch
>
>
> Oozie launcher jobs trying to launch Tez jobs now fail to render Oozie
> Launcher Job AM due to both 2.5 (from tez) and 3.1.0 (from hadoop)
> servlet-api both being in the classpath. Tez should sync with servlet api
> version from tez master branch that only supports hadoop 3+
> {code}
> 2019-04-30 14:53:02,747 WARN [qtp1213419524-119]
> org.eclipse.jetty.server.HttpChannel:
> java.lang.NoSuchMethodError:
> javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:688)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4066) Upgrade servlet-api from 2.5 to 3.1.0
[
https://issues.apache.org/jira/browse/TEZ-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4066:
-
Attachment: TEZ-4066.001.patch
> Upgrade servlet-api from 2.5 to 3.1.0
> -
>
> Key: TEZ-4066
> URL: https://issues.apache.org/jira/browse/TEZ-4066
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4066.001.patch
>
>
> Oozie launcher jobs trying to launch Tez jobs now fail to render Oozie
> Launcher Job AM due to both 2.5 (from tez) and 3.1.0 (from hadoop)
> servlet-api both being in the classpath. Tez should sync with servlet api
> version from tez master branch that only supports hadoop 3+
> {code}
> 2019-04-30 14:53:02,747 WARN [qtp1213419524-119]
> org.eclipse.jetty.server.HttpChannel:
> java.lang.NoSuchMethodError:
> javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:688)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4066) Upgrade servlet-api from 2.5 to 3.1.0
[
https://issues.apache.org/jira/browse/TEZ-4066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832811#comment-16832811
]
Jonathan Eagles commented on TEZ-4066:
--
More to the point, we can see that hadoop updated jetty and servlet-api at the
same time while tez only updated jetty and not servlet-api
> Upgrade servlet-api from 2.5 to 3.1.0
> -
>
> Key: TEZ-4066
> URL: https://issues.apache.org/jira/browse/TEZ-4066
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
>
> Oozie launcher jobs trying to launch Tez jobs now fail to render Oozie
> Launcher Job AM due to both 2.5 (from tez) and 3.1.0 (from hadoop)
> servlet-api both being in the classpath. Tez should sync with servlet api
> version from tez master branch that only supports hadoop 3+
> {code}
> 2019-04-30 14:53:02,747 WARN [qtp1213419524-119]
> org.eclipse.jetty.server.HttpChannel:
> java.lang.NoSuchMethodError:
> javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:688)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4066) Upgrade servlet-api from 2.5 to 3.1.0
Jonathan Eagles created TEZ-4066:
Summary: Upgrade servlet-api from 2.5 to 3.1.0
Key: TEZ-4066
URL: https://issues.apache.org/jira/browse/TEZ-4066
Project: Apache Tez
Issue Type: Bug
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Oozie launcher jobs trying to launch Tez jobs now fail to render Oozie Launcher
Job AM due to both 2.5 (from tez) and 3.1.0 (from hadoop) servlet-api both
being in the classpath. Tez should sync with servlet api version from tez
master branch that only supports hadoop 3+
{code}
2019-04-30 14:53:02,747 WARN [qtp1213419524-119]
org.eclipse.jetty.server.HttpChannel:
java.lang.NoSuchMethodError:
javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:688)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-1348) Allow Tez local mode to run against filesystems other than local FS
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831149#comment-16831149 ] Jonathan Eagles commented on TEZ-1348: -- [~sseth], it looks like the wrong version of the patch was committed and the cherry-picked. Do you have time to fix? > Allow Tez local mode to run against filesystems other than local FS > --- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task > Environment: Committed to branch-0.9. >Reporter: Siddharth Seth >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.9.2, 0.10.1 > > Attachments: tez-1348.patch, tez-1348.patch, tez-1348.txt > > Time Spent: 20m > Remaining Estimate: 0h > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-1348) Allow Tez local mode to run against filesystems other than local FS
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829373#comment-16829373 ] Jonathan Eagles commented on TEZ-1348: -- [~sseth], branch-0.9 is still active. Please cherry-pick to that line if possible. I'm in the process of getting an updated status for 0.10.0 release. For now, Don't worry about that. > Allow Tez local mode to run against filesystems other than local FS > --- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.10.1 > > Attachments: tez-1348.patch, tez-1348.patch, tez-1348.txt > > Time Spent: 20m > Remaining Estimate: 0h > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4062) Speculative attempt scheduling should be aborted when Task has completed
[ https://issues.apache.org/jira/browse/TEZ-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827256#comment-16827256 ] Jonathan Eagles commented on TEZ-4062: -- +1. Committing this to master and branch-0.9. [~yingdachen], as to the origin of build failure, there is some details in TEZ-4065 on the proper fix. Essentially, Yetus is having some difficulty with determining the root of the patch and improperly assumes tez-dag as the top level directory. Since dependent modules in Tez were not built, 0.10.1-SNAPSHOT dependencies were downloaded, but were outdated. To bypass the issue, I built new 0.10.1-SNAPSHOT snapshot artifacts and published them to allow this build to pass until Yetus patch issue is fixed. > Speculative attempt scheduling should be aborted when Task has completed > > > Key: TEZ-4062 > URL: https://issues.apache.org/jira/browse/TEZ-4062 > Project: Apache Tez > Issue Type: Bug >Reporter: Yingda Chen >Assignee: Ying Han >Priority: Major > Attachments: TEZ-4062.001.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > In RedundantScheduleTransition (inside TaskImpl), we try to find the oldest > running attempt and use it as the causual attempt when doing > "addAndScheduleAttempt". > > However, the task may have completed at this moment, i.e., the task attempt > that was considered running and long-tailed by speculator is now completed. > In this case, we may not be able to find any unfinished attempt, which will > lead to NPE in following logic (even without NPE, it still makes no sense to > proceed with scheduling speculative attempt anyway) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4065) Yetus build fails on trunk due to relying on snapshot dependencies
[
https://issues.apache.org/jira/browse/TEZ-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825267#comment-16825267
]
Jonathan Eagles commented on TEZ-4065:
--
Another consideration is that this only seems intermittent as TEZ-1348
correctly does a top level build. And I'm no expert on Yetus, but it seems like
TEZ-4062 is trying to do a top-level build, but incorrectly picking the top
level as tez/tez-dag.
https://builds.apache.org/job/PreCommit-TEZ-Build/140/consoleText
> Yetus build fails on trunk due to relying on snapshot dependencies
> --
>
> Key: TEZ-4065
> URL: https://issues.apache.org/jira/browse/TEZ-4065
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Priority: Major
>
> As noted in TEZ-4062, Yetus tez builds for provided patch. Very first maven
> build step fails.
> https://builds.apache.org/job/PreCommit-TEZ-Build/139/consoleText
> {code}
> cd /testptch/tez/tez-dag
> /usr/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus-m2/tez-master-patch-0
> -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dcheckstyle.skip=true -Dfindbugs.skip=true >
> /testptch/patchprocess/branch-mvninstall-tez-dag.txt 2>&1
> Elapsed: 3m 19s
> tez-dag in master failed.
> {code}
> The cause is because there is no top level master install step. Instead, it
> tries to download Tez snapshot dependencies which are out of date.
> How do I convince Yetus to do a top level build like PreCommit-YARN?
> Looking at a similar build in YARN first build step installs at the top level.
> https://builds.apache.org/job/PreCommit-YARN-Build/24016/consoleText
> {code}
> cd /testptch/hadoop
> /usr/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/yetus-m2/hadoop-trunk-patch-1
> -Ptest-patch -DskipTests -fae clean install -DskipTests=true
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true >
> /testptch/patchprocess/branch-mvninstall-root.txt 2>&1
> Elapsed: 18m 20s
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4065) Yetus build fails on trunk due to relying on snapshot dependencies
[
https://issues.apache.org/jira/browse/TEZ-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825253#comment-16825253
]
Jonathan Eagles commented on TEZ-4065:
--
[~aw], I'm having an issue where a Yetus 0.8 patch build does not do a top
level mvn install step and submodules fail back to SNAPSHOT dependencies for
dependencies provided within the project. I see that the YARN precommit build
does do top level install. Is there a step missing? Corner case bug? Some
customization needed to get this to work?
> Yetus build fails on trunk due to relying on snapshot dependencies
> --
>
> Key: TEZ-4065
> URL: https://issues.apache.org/jira/browse/TEZ-4065
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Priority: Major
>
> As noted in TEZ-4062, Yetus tez builds for provided patch. Very first maven
> build step fails.
> https://builds.apache.org/job/PreCommit-TEZ-Build/139/consoleText
> {code}
> cd /testptch/tez/tez-dag
> /usr/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus-m2/tez-master-patch-0
> -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dcheckstyle.skip=true -Dfindbugs.skip=true >
> /testptch/patchprocess/branch-mvninstall-tez-dag.txt 2>&1
> Elapsed: 3m 19s
> tez-dag in master failed.
> {code}
> The cause is because there is no top level master install step. Instead, it
> tries to download Tez snapshot dependencies which are out of date.
> How do I convince Yetus to do a top level build like PreCommit-YARN?
> Looking at a similar build in YARN first build step installs at the top level.
> https://builds.apache.org/job/PreCommit-YARN-Build/24016/consoleText
> {code}
> cd /testptch/hadoop
> /usr/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/yetus-m2/hadoop-trunk-patch-1
> -Ptest-patch -DskipTests -fae clean install -DskipTests=true
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true >
> /testptch/patchprocess/branch-mvninstall-root.txt 2>&1
> Elapsed: 18m 20s
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4065) Yetus build fails on trunk due to relying on snapshot dependencies
[
https://issues.apache.org/jira/browse/TEZ-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4065:
-
Description:
As noted in TEZ-4062, Yetus tez builds for provided patch. Very first maven
build step fails.
https://builds.apache.org/job/PreCommit-TEZ-Build/139/consoleText
{code}
cd /testptch/tez/tez-dag
/usr/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus-m2/tez-master-patch-0
-fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true -Dfindbugs.skip=true >
/testptch/patchprocess/branch-mvninstall-tez-dag.txt 2>&1
Elapsed: 3m 19s
tez-dag in master failed.
{code}
The cause is because there is no top level master install step. Instead, it
tries to download Tez snapshot dependencies which are out of date.
How do I convince Yetus to do a top level build like PreCommit-YARN?
Looking at a similar build in YARN first build step installs at the top level.
https://builds.apache.org/job/PreCommit-YARN-Build/24016/consoleText
{code}
cd /testptch/hadoop
/usr/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/yetus-m2/hadoop-trunk-patch-1
-Ptest-patch -DskipTests -fae clean install -DskipTests=true
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true >
/testptch/patchprocess/branch-mvninstall-root.txt 2>&1
Elapsed: 18m 20s
{code}
was:
As noted in TEZ-4062, Yetus tez builds for provided patch. Very first maven
build step fails.
{code}
cd /testptch/tez/tez-dag
/usr/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus-m2/tez-master-patch-0
-fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true -Dfindbugs.skip=true >
/testptch/patchprocess/branch-mvninstall-tez-dag.txt 2>&1
Elapsed: 3m 19s
tez-dag in master failed.
{code}
The cause is because there is no top level master install step. Instead, it
tries to download Tez snapshot dependencies which are out of date.
How do I convince Yetus to do a top level build like PreCommit-YARN?
Looking at a similar build in YARN first build step installs at the top level.
https://builds.apache.org/job/PreCommit-YARN-Build/24016/consoleText
{code}
cd /testptch/hadoop
/usr/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/yetus-m2/hadoop-trunk-patch-1
-Ptest-patch -DskipTests -fae clean install -DskipTests=true
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true >
/testptch/patchprocess/branch-mvninstall-root.txt 2>&1
Elapsed: 18m 20s
{code}
> Yetus build fails on trunk due to relying on snapshot dependencies
> --
>
> Key: TEZ-4065
> URL: https://issues.apache.org/jira/browse/TEZ-4065
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Priority: Major
>
> As noted in TEZ-4062, Yetus tez builds for provided patch. Very first maven
> build step fails.
> https://builds.apache.org/job/PreCommit-TEZ-Build/139/consoleText
> {code}
> cd /testptch/tez/tez-dag
> /usr/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus-m2/tez-master-patch-0
> -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dcheckstyle.skip=true -Dfindbugs.skip=true >
> /testptch/patchprocess/branch-mvninstall-tez-dag.txt 2>&1
> Elapsed: 3m 19s
> tez-dag in master failed.
> {code}
> The cause is because there is no top level master install step. Instead, it
> tries to download Tez snapshot dependencies which are out of date.
> How do I convince Yetus to do a top level build like PreCommit-YARN?
> Looking at a similar build in YARN first build step installs at the top level.
> https://builds.apache.org/job/PreCommit-YARN-Build/24016/consoleText
> {code}
> cd /testptch/hadoop
> /usr/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/yetus-m2/hadoop-trunk-patch-1
> -Ptest-patch -DskipTests -fae clean install -DskipTests=true
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true >
> /testptch/patchprocess/branch-mvninstall-root.txt 2>&1
> Elapsed: 18m 20s
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4065) Yetus build fails on trunk due to relying on snapshot dependencies
Jonathan Eagles created TEZ-4065:
Summary: Yetus build fails on trunk due to relying on snapshot
dependencies
Key: TEZ-4065
URL: https://issues.apache.org/jira/browse/TEZ-4065
Project: Apache Tez
Issue Type: Bug
Reporter: Jonathan Eagles
As noted in TEZ-4062, Yetus tez builds for provided patch. Very first maven
build step fails.
{code}
cd /testptch/tez/tez-dag
/usr/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus-m2/tez-master-patch-0
-fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
-Dcheckstyle.skip=true -Dfindbugs.skip=true >
/testptch/patchprocess/branch-mvninstall-tez-dag.txt 2>&1
Elapsed: 3m 19s
tez-dag in master failed.
{code}
The cause is because there is no top level master install step. Instead, it
tries to download Tez snapshot dependencies which are out of date.
How do I convince Yetus to do a top level build like PreCommit-YARN?
Looking at a similar build in YARN first build step installs at the top level.
https://builds.apache.org/job/PreCommit-YARN-Build/24016/consoleText
{code}
cd /testptch/hadoop
/usr/bin/mvn --batch-mode
-Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/yetus-m2/hadoop-trunk-patch-1
-Ptest-patch -DskipTests -fae clean install -DskipTests=true
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true >
/testptch/patchprocess/branch-mvninstall-root.txt 2>&1
Elapsed: 18m 20s
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4062) Speculative attempt scheduling should be aborted when Task has completed
[ https://issues.apache.org/jira/browse/TEZ-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4062: - Summary: Speculative attempt scheduling should be aborted when Task has completed (was: Speculative attempt scheduling should be aborted when Task has complelted) > Speculative attempt scheduling should be aborted when Task has completed > > > Key: TEZ-4062 > URL: https://issues.apache.org/jira/browse/TEZ-4062 > Project: Apache Tez > Issue Type: Bug >Reporter: Yingda Chen >Assignee: Ying Han >Priority: Major > Attachments: TEZ-4062.001.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > In RedundantScheduleTransition (inside TaskImpl), we try to find the oldest > running attempt and use it as the causual attempt when doing > "addAndScheduleAttempt". > > However, the task may have completed at this moment, i.e., the task attempt > that was considered running and long-tailed by speculator is now completed. > In this case, we may not be able to find any unfinished attempt, which will > lead to NPE in following logic (even without NPE, it still makes no sense to > proceed with scheduling speculative attempt anyway) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4062) Speculative attempt scheduling should be aborted when Task has complelted
[ https://issues.apache.org/jira/browse/TEZ-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823537#comment-16823537 ] Jonathan Eagles commented on TEZ-4062: -- [~Chyler], I analyzed why this failure was happening to help get the TezQA results for this patch. Recently the switch from git to gitbox changed the code repository. The job that generates Tez Snapshot builds stopped working at the time of gitbox migration. This caused the code to be compiled against older Tez jars. I have tried to fix the snapshot build and will try TezQA for this patch tonight. > Speculative attempt scheduling should be aborted when Task has complelted > - > > Key: TEZ-4062 > URL: https://issues.apache.org/jira/browse/TEZ-4062 > Project: Apache Tez > Issue Type: Bug >Reporter: Yingda Chen >Assignee: Ying Han >Priority: Major > Attachments: TEZ-4062.001.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > In RedundantScheduleTransition (inside TaskImpl), we try to find the oldest > running attempt and use it as the causual attempt when doing > "addAndScheduleAttempt". > > However, the task may have completed at this moment, i.e., the task attempt > that was considered running and long-tailed by speculator is now completed. > In this case, we may not be able to find any unfinished attempt, which will > lead to NPE in following logic (even without NPE, it still makes no sense to > proceed with scheduling speculative attempt anyway) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4062) Speculative attempt scheduling should be aborted when Task has complelted
[ https://issues.apache.org/jira/browse/TEZ-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4062: - Attachment: TEZ-4062.001.patch > Speculative attempt scheduling should be aborted when Task has complelted > - > > Key: TEZ-4062 > URL: https://issues.apache.org/jira/browse/TEZ-4062 > Project: Apache Tez > Issue Type: Bug >Reporter: Yingda Chen >Assignee: Ying Han >Priority: Major > Attachments: TEZ-4062.001.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > In RedundantScheduleTransition (inside TaskImpl), we try to find the oldest > running attempt and use it as the causual attempt when doing > "addAndScheduleAttempt". > > However, the task may have completed at this moment, i.e., the task attempt > that was considered running and long-tailed by speculator is now completed. > In this case, we may not be able to find any unfinished attempt, which will > lead to NPE in following logic (even without NPE, it still makes no sense to > proceed with scheduling speculative attempt anyway) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4064) Integrate Tez with Github
[ https://issues.apache.org/jira/browse/TEZ-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823221#comment-16823221 ] Jonathan Eagles commented on TEZ-4064: -- [~ste...@apache.org], [~aw], I would like to add github PR pre-commit build for Tez. Currently Tez uses Yetus 0.8.0 to build. Found HADOOP-16035, but not sure I fully grasp the steps. Yetus documentation https://yetus.apache.org/documentation/0.9.0/precommit-robots/ points to some capabilities that seem relevant here. Can either of you shed some light on the Yetus github pre-commit integration process? > Integrate Tez with Github > - > > Key: TEZ-4064 > URL: https://issues.apache.org/jira/browse/TEZ-4064 > Project: Apache Tez > Issue Type: New Feature >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > > According to HADOOP-16035, steps are as follows. > - an account that can read Github > - Apache Yetus 0.9.0+ > - a Jenkinsfile that uses the above -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4064) Integrate Tez with Github
Jonathan Eagles created TEZ-4064: Summary: Integrate Tez with Github Key: TEZ-4064 URL: https://issues.apache.org/jira/browse/TEZ-4064 Project: Apache Tez Issue Type: New Feature Reporter: Jonathan Eagles Assignee: Jonathan Eagles According to HADOOP-16035, steps are as follows. - an account that can read Github - Apache Yetus 0.9.0+ - a Jenkinsfile that uses the above -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-1348) Setup configs required for local mode automatically, instead of relying on changes to tez-site
[ https://issues.apache.org/jira/browse/TEZ-1348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810344#comment-16810344 ] Jonathan Eagles commented on TEZ-1348: -- [~tlipcon], Tez doesn't have github integration. Do you mind putting up the patch here for Tez QA report? > Setup configs required for local mode automatically, instead of relying on > changes to tez-site > -- > > Key: TEZ-1348 > URL: https://issues.apache.org/jira/browse/TEZ-1348 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Todd Lipcon >Priority: Critical > Time Spent: 20m > Remaining Estimate: 0h > > In TEZ-717, I incorrect thought setting fs.defaultFS programmatically in > tez-site would work for local mode. > Currently the requirement is that tez-site.xml must have fs.defaultFS set to > file:///. > While that works, it doesn't allow for seamless execution in either > local-mode or on a cluster. > The main issue here is that when Inputs / Outputs are configured - they use a > version of configuration which reads tez-site, and do not use the > configuration from the client itself (which is correct behaviour). > Not sure what a good way to fix this is > 1) It may be possible to override this value each time an instance of > Configuration/TezConfiguration is created. One possible way would be to > statically add a default resource to Configuration the moment a local client > is created. > 2) Provide information in the contexts on whether this is local or not. This > is fairly ugly, and would get in the way of running mixed mode tasks. > Anyone have other suggestions ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4058) Changes for 0.9.2 release
[ https://issues.apache.org/jira/browse/TEZ-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805570#comment-16805570 ] Jonathan Eagles commented on TEZ-4058: -- +1 > Changes for 0.9.2 release > - > > Key: TEZ-4058 > URL: https://issues.apache.org/jira/browse/TEZ-4058 > Project: Apache Tez > Issue Type: Bug >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-4058.001.patch > > > Update Tez_DOAP.rtf, index.md etc. for 0.9.2 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4057) Fix Unsorted broadcast shuffle umasks
[
https://issues.apache.org/jira/browse/TEZ-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805462#comment-16805462
]
Jonathan Eagles commented on TEZ-4057:
--
[~gopalv], there is a vote currently out for Tez 0.9.2. Probably fixed Version
is 0.9.3?
> Fix Unsorted broadcast shuffle umasks
> -
>
> Key: TEZ-4057
> URL: https://issues.apache.org/jira/browse/TEZ-4057
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.2
>Reporter: Gopal V
>Assignee: Eric Wohlstadter
>Priority: Major
> Fix For: 0.9.2, 0.10.1
>
> Attachments: TEZ-4057.1.patch
>
>
> {code}
> if (numPartitions == 1 && !pipelinedShuffle) {
> //special case, where in only one partition is available.
> finalOutPath = outputFileHandler.getOutputFileForWrite();
> finalIndexPath =
> outputFileHandler.getOutputIndexFileForWrite(indexFileSizeEstimate);
> skipBuffers = true;
> writer = new IFile.Writer(conf, rfs, finalOutPath, keyClass, valClass,
> codec, outputRecordsCounter, outputRecordBytesCounter);
> } else {
> skipBuffers = false;
> writer = null;
> }
> {code}
> The broadcast events don't update the file umasks, because they have 1
> partition.
> {code}
> total 8.0K
> -rw--- 1 hive hadoop 15 Mar 27 20:30 file.out
> -rw-r- 1 hive hadoop 32 Mar 27 20:30 file.out.index
> {code}
> ending up with readable index files and unreadable .out files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4045) Task should be accessible from TaskAttempt
[ https://issues.apache.org/jira/browse/TEZ-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4045: - Attachment: TEZ-4045.001.patch > Task should be accessible from TaskAttempt > -- > > Key: TEZ-4045 > URL: https://issues.apache.org/jira/browse/TEZ-4045 > Project: Apache Tez > Issue Type: Improvement >Reporter: Yingda Chen >Assignee: Ying Han >Priority: Major > Attachments: TEZ-4045.001.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently In the DAG component hierarchy of DAG->Vertex->Task->TaskAttempt, > we usually allows the entity lower in the hierarchy to access its upstream > entity, for example, we have Task.getVertex(), and Vertex.getDAG(). > > However, TaskAttempt today is missing an interface to easily retreat the Task > it belongs to. This can be tricky (it is still doable today, but quite messy > and inefficient) when TaskAttempt is trying to access properties/interfaces > defined by Task. See, for example, the discussion in > [https://github.com/apache/tez/pull/37.] > > This Jira proposes to add an TaskAttempt.getTask() method, and refactor codes > where this new method can help clarify the logic. Ideally, A TaskAttempt > should get Task object passing in as a constructing parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4056) Update version for 0.9.2 release
[
https://issues.apache.org/jira/browse/TEZ-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796245#comment-16796245
]
Jonathan Eagles commented on TEZ-4056:
--
+1. Thanks. \\cc [~kshukla]
> Update version for 0.9.2 release
>
>
> Key: TEZ-4056
> URL: https://issues.apache.org/jira/browse/TEZ-4056
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Major
> Fix For: 0.9.2
>
> Attachments: TEZ-4056.001.patch
>
>
> Tracks the release process sub section:
> {code:java}
> mvn versions:set -DnewVersion="x.y.z"
> {code}{code}
> CC: [~jeagles]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4002) CHANGES.txt for 0.9.2 Release
[ https://issues.apache.org/jira/browse/TEZ-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796212#comment-16796212 ] Jonathan Eagles commented on TEZ-4002: -- +1. Please check this into branch-0.9.2 > CHANGES.txt for 0.9.2 Release > - > > Key: TEZ-4002 > URL: https://issues.apache.org/jira/browse/TEZ-4002 > Project: Apache Tez > Issue Type: Task >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-4002.001.patch, TEZ-4002.002.patch, > TEZ-4002.003.patch > > > Add CHANGES.txt for 0.9.2 line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4002) CHANGES.txt for 0.9.2 Release
[
https://issues.apache.org/jira/browse/TEZ-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795192#comment-16795192
]
Jonathan Eagles commented on TEZ-4002:
--
Thanks for getting this documented more precisely. Please try
{code:java}
git log rel/release-0.9.1..origin/branch-0.9 --format="%s"
{code}
to avoid some of the complexity. Also, this has some assumptions regarding what
origin is tied to. Please clarify the documentation the assumptions regarding
origin if that isn't already done.
> CHANGES.txt for 0.9.2 Release
> -
>
> Key: TEZ-4002
> URL: https://issues.apache.org/jira/browse/TEZ-4002
> Project: Apache Tez
> Issue Type: Task
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
>Priority: Major
> Attachments: TEZ-4002.001.patch, TEZ-4002.002.patch,
> TEZ-4002.003.patch
>
>
> Add CHANGES.txt for 0.9.2 line.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4055) Align slf4j version to be same as Hadoop3
[
https://issues.apache.org/jira/browse/TEZ-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795155#comment-16795155
]
Jonathan Eagles commented on TEZ-4055:
--
[~shameer], It is true that Hadoop has moved from 1.7.10 to 1.7.25 as part of
HADOOP-14290. According to the jira, was not only part of hadoop 3+, but also
2.9.0. I agree that it makes sense for this to align with hadoop. I am going
investigate missing logs to see the origin of the cause. I have run Tez on top
of Hadoop3+ (though not through HS2), but have never seen missing logs. If you
have any additional thoughts on why the missing logs and as to whether your
attached patch fixes the issue that will be great to know.
> Align slf4j version to be same as Hadoop3
> -
>
> Key: TEZ-4055
> URL: https://issues.apache.org/jira/browse/TEZ-4055
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.0, 0.9.1
>Reporter: Syed Shameerur Rahman
>Priority: Major
> Fix For: 0.10.0
>
> Attachments: TEZ-4055.patch
>
>
> Currently tez is using slf4j version 1.7.10 where as hadoop3 is using slf4j
> version 1.7.25 , due to conflicting version of slf4j version no logs are
> getting written in ${sys:test.tmp.dir}/log/hive.log. The statements
> LOG.info("") , LOG.error("") . are directed to write logs into
> ${sys:test.tmp.dir}/log/hive.log.
> When we connect to HS2 with beeline the slf4j jars are being searched
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:]
> SLF4J: Found binding in [jar:]
> SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an
> explanation.
> SLF4J: Actual binding is of type
> [org.apache.logging.slf4j.Log4jLoggerFactory]
> cc [~hitesh] [~kshukla]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4044) Zookeeper: exclude jline from Zookeeper client from tez dist
[
https://issues.apache.org/jira/browse/TEZ-4044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789892#comment-16789892
]
Jonathan Eagles commented on TEZ-4044:
--
I wonder how this is failing for you. Are you shipping all Tez dependencies as
part of Tez installation? It's a simple change, but it seems more natural to me
that the version of zookeeper would be provided by the environment and not Tez
at all.
> Zookeeper: exclude jline from Zookeeper client from tez dist
>
>
> Key: TEZ-4044
> URL: https://issues.apache.org/jira/browse/TEZ-4044
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.10.0
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: TEZ-4044.1.patch
>
>
> {code}
> [INFO] | +- org.apache.zookeeper:zookeeper:jar:3.4.9:compile
> [INFO] | | \- jline:jline:jar:0.9.94:compile
> {code}
> Breaks CLI clients further down the dependency tree.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4052) Fit dot files ASF License issues - part 2
[
https://issues.apache.org/jira/browse/TEZ-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4052:
-
Attachment: TEZ-4052.001.patch
> Fit dot files ASF License issues - part 2
> -
>
> Key: TEZ-4052
> URL: https://issues.apache.org/jira/browse/TEZ-4052
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4052.001.patch
>
>
> Continuing the effort in TEZ-3995.
> https://issues.apache.org/jira/browse/TEZ-3995?focusedCommentId=16784595=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16784595
> {code}
> 1) Please extend this to tez-ext-service-tests 2) Also, please consider
> directory tez.log.dir with path ${project.build.directory}/logs.
> {code}
> This jira is to making sure all dot files are correctly placed under target
> directory as to 1) make sure file aren't created outside the build directory
> and 2) and named as part of a broader test directory design
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4052) Fit dot files ASF License issues - part 2
[
https://issues.apache.org/jira/browse/TEZ-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788260#comment-16788260
]
Jonathan Eagles commented on TEZ-4052:
--
By looking at TEZ-3664 PreCommit run, here are the dot files still left to
place under target directory.
{code}
!?
/testptch/tez/tez-ext-service-tests/dag_1552000666587_0001_2_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552003561122_0001_1_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552000666587_0001_5_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552000666587_0001_3_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552003561122_0001_2_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552003561122_0001_5_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552000666587_0001_1_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552003561122_0001_3_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552000666587_0001_4_priority.dot
!?
/testptch/tez/tez-ext-service-tests/dag_1552003561122_0001_4_priority.dot
{code}
> Fit dot files ASF License issues - part 2
> -
>
> Key: TEZ-4052
> URL: https://issues.apache.org/jira/browse/TEZ-4052
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
>
> Continuing the effort in TEZ-3995.
> https://issues.apache.org/jira/browse/TEZ-3995?focusedCommentId=16784595=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16784595
> {code}
> 1) Please extend this to tez-ext-service-tests 2) Also, please consider
> directory tez.log.dir with path ${project.build.directory}/logs.
> {code}
> This jira is to making sure all dot files are correctly placed under target
> directory as to 1) make sure file aren't created outside the build directory
> and 2) and named as part of a broader test directory design
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4052) Fit dot files ASF License issues - part 2
Jonathan Eagles created TEZ-4052: Summary: Fit dot files ASF License issues - part 2 Key: TEZ-4052 URL: https://issues.apache.org/jira/browse/TEZ-4052 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles Continuing the effort in TEZ-3995. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4052) Fit dot files ASF License issues - part 2
[
https://issues.apache.org/jira/browse/TEZ-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4052:
-
Description:
Continuing the effort in TEZ-3995.
https://issues.apache.org/jira/browse/TEZ-3995?focusedCommentId=16784595=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16784595
{code}
1) Please extend this to tez-ext-service-tests 2) Also, please consider
directory tez.log.dir with path ${project.build.directory}/logs.
{code}
This jira is to making sure all dot files are correctly placed under target
directory as to 1) make sure file aren't created outside the build directory
and 2) and named as part of a broader test directory design
was:Continuing the effort in TEZ-3995.
> Fit dot files ASF License issues - part 2
> -
>
> Key: TEZ-4052
> URL: https://issues.apache.org/jira/browse/TEZ-4052
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
>
> Continuing the effort in TEZ-3995.
> https://issues.apache.org/jira/browse/TEZ-3995?focusedCommentId=16784595=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16784595
> {code}
> 1) Please extend this to tez-ext-service-tests 2) Also, please consider
> directory tez.log.dir with path ${project.build.directory}/logs.
> {code}
> This jira is to making sure all dot files are correctly placed under target
> directory as to 1) make sure file aren't created outside the build directory
> and 2) and named as part of a broader test directory design
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4052) Fit dot files ASF License issues - part 2
[ https://issues.apache.org/jira/browse/TEZ-4052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4052: - Issue Type: Sub-task (was: Bug) Parent: TEZ-3891 > Fit dot files ASF License issues - part 2 > - > > Key: TEZ-4052 > URL: https://issues.apache.org/jira/browse/TEZ-4052 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > > Continuing the effort in TEZ-3995. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3995) Fix dot files produced by tests to prevent ASF license warnings in yetus
[
https://issues.apache.org/jira/browse/TEZ-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16788085#comment-16788085
]
Jonathan Eagles commented on TEZ-3995:
--
+1. As a path to going forward, I will commit this addendum patch as is. And
create a separate JIRA to address my comments above.
> Fix dot files produced by tests to prevent ASF license warnings in yetus
>
>
> Key: TEZ-3995
> URL: https://issues.apache.org/jira/browse/TEZ-3995
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Eagles
>Assignee: Jaume M
>Priority: Major
> Fix For: 0.9.2, 0.10.0
>
> Attachments: TEZ-3995.1-branch-0.9.patch, TEZ-3995.1.patch,
> TEZ-3995.1.patch, TEZ-3995.2.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-TEZ-Build-Yetus/10/artifact/out/patch-asflicense-problems.txt
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name1/current/seen_txid
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name1/current/fsimage_000.md5
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name1/current/fsimage_000
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name1/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name2/current/seen_txid
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name2/current/fsimage_000.md5
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name2/current/fsimage_000
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name2/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/BP-1822420483-67.195.81.145-1538067940282/scanner.cursor
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/BP-1822420483-67.195.81.145-1538067940282/current/dfsUsed
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/BP-1822420483-67.195.81.145-1538067940282/current/finalized/subdir0/subdir0/blk_1073741827
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/BP-1822420483-67.195.81.145-1538067940282/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/BP-1822420483-67.195.81.145-1538067940282/scanner.cursor
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/BP-1822420483-67.195.81.145-1538067940282/current/dfsUsed
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/BP-1822420483-67.195.81.145-1538067940282/current/finalized/subdir0/subdir0/blk_1073741826
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/BP-1822420483-67.195.81.145-1538067940282/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data10/current/BP-1822420483-67.195.81.145-1538067940282/scanner.cursor
> !?
>
[jira] [Updated] (TEZ-3664) testSingleSpill_WithPipelinedShuffle[0] fails intermittently
[
https://issues.apache.org/jira/browse/TEZ-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3664:
-
Attachment: TEZ-3664.004.patch
> testSingleSpill_WithPipelinedShuffle[0] fails intermittently
>
>
> Key: TEZ-3664
> URL: https://issues.apache.org/jira/browse/TEZ-3664
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3664.001.patch, TEZ-3664.002.patch,
> TEZ-3664.003.patch, TEZ-3664.004.patch
>
>
> testSingleSpill_WithPipelinedShuffle[0] unit test fails intermittently
> {code}
> Stacktrace
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTestWithPipelinedTransfer(TestUnorderedPartitionedKVWriter.java:503)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSingleSpill_WithPipelinedShuffle(TestUnorderedPartitionedKVWriter.java:414)
> Standard Output
> 2017-03-20 19:50:52,538 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:setup(124)) - Setup. Using test dir:
> /tmp/TestUnorderedPartitionedKVWriter
> 2017-03-20 19:50:53,065 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2017-03-20 19:50:53,856 INFO [Thread-5] counters.Limits
> (Limits.java:ensureInitialized(60)) - Counter limits initialized with
> parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64,
> MAX_COUNTERS=1200
> 2017-03-20 19:50:53,861 INFO [Thread-5]
> writers.BaseUnorderedPartitionedKVWriter
> (BaseUnorderedPartitionedKVWriter.java:(148)) - Instantiating
> Partitioner:
> [org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
> 2017-03-20 19:50:53,870 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:(221)) - destinationVertexName:
> numBuffers=2, sizePerBuffer=1024, skipBuffers=false, pipelinedShuffle=true,
> numPartitions=10
> 2017-03-20 19:50:53,871 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:setupNextBuffer(333)) -
> destinationVertexName: Moving to next buffer and triggering spill
> 2017-03-20 19:50:53,913 INFO [UnorderedOutSpiller {destinationVertexName}]
> writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 0
> 2017-03-20 19:50:53,934 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=false),
> spillId=0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(473)) - destinationVertexName:
> Waiting for all spills to complete : Pending : 0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(492)) - destinationVertexName: All
> spills complete
> 2017-03-20 19:50:53,948 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 1
> 2017-03-20 19:50:53,959 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=true),
> spillId=1
> 2017-03-20 19:50:53,972 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:cleanup(132)) - CleanUp{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3664) testSingleSpill_WithPipelinedShuffle[0] fails intermittently
[
https://issues.apache.org/jira/browse/TEZ-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3664:
-
Attachment: TEZ-3664.002.patch
> testSingleSpill_WithPipelinedShuffle[0] fails intermittently
>
>
> Key: TEZ-3664
> URL: https://issues.apache.org/jira/browse/TEZ-3664
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3664.001.patch, TEZ-3664.002.patch
>
>
> testSingleSpill_WithPipelinedShuffle[0] unit test fails intermittently
> {code}
> Stacktrace
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTestWithPipelinedTransfer(TestUnorderedPartitionedKVWriter.java:503)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSingleSpill_WithPipelinedShuffle(TestUnorderedPartitionedKVWriter.java:414)
> Standard Output
> 2017-03-20 19:50:52,538 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:setup(124)) - Setup. Using test dir:
> /tmp/TestUnorderedPartitionedKVWriter
> 2017-03-20 19:50:53,065 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2017-03-20 19:50:53,856 INFO [Thread-5] counters.Limits
> (Limits.java:ensureInitialized(60)) - Counter limits initialized with
> parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64,
> MAX_COUNTERS=1200
> 2017-03-20 19:50:53,861 INFO [Thread-5]
> writers.BaseUnorderedPartitionedKVWriter
> (BaseUnorderedPartitionedKVWriter.java:(148)) - Instantiating
> Partitioner:
> [org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
> 2017-03-20 19:50:53,870 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:(221)) - destinationVertexName:
> numBuffers=2, sizePerBuffer=1024, skipBuffers=false, pipelinedShuffle=true,
> numPartitions=10
> 2017-03-20 19:50:53,871 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:setupNextBuffer(333)) -
> destinationVertexName: Moving to next buffer and triggering spill
> 2017-03-20 19:50:53,913 INFO [UnorderedOutSpiller {destinationVertexName}]
> writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 0
> 2017-03-20 19:50:53,934 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=false),
> spillId=0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(473)) - destinationVertexName:
> Waiting for all spills to complete : Pending : 0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(492)) - destinationVertexName: All
> spills complete
> 2017-03-20 19:50:53,948 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 1
> 2017-03-20 19:50:53,959 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=true),
> spillId=1
> 2017-03-20 19:50:53,972 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:cleanup(132)) - CleanUp{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TEZ-4046) insert overwrite table without data
[
https://issues.apache.org/jira/browse/TEZ-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles resolved TEZ-4046.
--
Resolution: Cannot Reproduce
[~jipeng], closing this issue. Please direct questions as directed to the
proper user lists.
> insert overwrite table without data
> ---
>
> Key: TEZ-4046
> URL: https://issues.apache.org/jira/browse/TEZ-4046
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: zengjipeng
>Priority: Major
>
> use hive on tez.
> execute as below statement, the target table tmp.zjp_b have no data.
>
> {code:java}
> create table tmp.zjp_a(a string,b string);
> insert into tmp.zjp_a select "a1","1,2,3";
> create table tmp.zjp_b(name string,age string) partitioned by(dt string);
> insert overwrite table tmp.zjp_b partition(dt='2019-02-26')
> select a,age from tmp.zjp_a lateral view explode(split(b,",")) tb as age
> where age='1'
> union all
> select a,age from tmp.zjp_a lateral view explode(split(b,",")) tb as age
> where age='2';
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (TEZ-3664) testSingleSpill_WithPipelinedShuffle[0] fails intermittently
[
https://issues.apache.org/jira/browse/TEZ-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles reassigned TEZ-3664:
Assignee: Jonathan Eagles
001 patch address the lack of the tests relying on test.build.data and falling
back to /tmp/ if not specified. This will handle a majority of test cases
explicitly writing to tmp directory that could be subject to tmp dir cleaner.
> testSingleSpill_WithPipelinedShuffle[0] fails intermittently
>
>
> Key: TEZ-3664
> URL: https://issues.apache.org/jira/browse/TEZ-3664
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Yesha Vora
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3664.001.patch
>
>
> testSingleSpill_WithPipelinedShuffle[0] unit test fails intermittently
> {code}
> Stacktrace
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTestWithPipelinedTransfer(TestUnorderedPartitionedKVWriter.java:503)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSingleSpill_WithPipelinedShuffle(TestUnorderedPartitionedKVWriter.java:414)
> Standard Output
> 2017-03-20 19:50:52,538 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:setup(124)) - Setup. Using test dir:
> /tmp/TestUnorderedPartitionedKVWriter
> 2017-03-20 19:50:53,065 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2017-03-20 19:50:53,856 INFO [Thread-5] counters.Limits
> (Limits.java:ensureInitialized(60)) - Counter limits initialized with
> parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64,
> MAX_COUNTERS=1200
> 2017-03-20 19:50:53,861 INFO [Thread-5]
> writers.BaseUnorderedPartitionedKVWriter
> (BaseUnorderedPartitionedKVWriter.java:(148)) - Instantiating
> Partitioner:
> [org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
> 2017-03-20 19:50:53,870 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:(221)) - destinationVertexName:
> numBuffers=2, sizePerBuffer=1024, skipBuffers=false, pipelinedShuffle=true,
> numPartitions=10
> 2017-03-20 19:50:53,871 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:setupNextBuffer(333)) -
> destinationVertexName: Moving to next buffer and triggering spill
> 2017-03-20 19:50:53,913 INFO [UnorderedOutSpiller {destinationVertexName}]
> writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 0
> 2017-03-20 19:50:53,934 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=false),
> spillId=0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(473)) - destinationVertexName:
> Waiting for all spills to complete : Pending : 0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(492)) - destinationVertexName: All
> spills complete
> 2017-03-20 19:50:53,948 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 1
> 2017-03-20 19:50:53,959 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=true),
> spillId=1
> 2017-03-20 19:50:53,972 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:cleanup(132)) - CleanUp{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3664) testSingleSpill_WithPipelinedShuffle[0] fails intermittently
[
https://issues.apache.org/jira/browse/TEZ-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3664:
-
Attachment: TEZ-3664.001.patch
> testSingleSpill_WithPipelinedShuffle[0] fails intermittently
>
>
> Key: TEZ-3664
> URL: https://issues.apache.org/jira/browse/TEZ-3664
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Yesha Vora
>Priority: Major
> Attachments: TEZ-3664.001.patch
>
>
> testSingleSpill_WithPipelinedShuffle[0] unit test fails intermittently
> {code}
> Stacktrace
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTestWithPipelinedTransfer(TestUnorderedPartitionedKVWriter.java:503)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSingleSpill_WithPipelinedShuffle(TestUnorderedPartitionedKVWriter.java:414)
> Standard Output
> 2017-03-20 19:50:52,538 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:setup(124)) - Setup. Using test dir:
> /tmp/TestUnorderedPartitionedKVWriter
> 2017-03-20 19:50:53,065 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2017-03-20 19:50:53,856 INFO [Thread-5] counters.Limits
> (Limits.java:ensureInitialized(60)) - Counter limits initialized with
> parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64,
> MAX_COUNTERS=1200
> 2017-03-20 19:50:53,861 INFO [Thread-5]
> writers.BaseUnorderedPartitionedKVWriter
> (BaseUnorderedPartitionedKVWriter.java:(148)) - Instantiating
> Partitioner:
> [org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
> 2017-03-20 19:50:53,870 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:(221)) - destinationVertexName:
> numBuffers=2, sizePerBuffer=1024, skipBuffers=false, pipelinedShuffle=true,
> numPartitions=10
> 2017-03-20 19:50:53,871 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:setupNextBuffer(333)) -
> destinationVertexName: Moving to next buffer and triggering spill
> 2017-03-20 19:50:53,913 INFO [UnorderedOutSpiller {destinationVertexName}]
> writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 0
> 2017-03-20 19:50:53,934 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=false),
> spillId=0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(473)) - destinationVertexName:
> Waiting for all spills to complete : Pending : 0
> 2017-03-20 19:50:53,938 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:close(492)) - destinationVertexName: All
> spills complete
> 2017-03-20 19:50:53,948 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:callInternal(432)) -
> destinationVertexName: Finished spill 1
> 2017-03-20 19:50:53,959 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) -
> destinationVertexName: Adding spill event for spill (final update=true),
> spillId=1
> 2017-03-20 19:50:53,972 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:cleanup(132)) - CleanUp{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4039) Tez should inject dag id, query id into MDC
[
https://issues.apache.org/jira/browse/TEZ-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784618#comment-16784618
]
Jonathan Eagles commented on TEZ-4039:
--
[~prasanth_j], I can quickly see how usefully this will be. Directly, I see how
dagId is relevant, but queryId has no meaning in tez. What does that represent
and does that have to be done explicitly, or is setting MDC based on NDC enough
(hive sets NDC and then Tez copied NDC to MDC)? Also, is log4j2 required for
this functionality?
> Tez should inject dag id, query id into MDC
> ---
>
> Key: TEZ-4039
> URL: https://issues.apache.org/jira/browse/TEZ-4039
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.next
>Reporter: Prasanth Jayachandran
>Priority: Major
>
> Tez currently uses CallableWithNdc to store thread specific context. It
> should also inject the context into MDC so that pattern layout can dump the
> contexts from MDC (with NDC it is not possible to read the context in pattern
> lyaout).
> Hive for example, sets queryId in the MDC and pattern layout prints the
> queryId
>
> {code:java}
> %d{ISO8601} %-5p [%t (%X{queryId})] %c{2}: %m%n
> {code}
> Llap sets dagId, fragmentId and queryId into MDC which is used for queryId
> based routing of logging.
> Similarly, Tez AM should set dagId and queryId (if available) into MDC.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3995) Fix dot files produced by tests to prevent ASF license warnings in yetus
[
https://issues.apache.org/jira/browse/TEZ-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784595#comment-16784595
]
Jonathan Eagles commented on TEZ-3995:
--
[~jmarhuen], just getting back to this patch. Couple of small things. 1) Please
extend this to tez-ext-service-tests 2) Also, please consider directory
tez.log.dir with path ${project.build.directory}/logs.
> Fix dot files produced by tests to prevent ASF license warnings in yetus
>
>
> Key: TEZ-3995
> URL: https://issues.apache.org/jira/browse/TEZ-3995
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Eagles
>Assignee: Jaume M
>Priority: Major
> Fix For: 0.9.2, 0.10.0
>
> Attachments: TEZ-3995.1-branch-0.9.patch, TEZ-3995.1.patch,
> TEZ-3995.1.patch, TEZ-3995.2.patch
>
>
> From
> https://builds.apache.org/job/PreCommit-TEZ-Build-Yetus/10/artifact/out/patch-asflicense-problems.txt
> {code}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name1/current/seen_txid
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name1/current/fsimage_000.md5
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name1/current/fsimage_000
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name1/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name2/current/seen_txid
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name2/current/fsimage_000.md5
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name2/current/fsimage_000
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/name2/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/BP-1822420483-67.195.81.145-1538067940282/scanner.cursor
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/BP-1822420483-67.195.81.145-1538067940282/current/dfsUsed
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/BP-1822420483-67.195.81.145-1538067940282/current/finalized/subdir0/subdir0/blk_1073741827
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/BP-1822420483-67.195.81.145-1538067940282/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data4/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/BP-1822420483-67.195.81.145-1538067940282/scanner.cursor
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/BP-1822420483-67.195.81.145-1538067940282/current/dfsUsed
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/BP-1822420483-67.195.81.145-1538067940282/current/finalized/subdir0/subdir0/blk_1073741826
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/BP-1822420483-67.195.81.145-1538067940282/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data9/current/VERSION
> !?
> /home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build-Yetus/sourcedir/tez-plugins/tez-aux-services/build/test/data/dfs/data/data10/current/BP-1822420483-67.195.81.145-1538067940282/scanner.cursor
> !?
>
[jira] [Updated] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4028: - Fix Version/s: 0.9.2 > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Labels: history > Fix For: 0.9.2, 0.10.1 > > Attachments: TEZ-4028.01.patch, TEZ-4028.02.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783801#comment-16783801 ] Jonathan Eagles commented on TEZ-4028: -- Cherry-picked this change to branch-0.9 line as well. > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Labels: history > Fix For: 0.9.2, 0.10.1 > > Attachments: TEZ-4028.01.patch, TEZ-4028.02.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4031) Support tez gitbox migration
[
https://issues.apache.org/jira/browse/TEZ-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782183#comment-16782183
]
Jonathan Eagles commented on TEZ-4031:
--
[~kshukla], latest patch is ready for review on gitbox changes. Rolled back
maven-site-plugin version change and switched maven-info-project-reports-plugin
to the last version before the new format 2.9
> Support tez gitbox migration
>
>
> Key: TEZ-4031
> URL: https://issues.apache.org/jira/browse/TEZ-4031
> Project: Apache Tez
> Issue Type: Task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4031.001.patch, TEZ-4031.002.patch,
> TEZ-4031.003.patch, TEZ-4031.004.patch
>
>
> {code}
> $ git grep git-wip
> Tez_DOAP.rdf: rdf:resource="https://git-wip-us.apache.org/repos/asf/tez.git"/>
> Tez_DOAP.rdf: rdf:resource="https://git-wip-us.apache.org/repos/asf?p=tez.git"/>
> docs/src/site/site.xml: href="https://git-wip-us.apache.org/repos/asf/tez.git; alt="Use git clone
> https://git-wip-us.apache.org/repos/asf/tez.git; />
> pom.xml:
> scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-api/src/test/java/org/apache/tez/common/TestVersionInfo.java: final
> String scmUrl = "scm:git:https://git-wip-us.apache.org/repos/asf/tez.git;;
> tez-api/src/test/resources/test1-version-info.properties:scmurl=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-api/src/test/resources/test3-version-info.properties:scmurl=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-ui/src/main/webapp/package.json:"url":
> "https://git-wip-us.apache.org/repos/asf/tez.git;
> {code}
> In addition the cwiki needs to be updated
> https://cwiki.apache.org/confluence/display/TEZ/Committer+Guide
> https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez
> https://cwiki.apache.org/confluence/display/TEZ/Making+a+TEZ+Release
> https://cwiki.apache.org/confluence/display/TEZ/Updating+the+Tez+Website
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (TEZ-3205) Update tez poms to remove hadoop 2.2/2.4 profiles
[ https://issues.apache.org/jira/browse/TEZ-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-3205. -- Resolution: Invalid Closing now that these profiles are no longer valid and have already been removed > Update tez poms to remove hadoop 2.2/2.4 profiles > - > > Key: TEZ-3205 > URL: https://issues.apache.org/jira/browse/TEZ-3205 > Project: Apache Tez > Issue Type: Bug >Reporter: Siddharth Seth >Priority: Major > Labels: newbie > Attachments: TEZ-3205.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-2884) Allow javadocs to be generated with Java8
[ https://issues.apache.org/jira/browse/TEZ-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782045#comment-16782045 ] Jonathan Eagles commented on TEZ-2884: -- Closing this in favor for TEZ-4025 > Allow javadocs to be generated with Java8 > - > > Key: TEZ-2884 > URL: https://issues.apache.org/jira/browse/TEZ-2884 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > > Java 8 introduces stricter javadoc checks, which causes javadoc generation to > fail. > Allow javadocs to be generated, while we fix the actual issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-2884) Allow javadocs to be generated with Java8
[ https://issues.apache.org/jira/browse/TEZ-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-2884. -- Resolution: Duplicate > Allow javadocs to be generated with Java8 > - > > Key: TEZ-2884 > URL: https://issues.apache.org/jira/browse/TEZ-2884 > Project: Apache Tez > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Major > > Java 8 introduces stricter javadoc checks, which causes javadoc generation to > fail. > Allow javadocs to be generated, while we fix the actual issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4031) Support tez gitbox migration
[
https://issues.apache.org/jira/browse/TEZ-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4031:
-
Attachment: TEZ-4031.004.patch
> Support tez gitbox migration
>
>
> Key: TEZ-4031
> URL: https://issues.apache.org/jira/browse/TEZ-4031
> Project: Apache Tez
> Issue Type: Task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4031.001.patch, TEZ-4031.002.patch,
> TEZ-4031.003.patch, TEZ-4031.004.patch
>
>
> {code}
> $ git grep git-wip
> Tez_DOAP.rdf: rdf:resource="https://git-wip-us.apache.org/repos/asf/tez.git"/>
> Tez_DOAP.rdf: rdf:resource="https://git-wip-us.apache.org/repos/asf?p=tez.git"/>
> docs/src/site/site.xml: href="https://git-wip-us.apache.org/repos/asf/tez.git; alt="Use git clone
> https://git-wip-us.apache.org/repos/asf/tez.git; />
> pom.xml:
> scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-api/src/test/java/org/apache/tez/common/TestVersionInfo.java: final
> String scmUrl = "scm:git:https://git-wip-us.apache.org/repos/asf/tez.git;;
> tez-api/src/test/resources/test1-version-info.properties:scmurl=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-api/src/test/resources/test3-version-info.properties:scmurl=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-ui/src/main/webapp/package.json:"url":
> "https://git-wip-us.apache.org/repos/asf/tez.git;
> {code}
> In addition the cwiki needs to be updated
> https://cwiki.apache.org/confluence/display/TEZ/Committer+Guide
> https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez
> https://cwiki.apache.org/confluence/display/TEZ/Making+a+TEZ+Release
> https://cwiki.apache.org/confluence/display/TEZ/Updating+the+Tez+Website
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4031) Support tez gitbox migration
[
https://issues.apache.org/jira/browse/TEZ-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781895#comment-16781895
]
Jonathan Eagles commented on TEZ-4031:
--
The need for that maven-site-plugin change and
maven-project-info-reports-plugin is needed to address the byte code issue as
shown in the mvnsite goals above. Partly this was needed to address the new
jetty 9.3.24 jar which has this new byte code formatting (java 8 thing?). In
Hadoop this was covered by HADOOP-13671 (2.7->2.9) HADOOP-14401
(2.9->latest)(meaning when run, get the latest version. IMHO not the best way)
{code}
[WARNING] Unable to process class
org/eclipse/jetty/server/SslConnectionFactory.class in JarAnalyzer File
/home/jenkins/jenkins-slave/workspace/PreCommit-TEZ-Build/yetus-m2/tez-master-patch-1/org/eclipse/jetty/jetty-server/9.3.24.v20180605/jetty-server-9.3.24.v20180605.jar
org.apache.bcel.classfile.ClassFormatException: Invalid byte tag in constant
pool: 18
{code}
> Support tez gitbox migration
>
>
> Key: TEZ-4031
> URL: https://issues.apache.org/jira/browse/TEZ-4031
> Project: Apache Tez
> Issue Type: Task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4031.001.patch, TEZ-4031.002.patch,
> TEZ-4031.003.patch
>
>
> {code}
> $ git grep git-wip
> Tez_DOAP.rdf: rdf:resource="https://git-wip-us.apache.org/repos/asf/tez.git"/>
> Tez_DOAP.rdf: rdf:resource="https://git-wip-us.apache.org/repos/asf?p=tez.git"/>
> docs/src/site/site.xml: href="https://git-wip-us.apache.org/repos/asf/tez.git; alt="Use git clone
> https://git-wip-us.apache.org/repos/asf/tez.git; />
> pom.xml:
> scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-api/src/test/java/org/apache/tez/common/TestVersionInfo.java: final
> String scmUrl = "scm:git:https://git-wip-us.apache.org/repos/asf/tez.git;;
> tez-api/src/test/resources/test1-version-info.properties:scmurl=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-api/src/test/resources/test3-version-info.properties:scmurl=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-ui/src/main/webapp/package.json:"url":
> "https://git-wip-us.apache.org/repos/asf/tez.git;
> {code}
> In addition the cwiki needs to be updated
> https://cwiki.apache.org/confluence/display/TEZ/Committer+Guide
> https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez
> https://cwiki.apache.org/confluence/display/TEZ/Making+a+TEZ+Release
> https://cwiki.apache.org/confluence/display/TEZ/Updating+the+Tez+Website
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-4031) Support tez gitbox migration
[
https://issues.apache.org/jira/browse/TEZ-4031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4031:
-
Attachment: TEZ-4031.003.patch
> Support tez gitbox migration
>
>
> Key: TEZ-4031
> URL: https://issues.apache.org/jira/browse/TEZ-4031
> Project: Apache Tez
> Issue Type: Task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4031.001.patch, TEZ-4031.002.patch,
> TEZ-4031.003.patch
>
>
> {code}
> $ git grep git-wip
> Tez_DOAP.rdf: rdf:resource="https://git-wip-us.apache.org/repos/asf/tez.git"/>
> Tez_DOAP.rdf: rdf:resource="https://git-wip-us.apache.org/repos/asf?p=tez.git"/>
> docs/src/site/site.xml: href="https://git-wip-us.apache.org/repos/asf/tez.git; alt="Use git clone
> https://git-wip-us.apache.org/repos/asf/tez.git; />
> pom.xml:
> scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-api/src/test/java/org/apache/tez/common/TestVersionInfo.java: final
> String scmUrl = "scm:git:https://git-wip-us.apache.org/repos/asf/tez.git;;
> tez-api/src/test/resources/test1-version-info.properties:scmurl=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-api/src/test/resources/test3-version-info.properties:scmurl=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git
> tez-ui/src/main/webapp/package.json:"url":
> "https://git-wip-us.apache.org/repos/asf/tez.git;
> {code}
> In addition the cwiki needs to be updated
> https://cwiki.apache.org/confluence/display/TEZ/Committer+Guide
> https://cwiki.apache.org/confluence/display/TEZ/How+to+Contribute+to+Tez
> https://cwiki.apache.org/confluence/display/TEZ/Making+a+TEZ+Release
> https://cwiki.apache.org/confluence/display/TEZ/Updating+the+Tez+Website
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4002) CHANGES.txt for 0.9.2 Release
[ https://issues.apache.org/jira/browse/TEZ-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781821#comment-16781821 ] Jonathan Eagles commented on TEZ-4002: -- [~kshukla], Apart from the whitespace issues, we will want to look at the current set of issues to make sure there is no blockers for the release. In particular, I think we will want TEZ-4031 to reflect the new git repo location and possibly TEZ-3995 which may cause issues with producing a successful build. I also see lines like "Reverted" and I we should discuss what to do about that. > CHANGES.txt for 0.9.2 Release > - > > Key: TEZ-4002 > URL: https://issues.apache.org/jira/browse/TEZ-4002 > Project: Apache Tez > Issue Type: Task >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: TEZ-4002.001.patch, TEZ-4002.002.patch > > > Add CHANGES.txt for 0.9.2 line. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4048) Make proto history logger queue size configurable
[ https://issues.apache.org/jira/browse/TEZ-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781808#comment-16781808 ] Jonathan Eagles commented on TEZ-4048: -- +1. I'm plus one on this patch. If I don't hear anything [~prasanth_j] or [~gopalv], I'll assume everyone is fine with going into both master and branch-0.9 lines. > Make proto history logger queue size configurable > - > > Key: TEZ-4048 > URL: https://issues.apache.org/jira/browse/TEZ-4048 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.9.next >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Attachments: TEZ-4048.1.patch > > > Currently, the queue size is hard-coded to 10K which may be small for some > bigger cluster. Make it configurable and bump up the default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3664) testSingleSpill_WithPipelinedShuffle[0] fails intermittently
[
https://issues.apache.org/jira/browse/TEZ-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781080#comment-16781080
]
Jonathan Eagles commented on TEZ-3664:
--
[~yeshavora], I was able to find recent occurrences of this issue and was able
to possibly analyze the cause.
Similar but not exact test case failing with cause
{code}
FAILED:
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testTextMixedRecordsWithoutFinalMerge[test[false,
PRECISE]]
Error Message:
chmod: cannot access
'/tmp/TestUnorderedPartitionedKVWriter/outDir_1552b697-70ad-4f9a-85bf-c0155af75fd5/output/1552b697-70ad-4f9a-85bf-c0155af75fd5_17/file.out.index':
No such file or directory
Stack Trace:
org.apache.hadoop.util.Shell$ExitCodeException: chmod: cannot access
'/tmp/TestUnorderedPartitionedKVWriter/outDir_1552b697-70ad-4f9a-85bf-c0155af75fd5/output/1552b697-70ad-4f9a-85bf-c0155af75fd5_17/file.out.index':
No such file or directory
at org.apache.hadoop.util.Shell.runCommand(Shell.java:994)
at org.apache.hadoop.util.Shell.run(Shell.java:887)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1212)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1306)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1288)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:777)
at
org.apache.tez.runtime.library.common.sort.impl.TezSpillRecord.writeToFile(TezSpillRecord.java:146)
at
org.apache.tez.runtime.library.common.sort.impl.TezSpillRecord.writeToFile(TezSpillRecord.java:122)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.handleSpillIndex(UnorderedPartitionedKVWriter.java:1147)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.writeLargeRecord(UnorderedPartitionedKVWriter.java:1125)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:412)
at
org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:368)
at
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.textTest(TestUnorderedPartitionedKVWriter.java:424)
at
org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testTextMixedRecordsWithoutFinalMerge(TestUnorderedPartitionedKVWriter.java:344)
{code}
I think the key take-away is that tests are writing outside of the test
directory and in this case the '/tmp' directory. Many machines may have a temp
cleaner that deletes temporary files written to this directory. A fix is to
write test data within the repository as does Apache Hadoop to keep better
control of the files under test.
> testSingleSpill_WithPipelinedShuffle[0] fails intermittently
>
>
> Key: TEZ-3664
> URL: https://issues.apache.org/jira/browse/TEZ-3664
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Yesha Vora
>Priority: Major
>
> testSingleSpill_WithPipelinedShuffle[0] unit test fails intermittently
> {code}
> Stacktrace
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTestWithPipelinedTransfer(TestUnorderedPartitionedKVWriter.java:503)
> at
> org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSingleSpill_WithPipelinedShuffle(TestUnorderedPartitionedKVWriter.java:414)
> Standard Output
> 2017-03-20 19:50:52,538 INFO [main] writers.TestUnorderedPartitionedKVWriter
> (TestUnorderedPartitionedKVWriter.java:setup(124)) - Setup. Using test dir:
> /tmp/TestUnorderedPartitionedKVWriter
> 2017-03-20 19:50:53,065 WARN [main] util.NativeCodeLoader
> (NativeCodeLoader.java:(62)) - Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 2017-03-20 19:50:53,856 INFO [Thread-5] counters.Limits
> (Limits.java:ensureInitialized(60)) - Counter limits initialized with
> parameters: GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64,
> MAX_COUNTERS=1200
> 2017-03-20 19:50:53,861 INFO [Thread-5]
> writers.BaseUnorderedPartitionedKVWriter
> (BaseUnorderedPartitionedKVWriter.java:(148)) - Instantiating
> Partitioner:
> [org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
> 2017-03-20 19:50:53,870 INFO [Thread-5] writers.UnorderedPartitionedKVWriter
> (UnorderedPartitionedKVWriter.java:(221)) - destinationVertexName:
> numBuffers=2, sizePerBuffer=1024,
[jira] [Commented] (TEZ-3704) Tez-UI unit test failing
[
https://issues.apache.org/jira/browse/TEZ-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780882#comment-16780882
]
Jonathan Eagles commented on TEZ-3704:
--
[~yeshavora], if you are still having this issue please reach out to
u...@tez.apache.org to get the best help for this issue. Closing this issue.
> Tez-UI unit test failing
>
>
> Key: TEZ-3704
> URL: https://issues.apache.org/jira/browse/TEZ-3704
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Yesha Vora
>Priority: Major
>
> tez-ui unit test is failing as below.
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] tez SUCCESS [ 1.171
> s]
> [INFO] tez-api SUCCESS [ 25.416
> s]
> [INFO] tez-common . SUCCESS [ 0.156
> s]
> [INFO] tez-runtime-internals .. SUCCESS [ 0.812
> s]
> [INFO] tez-runtime-library SUCCESS [ 1.190
> s]
> [INFO] tez-mapreduce .. SUCCESS [ 2.787
> s]
> [INFO] tez-examples ... SUCCESS [ 0.127
> s]
> [INFO] tez-dag SUCCESS [ 4.707
> s]
> [INFO] tez-tests .. SUCCESS [ 7.205
> s]
> [INFO] tez-ui . FAILURE [01:29
> min]
> [INFO] tez-plugins SKIPPED
> [INFO] tez-yarn-timeline-history .. SKIPPED
> [INFO] tez-history-parser . SKIPPED
> [INFO] tez-yarn-timeline-history-with-acls SKIPPED
> [INFO] tez-yarn-timeline-cache-plugin . SKIPPED
> [INFO] tez-yarn-timeline-history-with-fs .. SKIPPED
> [INFO] tez-tools .. SKIPPED
> [INFO] tez-perf-analyzer .. SKIPPED
> [INFO] tez-job-analyzer ... SKIPPED
> [INFO] tez-dist ... SKIPPED
> [INFO] Tez SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:14 min
> [INFO] Finished at: 2017-04-19T19:31:02+00:00
> [INFO] Final Memory: 51M/885M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.2:exec
> (ember test) on project tez-ui: Command execution failed. Process exited with
> an error: 1 (Exit value: 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :tez-ui{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3686) Tez UI: Cleanup LICENSE file
[ https://issues.apache.org/jira/browse/TEZ-3686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3686: - Labels: UI (was: ) > Tez UI: Cleanup LICENSE file > > > Key: TEZ-3686 > URL: https://issues.apache.org/jira/browse/TEZ-3686 > Project: Apache Tez > Issue Type: Bug >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram >Priority: Major > Labels: UI > > LICENSE must just contain dependent packages that would be part of tez-dist. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-3704) Tez-UI unit test failing
[
https://issues.apache.org/jira/browse/TEZ-3704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles resolved TEZ-3704.
--
Resolution: Cannot Reproduce
> Tez-UI unit test failing
>
>
> Key: TEZ-3704
> URL: https://issues.apache.org/jira/browse/TEZ-3704
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Yesha Vora
>Priority: Major
>
> tez-ui unit test is failing as below.
> {code}
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] tez SUCCESS [ 1.171
> s]
> [INFO] tez-api SUCCESS [ 25.416
> s]
> [INFO] tez-common . SUCCESS [ 0.156
> s]
> [INFO] tez-runtime-internals .. SUCCESS [ 0.812
> s]
> [INFO] tez-runtime-library SUCCESS [ 1.190
> s]
> [INFO] tez-mapreduce .. SUCCESS [ 2.787
> s]
> [INFO] tez-examples ... SUCCESS [ 0.127
> s]
> [INFO] tez-dag SUCCESS [ 4.707
> s]
> [INFO] tez-tests .. SUCCESS [ 7.205
> s]
> [INFO] tez-ui . FAILURE [01:29
> min]
> [INFO] tez-plugins SKIPPED
> [INFO] tez-yarn-timeline-history .. SKIPPED
> [INFO] tez-history-parser . SKIPPED
> [INFO] tez-yarn-timeline-history-with-acls SKIPPED
> [INFO] tez-yarn-timeline-cache-plugin . SKIPPED
> [INFO] tez-yarn-timeline-history-with-fs .. SKIPPED
> [INFO] tez-tools .. SKIPPED
> [INFO] tez-perf-analyzer .. SKIPPED
> [INFO] tez-job-analyzer ... SKIPPED
> [INFO] tez-dist ... SKIPPED
> [INFO] Tez SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:14 min
> [INFO] Finished at: 2017-04-19T19:31:02+00:00
> [INFO] Final Memory: 51M/885M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.2:exec
> (ember test) on project tez-ui: Command execution failed. Process exited with
> an error: 1 (Exit value: 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :tez-ui{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3706) add option to skip Tez UI build
[ https://issues.apache.org/jira/browse/TEZ-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3706: - Labels: UI build (was: ) > add option to skip Tez UI build > --- > > Key: TEZ-3706 > URL: https://issues.apache.org/jira/browse/TEZ-3706 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > Labels: UI, build > > The UI build takes forever downloading some files and messing around. It > should be possible to skip it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3727) When using HDFS federation, token of tez.simple.history.logging.dir is not added, causing AM to fail
[
https://issues.apache.org/jira/browse/TEZ-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780880#comment-16780880
]
Jonathan Eagles commented on TEZ-3727:
--
[~jshmchenxi], does the fix provided in TEZ-4032 help to solve this issue?
> When using HDFS federation, token of tez.simple.history.logging.dir is not
> added, causing AM to fail
>
>
> Key: TEZ-3727
> URL: https://issues.apache.org/jira/browse/TEZ-3727
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.5
> Environment: hive1.1.0 + tez0.8.5
>Reporter: Xi Chen
>Priority: Major
> Fix For: 0.8.5
>
> Attachments: TEZ-3727.patch
>
>
> If we use different fs for tez.simple.history.logging.dir and
> hive.exec.scratchdir, the tez AM throws such exception:
> {noformat}
> [INFO] [main] |retry.RetryInvocationHandler|: Exception while invoking
> getFileInfo of class ClientNamenodeProtocolTranslatorPB over
> ns/xx.xx.xx.xx: after 1 fail over attempts. Trying to fail over
> immediately.
> java.io.IOException: Failed on local exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]; Host Details : local host is: "nm1/xx.xx.xx.xx";
> destination host is: "ns":;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1472)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy12.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy13.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1982)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1128)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1124)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1124)
> at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
> at
> org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService.serviceInit(SimpleHistoryLoggingService.java:81)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.tez.dag.history.HistoryEventHandler.serviceInit(HistoryEventHandler.java:100)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.tez.dag.app.DAGAppMaster.initServices(DAGAppMaster.java:1933)
> at
> org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:622)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at org.apache.tez.dag.app.DAGAppMaster$8.run(DAGAppMaster.java:2586)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2583)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2388)
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
> at
>
[jira] [Commented] (TEZ-3706) add option to skip Tez UI build
[ https://issues.apache.org/jira/browse/TEZ-3706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780879#comment-16780879 ] Jonathan Eagles commented on TEZ-3706: -- mvn clean install -pl '!tez-ui' is a one way to skip this. [~sershe], do you want something more formal or documentation. there is a sub issue where if running tests, all tez-ui tests are run whether or not tests match the -Dtest=. What do you recommend for this issue? > add option to skip Tez UI build > --- > > Key: TEZ-3706 > URL: https://issues.apache.org/jira/browse/TEZ-3706 > Project: Apache Tez > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Major > > The UI build takes forever downloading some files and messing around. It > should be possible to skip it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3993) Tez fails to parse windows file paths in local mode
[
https://issues.apache.org/jira/browse/TEZ-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-3993:
-
Attachment: TEZ-3993.003.patch
> Tez fails to parse windows file paths in local mode
> ---
>
> Key: TEZ-3993
> URL: https://issues.apache.org/jira/browse/TEZ-3993
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-3993.001.patch, TEZ-3993.002.patch,
> TEZ-3993.003.patch
>
>
> TezLocalCacheManager tries to generate symlinks to files that it puts in the
> local cache, but the code that it uses to construct the path names is not
> safe on Windows and causes bad file names to be constructed when run in a
> Windows environment. On Windows, a path like file:/c:/path/to/my/file should
> be legal and transform to c:\path\to\my\file, but with the invalid construct,
> it turns into the illegal value /c:/path/to/my/file instead.
> In TezLocalCacheManager, there is code that does
> {code}
> private boolean createSymlink(Path target, Path link) throws IOException {
> LOG.info("Creating symlink: {} <- {}", target, link);
> String targetPath = target.toUri().getPath();
> String linkPath = link.toUri().getPath();
> {code}
> It looks like there are several other places in the Tez code that also use
> the Path.toUri().getPath() construct that probably also need to be fixed in
> order to work correctly on Windows.
> The construct Path.toUri().getPath() doesn't handle windows directories
> correctly. The Java File class understands how to do this correctly, so this
> should really be replaced by
> {code}
> private boolean createSymlink(Path target, Path link) throws IOException {
> LOG.info("Creating symlink: {} <- {}", target, link);
> String targetPath = new File(target.toUri()).getCanonicalPath();
> String linkPath = new File(link.toUri()).getCanonicalPath();
> {code}
> {code}
> 2018-09-19T16:32:53,287 ERROR [LocalContainerLauncher-SubTaskRunner]
> org.apache.tez.dag.app.launcher.LocalContainerLauncher - TezSubTaskRunner
> failed due to exception
> java.nio.file.InvalidPathException: Illegal char <:> at index 2:
> /C:/Users/...fullpath
> at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
> at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
> at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
> at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
> at java.nio.file.Paths.get(Paths.java:84)
> at
> org.apache.tez.dag.app.launcher.TezLocalCacheManager.createSymlink(TezLocalCacheManager.java:173)
> at
> org.apache.tez.dag.app.launcher.TezLocalCacheManager.localize(TezLocalCacheManager.java:126)
> at
> org.apache.tez.dag.app.launcher.LocalContainerLauncher.launch(LocalContainerLauncher.java:263)
> at
> org.apache.tez.dag.app.launcher.LocalContainerLauncher.access$300(LocalContainerLauncher.java:82)
> at
> org.apache.tez.dag.app.launcher.LocalContainerLauncher$TezSubTaskRunner.run(LocalContainerLauncher.java:207)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (TEZ-1069) Support ability to re-size a task attempt when previous attempts fail due to resource constraints
[ https://issues.apache.org/jira/browse/TEZ-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780763#comment-16780763 ] Jonathan Eagles edited comment on TEZ-1069 at 2/28/19 5:30 PM: --- Planning on closing this as won't fix unless I hear objections was (Author: jeagles): Planning on closing this as won't fix unless I hear objects > Support ability to re-size a task attempt when previous attempts fail due to > resource constraints > - > > Key: TEZ-1069 > URL: https://issues.apache.org/jira/browse/TEZ-1069 > Project: Apache Tez > Issue Type: Improvement >Reporter: Hitesh Shah >Assignee: Jeff Zhang >Priority: Major > Attachments: TEZ-1069-1.patch > > > Consider a case where attempts for the final stage in a long DAG fails due to > out of memory. In such a scenario, the framework ( or via the base vertex > manager ) should be able to change the task specifications on the fly to > trigger a re-run with modified specs. > Changes could be both java opts changes as well as container resource > requirements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-1069) Support ability to re-size a task attempt when previous attempts fail due to resource constraints
[ https://issues.apache.org/jira/browse/TEZ-1069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780763#comment-16780763 ] Jonathan Eagles commented on TEZ-1069: -- Planning on closing this as won't fix unless I hear objects > Support ability to re-size a task attempt when previous attempts fail due to > resource constraints > - > > Key: TEZ-1069 > URL: https://issues.apache.org/jira/browse/TEZ-1069 > Project: Apache Tez > Issue Type: Improvement >Reporter: Hitesh Shah >Assignee: Jeff Zhang >Priority: Major > Attachments: TEZ-1069-1.patch > > > Consider a case where attempts for the final stage in a long DAG fails due to > out of memory. In such a scenario, the framework ( or via the base vertex > manager ) should be able to change the task specifications on the fly to > trigger a re-run with modified specs. > Changes could be both java opts changes as well as container resource > requirements. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (TEZ-2230) Speculative attempt should not have the original attempts machine in its preferred locations
[ https://issues.apache.org/jira/browse/TEZ-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-2230. -- Resolution: Duplicate > Speculative attempt should not have the original attempts machine in its > preferred locations > > > Key: TEZ-2230 > URL: https://issues.apache.org/jira/browse/TEZ-2230 > Project: Apache Tez > Issue Type: Sub-task >Affects Versions: 0.6.0 >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3842) Tez UI: Upgrade to the latest em-table
[ https://issues.apache.org/jira/browse/TEZ-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-3842: - Component/s: UI > Tez UI: Upgrade to the latest em-table > -- > > Key: TEZ-3842 > URL: https://issues.apache.org/jira/browse/TEZ-3842 > Project: Apache Tez > Issue Type: Bug > Components: UI >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram >Priority: Major > > em-table have improved a lot in the past few months. SQL like advanced > searching capability and Faceted filters are the best among them. All the > inner tables could take advantage of these features and pump Tez UI to the > next level! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4047) Tez trademark in xml is causing xml parsing issue
[
https://issues.apache.org/jira/browse/TEZ-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4047:
-
Description:
{code}
docs/src/site/site.xml:
[Fatal Error] site.xml:97:34: The entity "reg" was referenced, but not declared.
java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId:
file:/testptch/tez/./docs/src/site/site.xml; lineNumber: 97; columnNumber: 34;
The entity "reg" was referenced, but not declared.
at
jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
at
jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at com.sun.tools.script.shell.Main.evaluateString(Main.java:298)
at com.sun.tools.script.shell.Main.evaluateString(Main.java:319)
at com.sun.tools.script.shell.Main.access$300(Main.java:37)
at com.sun.tools.script.shell.Main$3.run(Main.java:217)
at com.sun.tools.script.shell.Main.main(Main.java:48)
Caused by: org.xml.sax.SAXParseException; systemId:
file:/testptch/tez/./docs/src/site/site.xml; lineNumber: 97; columnNumber: 34;
The entity "reg" was referenced, but not declared.
at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:205)
at
jdk.nashorn.internal.scripts.Script$Recompilation$2$19313A$\^system_init\_.XMLDocument(:747)
at jdk.nashorn.internal.scripts.Script$1$\^string\_.:program(:1)
at
jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637)
at
jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494)
at
jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393)
... 10 more
{code}
Also output from xmllint verifies xml issue as well.
{code}
xmllint ./docs/src/site/site.xml
.//src/site/site.xml:97: parser error : Entity 'reg' not defined
http://tez.apache.org/"/>
^
.//src/site/site.xml:123: parser error : Entity 'reg' not defined
{code}
was:
{code}
docs/src/site/site.xml:
[Fatal Error] site.xml:97:34: The entity "reg" was referenced, but not declared.
java.lang.RuntimeException: org.xml.sax.SAXParseException; systemId:
file:/testptch/tez/./docs/src/site/site.xml; lineNumber: 97; columnNumber: 34;
The entity "reg" was referenced, but not declared.
at
jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:397)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:449)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:406)
at
jdk.nashorn.api.scripting.NashornScriptEngine.evalImpl(NashornScriptEngine.java:402)
at
jdk.nashorn.api.scripting.NashornScriptEngine.eval(NashornScriptEngine.java:155)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
at com.sun.tools.script.shell.Main.evaluateString(Main.java:298)
at com.sun.tools.script.shell.Main.evaluateString(Main.java:319)
at com.sun.tools.script.shell.Main.access$300(Main.java:37)
at com.sun.tools.script.shell.Main$3.run(Main.java:217)
at com.sun.tools.script.shell.Main.main(Main.java:48)
Caused by: org.xml.sax.SAXParseException; systemId:
file:/testptch/tez/./docs/src/site/site.xml; lineNumber: 97; columnNumber: 34;
The entity "reg" was referenced, but not declared.
at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:205)
at
jdk.nashorn.internal.scripts.Script$Recompilation$2$19313A$\^system_init\_.XMLDocument(:747)
at jdk.nashorn.internal.scripts.Script$1$\^string\_.:program(:1)
at
jdk.nashorn.internal.runtime.ScriptFunctionData.invoke(ScriptFunctionData.java:637)
at
jdk.nashorn.internal.runtime.ScriptFunction.invoke(ScriptFunction.java:494)
at
jdk.nashorn.internal.runtime.ScriptRuntime.apply(ScriptRuntime.java:393)
... 10 more
{code}
Also output from xmllint verifies xml issue as well.
{code}
xmllint .docs/src/site/site.xml
.//src/site/site.xml:97: parser error : Entity 'reg' not defined
http://tez.apache.org/"/>
^
.//src/site/site.xml:123: parser error :
[jira] [Resolved] (TEZ-3928) [Umbrella] Hadoop 3 test failures
[ https://issues.apache.org/jira/browse/TEZ-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved TEZ-3928. -- Resolution: Fixed > [Umbrella] Hadoop 3 test failures > - > > Key: TEZ-3928 > URL: https://issues.apache.org/jira/browse/TEZ-3928 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4050) maven site is failing due to missing configuration.
[
https://issues.apache.org/jira/browse/TEZ-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4050:
-
Issue Type: Sub-task (was: Bug)
Parent: TEZ-3891
> maven site is failing due to missing configuration.
> ---
>
> Key: TEZ-4050
> URL: https://issues.apache.org/jira/browse/TEZ-4050
> Project: Apache Tez
> Issue Type: Sub-task
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4050.001.patch
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.4:stage (default-cli) on project
> tez-docs: Missing site information in the distribution management of the
> project Tez (org.apache.tez:tez-docs:0.10.1-SNAPSHOT) -> [Help 1]
> {code}
> From maven site plugin usage we can see we are missing configuration.
> https://maven.apache.org/plugins/maven-site-plugin/usage.html
> {code}
>
> ...
>
>
> www.yourcompany.com
> scp://www.yourcompany.com/www/docs/project/
>
>
> ...
>
> {code}
> Tez does not use this url to deploy and neither does hadoop. But it is needed
> to stage site documentation. url is only used during site:deploy which is
> never called during Tez QA step.
> This jira aims to provide a place holder (the same as hadoop)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4032) TEZ will throw "Client cannot authenticate via:[TOKEN, KERBEROS]" when used with HDFS federation(non viewfs, only hdfs schema used).
[
https://issues.apache.org/jira/browse/TEZ-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780102#comment-16780102
]
Jonathan Eagles commented on TEZ-4032:
--
Thanks for getting back to me, [~zhangbutao]. Thanks again for this
contribution to Tez.
> TEZ will throw "Client cannot authenticate via:[TOKEN, KERBEROS]" when used
> with HDFS federation(non viewfs, only hdfs schema used).
> --
>
> Key: TEZ-4032
> URL: https://issues.apache.org/jira/browse/TEZ-4032
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Fix For: 0.9.2, 0.10.1
>
> Attachments: TEZ-4032.001.patch, TEZ-4032.002.patch,
> TEZ-4032.003.patch, TEZ-4032.004.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I execute hive tez job in HDFS federation and kerberos. The hadoop cluster
> has multiple namespace (hdfs://ns1,hdfs://ns2,hdfs://ns3 ...)and we don't
> use viewfs schema. Hive tez job will throw error as follows when the table
> is created in hdfs://ns2 (default configuration fs.defaluFS=hdfs://ns1)：
> {code:java}
> 2019-01-21 15:43:46,507 [WARN] [TezChild] |ipc.Client|: Exception encountered
> while connecting to the server :
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> 2019-01-21 15:43:46,507 [INFO] [TezChild] |retry.RetryInvocationHandler|:
> java.io.IOException: DestHost:destPort docker5.cmss.com:8020 ,
> LocalHost:localPort docker1.cmss.com/10.254.10.116:0. Failed on local
> exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS], while invoking
> ClientNamenodeProtocolTranslatorPB.getFileInfo over
> docker5.cmss.com/10.254.2.106:8020 after 14 failover attempts. Trying to
> failover after sleeping for 10827ms.
> 2019-01-21 15:43:57,338 [WARN] [TezChild] |ipc.Client|: Exception encountered
> while connecting to the server :
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> 2019-01-21 15:43:57,363 [ERROR] [TezChild] |tez.MapRecordSource|:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing writable (null)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> DestHost:destPort docker4.cmss.com:8020 , LocalHost:localPort
> docker1.cmss.com/10.254.10.116:0. Failed on local exception:
> java.io.IOException: org.apache.hadoop.security.AccessControlException:
> Client cannot authenticate via:[TOKEN,
[jira] [Commented] (TEZ-4032) TEZ will throw "Client cannot authenticate via:[TOKEN, KERBEROS]" when used with HDFS federation(non viewfs, only hdfs schema used).
[
https://issues.apache.org/jira/browse/TEZ-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780079#comment-16780079
]
Jonathan Eagles commented on TEZ-4032:
--
+1. Committing this to master and branch-0.9. Thank you [~zhangbutao]! On the
commit I made an assumption of what name you want associated with the commit.
Please check the commit to and let me know of any corrections needed to give
you proper credit. Once this is complete, please close the PR if you are able.
(I am not). I am working with hadoop, yetus, and infra to get this supported
soon.
> TEZ will throw "Client cannot authenticate via:[TOKEN, KERBEROS]" when used
> with HDFS federation(non viewfs, only hdfs schema used).
> --
>
> Key: TEZ-4032
> URL: https://issues.apache.org/jira/browse/TEZ-4032
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Attachments: TEZ-4032.001.patch, TEZ-4032.002.patch,
> TEZ-4032.003.patch, TEZ-4032.004.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I execute hive tez job in HDFS federation and kerberos. The hadoop cluster
> has multiple namespace (hdfs://ns1,hdfs://ns2,hdfs://ns3 ...)and we don't
> use viewfs schema. Hive tez job will throw error as follows when the table
> is created in hdfs://ns2 (default configuration fs.defaluFS=hdfs://ns1)：
> {code:java}
> 2019-01-21 15:43:46,507 [WARN] [TezChild] |ipc.Client|: Exception encountered
> while connecting to the server :
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> 2019-01-21 15:43:46,507 [INFO] [TezChild] |retry.RetryInvocationHandler|:
> java.io.IOException: DestHost:destPort docker5.cmss.com:8020 ,
> LocalHost:localPort docker1.cmss.com/10.254.10.116:0. Failed on local
> exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS], while invoking
> ClientNamenodeProtocolTranslatorPB.getFileInfo over
> docker5.cmss.com/10.254.2.106:8020 after 14 failover attempts. Trying to
> failover after sleeping for 10827ms.
> 2019-01-21 15:43:57,338 [WARN] [TezChild] |ipc.Client|: Exception encountered
> while connecting to the server :
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> 2019-01-21 15:43:57,363 [ERROR] [TezChild] |tez.MapRecordSource|:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing writable (null)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
[jira] [Commented] (TEZ-4032) TEZ will throw "Client cannot authenticate via:[TOKEN, KERBEROS]" when used with HDFS federation(non viewfs, only hdfs schema used).
[
https://issues.apache.org/jira/browse/TEZ-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779907#comment-16779907
]
Jonathan Eagles commented on TEZ-4032:
--
[~zhangbutao], change looks mostly good to me. A couple of things I want to
check with you. 1) Is Scope.Vertex needed on new configuration items. This
exposes this information to the VertexImpl running inside the AM. Should this
be Scope.Client or Scope.AM. Don't have enough background know what code this
config is intended for. 2) What email address should I attribute to this
contribution. I can see a chinamobile email address associated with your github
account. Is there a better email address?. 3) the public redundant modifier is
better left for now since the whole file needs to be changed. It is better to
do that all under a new jira. 4) the white space changes in tez-api could be
fixed if a new version of a patch is needed (ported from hadoop). Otherwise,
those can be cleaned as a pre-checkin step. 5) I filed TEZ-4049 to address the
findbugs issues already present in tez code.
Thanks again and I will be watching for your reply.
> TEZ will throw "Client cannot authenticate via:[TOKEN, KERBEROS]" when used
> with HDFS federation(non viewfs, only hdfs schema used).
> --
>
> Key: TEZ-4032
> URL: https://issues.apache.org/jira/browse/TEZ-4032
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.9.1
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Attachments: TEZ-4032.001.patch, TEZ-4032.002.patch,
> TEZ-4032.003.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I execute hive tez job in HDFS federation and kerberos. The hadoop cluster
> has multiple namespace (hdfs://ns1,hdfs://ns2,hdfs://ns3 ...)and we don't
> use viewfs schema. Hive tez job will throw error as follows when the table
> is created in hdfs://ns2 (default configuration fs.defaluFS=hdfs://ns1)：
> {code:java}
> 2019-01-21 15:43:46,507 [WARN] [TezChild] |ipc.Client|: Exception encountered
> while connecting to the server :
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> 2019-01-21 15:43:46,507 [INFO] [TezChild] |retry.RetryInvocationHandler|:
> java.io.IOException: DestHost:destPort docker5.cmss.com:8020 ,
> LocalHost:localPort docker1.cmss.com/10.254.10.116:0. Failed on local
> exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS], while invoking
> ClientNamenodeProtocolTranslatorPB.getFileInfo over
> docker5.cmss.com/10.254.2.106:8020 after 14 failover attempts. Trying to
> failover after sleeping for 10827ms.
> 2019-01-21 15:43:57,338 [WARN] [TezChild] |ipc.Client|: Exception encountered
> while connecting to the server :
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> 2019-01-21 15:43:57,363 [ERROR] [TezChild] |tez.MapRecordSource|:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing writable (null)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
>
[jira] [Updated] (TEZ-4050) maven site is failing due to missing configuration.
[
https://issues.apache.org/jira/browse/TEZ-4050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated TEZ-4050:
-
Attachment: TEZ-4050.001.patch
> maven site is failing due to missing configuration.
> ---
>
> Key: TEZ-4050
> URL: https://issues.apache.org/jira/browse/TEZ-4050
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
>Priority: Major
> Attachments: TEZ-4050.001.patch
>
>
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.4:stage (default-cli) on project
> tez-docs: Missing site information in the distribution management of the
> project Tez (org.apache.tez:tez-docs:0.10.1-SNAPSHOT) -> [Help 1]
> {code}
> From maven site plugin usage we can see we are missing configuration.
> https://maven.apache.org/plugins/maven-site-plugin/usage.html
> {code}
>
> ...
>
>
> www.yourcompany.com
> scp://www.yourcompany.com/www/docs/project/
>
>
> ...
>
> {code}
> Tez does not use this url to deploy and neither does hadoop. But it is needed
> to stage site documentation. url is only used during site:deploy which is
> never called during Tez QA step.
> This jira aims to provide a place holder (the same as hadoop)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4050) maven site is failing due to missing configuration.
Jonathan Eagles created TEZ-4050:
Summary: maven site is failing due to missing configuration.
Key: TEZ-4050
URL: https://issues.apache.org/jira/browse/TEZ-4050
Project: Apache Tez
Issue Type: Bug
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.4:stage (default-cli) on project
tez-docs: Missing site information in the distribution management of the
project Tez (org.apache.tez:tez-docs:0.10.1-SNAPSHOT) -> [Help 1]
{code}
>From maven site plugin usage we can see we are missing configuration.
https://maven.apache.org/plugins/maven-site-plugin/usage.html
{code}
...
www.yourcompany.com
scp://www.yourcompany.com/www/docs/project/
...
{code}
Tez does not use this url to deploy and neither does hadoop. But it is needed
to stage site documentation. url is only used during site:deploy which is never
called during Tez QA step.
This jira aims to provide a place holder (the same as hadoop)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-4046) insert overwrite table without data
[
https://issues.apache.org/jira/browse/TEZ-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779882#comment-16779882
]
Jonathan Eagles commented on TEZ-4046:
--
[~jipeng], this type of issue is better started on the Hive user list
u...@hive.apache.org to see what the issue is.
http://hive.apache.org/mailing_lists.html
> insert overwrite table without data
> ---
>
> Key: TEZ-4046
> URL: https://issues.apache.org/jira/browse/TEZ-4046
> Project: Apache Tez
> Issue Type: Bug
>Affects Versions: 0.8.4
>Reporter: zengjipeng
>Priority: Major
>
> use hive on tez.
> execute as below statement, the target table tmp.zjp_b have no data.
>
> {code:java}
> create table tmp.zjp_a(a string,b string);
> insert into tmp.zjp_a select "a1","1,2,3";
> create table tmp.zjp_b(name string,age string) partitioned by(dt string);
> insert overwrite table tmp.zjp_b partition(dt='2019-02-26')
> select a,age from tmp.zjp_a lateral view explode(split(b,",")) tb as age
> where age='1'
> union all
> select a,age from tmp.zjp_a lateral view explode(split(b,",")) tb as age
> where age='2';
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4049) Fix findbugs issues in NotRunningJob
Jonathan Eagles created TEZ-4049: Summary: Fix findbugs issues in NotRunningJob Key: TEZ-4049 URL: https://issues.apache.org/jira/browse/TEZ-4049 Project: Apache Tez Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles Introduced by TEZ-4035. Remove fixes while keeping 3.2.0 api compatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-4049) Fix findbugs issues in NotRunningJob
[ https://issues.apache.org/jira/browse/TEZ-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4049: - Attachment: TEZ-4049.001.patch > Fix findbugs issues in NotRunningJob > > > Key: TEZ-4049 > URL: https://issues.apache.org/jira/browse/TEZ-4049 > Project: Apache Tez > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: TEZ-4049.001.patch > > > Introduced by TEZ-4035. Remove fixes while keeping 3.2.0 api compatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4048) Make proto history logger queue size configurable
[ https://issues.apache.org/jira/browse/TEZ-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16779845#comment-16779845 ] Jonathan Eagles commented on TEZ-4048: -- Seems good not to be lossy. No other LinkedBlockingQueue in tez has a maxcapacity other that default (Integer.MAX_VALUE) > Make proto history logger queue size configurable > - > > Key: TEZ-4048 > URL: https://issues.apache.org/jira/browse/TEZ-4048 > Project: Apache Tez > Issue Type: Improvement >Affects Versions: 0.9.next >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Attachments: TEZ-4048.1.patch > > > Currently, the queue size is hard-coded to 10K which may be small for some > bigger cluster. Make it configurable and bump up the default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
