[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268239#comment-16268239
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153403800
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268238#comment-16268238
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153405899
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268242#comment-16268242
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153405107
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268243#comment-16268243
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153406144
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268241#comment-16268241
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153404840
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268236#comment-16268236
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153405303
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268247#comment-16268247
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153407320
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/ThrottledResourceManager.java
 ---
@@ -288,8 +287,10 @@ public void setCost(double cost) {
 }
 
 @Override
-public void admit() throws QueueTimeoutException, QueryQueueException {
-  lease = rm.queue().enqueue(foreman.getQueryId(), queryCost);
+public void admit() {
+  QueueAcquirer queueAcquirer = new QueueAcquirer(rm.queue(), foreman, 
queryCost);
+  queueAcquirerThread = new Thread(queueAcquirer);
--- End diff --

Wait thread should not be started here. See above comments. Let's discuss 
via e-mail.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268234#comment-16268234
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153389550
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileUtil.java
 ---
@@ -19,30 +19,35 @@
 
 import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
 
+import java.util.HashMap;
+import java.util.Map;
+
 public class ProfileUtil {
-  // Display names for QueryState enum in UserBitShared.proto
-  private static final String[] queryStateDisplayNames = {
-"Starting", // STARTING = 0
-"Running", // RUNNING = 1
-"Succeeded", // COMPLETED = 2
-"Canceled", // CANCELED = 3
-"Failed", // FAILED = 4
-"CancellationRequested", // CANCELLATION_REQUESTED = 5
-"Enqueued" // ENQUEUED = 6
-  };
 
+  private static final Map queryStateDisplayMap = new 
HashMap<>(QueryState.values().length);
+
+  static {
+queryStateDisplayMap.put(QueryState.PREPARING, "Preparing");
+queryStateDisplayMap.put(QueryState.PLANNING, "Planning");
+queryStateDisplayMap.put(QueryState.ENQUEUED, "Enqueued");
+queryStateDisplayMap.put(QueryState.STARTING, "Starting");
+queryStateDisplayMap.put(QueryState.RUNNING, "Running");
+queryStateDisplayMap.put(QueryState.COMPLETED, "Succeeded");
--- End diff --

We cannot change the QueryState: it is part of the public Protobuf API. The 
reason for this table is that the UI wants to use a different display name.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268244#comment-16268244
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153406564
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java
 ---
@@ -280,29 +280,29 @@ public void interrupted(final InterruptedException 
ex) {
   }
 
   void updateEphemeralState(final QueryState queryState) {
-  // If query is already in zk transient store, ignore the transient 
state update option.
-  // Else, they will not be removed from transient store upon 
completion.
-  if (!inTransientStore &&
-  
!foreman.getQueryContext().getOptions().getOption(ExecConstants.QUERY_TRANSIENT_STATE_UPDATE))
 {
-return;
-  }
+// If query is already in zk transient store, ignore the transient 
state update option.
+// Else, they will not be removed from transient store upon completion.
+if (!inTransientStore && 
!foreman.getQueryContext().getOptions().getOption(ExecConstants.QUERY_TRANSIENT_STATE_UPDATE))
 {
+  return;
+}
 
-  switch (queryState) {
+switch (queryState) {
+  case PREPARING:
+  case PLANNING:
   case ENQUEUED:
   case STARTING:
   case RUNNING:
   case CANCELLATION_REQUESTED:
 runningProfileStore.put(stringQueryId, getQueryInfo());  // store 
as ephemeral query profile.
 inTransientStore = true;
 break;
-
   case COMPLETED:
   case CANCELED:
   case FAILED:
 try {
   runningProfileStore.remove(stringQueryId);
   inTransientStore = false;
-} catch(final Exception e) {
+} catch (final Exception e) {
--- End diff --

Below. What is an "estore"? Is that a typo?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268235#comment-16268235
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153404245
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268237#comment-16268237
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153403438
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -873,105 +791,133 @@ public void close() throws Exception {
 }
   }
 
-  private void moveToState(final QueryState newState, final Exception 
exception) {
-logger.debug(queryIdString + ": State change requested {} --> {}", 
state, newState,
-  exception);
+  public synchronized void moveToState(final QueryState newState, final 
Exception exception) {
+logger.debug(queryIdString + ": State change requested {} --> {}", 
state, newState, exception);
 switch (state) {
-case ENQUEUED:
-  switch (newState) {
-  case FAILED:
-Preconditions.checkNotNull(exception, "exception cannot be null 
when new state is failed");
-recordNewState(newState);
-foremanResult.setFailed(exception);
-foremanResult.close();
-return;
+  case PLANNING:
+switch (newState) {
+  case ENQUEUED:
+recordNewState(newState);
+enqueuedQueries.inc();
+return;
+  case CANCELLATION_REQUESTED:
+assert exception == null;
+recordNewState(newState);
+foremanResult.setCompleted(QueryState.CANCELED);
+foremanResult.close();
+return;
+  case FAILED:
+assert exception != null;
+recordNewState(newState);
+foremanResult.setFailed(exception);
+foremanResult.close();
+return;
+}
+break;
+  case ENQUEUED:
+enqueuedQueries.dec();
+queryManager.markQueueWaitEndTime();
+switch (newState) {
--- End diff --

This is exactly the kind of double-switch statement that is hard to 
understand. Better, a method for each transition: `enqueue()`, `start()`, etc. 
Each transition is then easier to reason about. What are the valid from states? 
What are the accounting operations needed in each transition?

Even better would be to combine the logic for the state transition (adding 
the query to the queue, say) along with the state transition logic, since often 
the new state is the result not simply of a transition, but of some condition 
(such as, when queueing, moving to either ENQUEUED, RUNNING or FAILED depending 
on the queue result.)

Otherwise the logic has to be essentially duplicated.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268230#comment-16268230
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153389793
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/QueryWorkUnit.java ---
@@ -112,4 +115,46 @@ public void applyPlan(PhysicalPlanReader reader) 
throws ForemanSetupException {
   fragments.add(defn.applyPlan(reader));
 }
   }
+
+  /**
+   * Converts list of stores fragments into json,
+   * in case of exception returns text indicating that json was malformed.
+   * Is used for debugging purposes.
+   *
+   * @return fragments json
+   */
+  public String convertFragmentToJson() {
+StringBuilder stringBuilder = new StringBuilder();
+final int fragmentCount = fragments.size();
+int fragmentIndex = 0;
+for (final PlanFragment planFragment : fragments) {
+  final ExecProtos.FragmentHandle fragmentHandle = 
planFragment.getHandle();
+  stringBuilder.append("PlanFragment(");
--- End diff --

This isn't really JSON, is it? JSON syntax does not include parens.

Do we want the output to be valid JSON? Or, a text wrapper around a JSON 
payload?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268245#comment-16268245
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153407221
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/QueryResourceManager.java
 ---
@@ -38,22 +35,17 @@
* For some cases the foreman does not have a full plan, just a cost. In
* this case, this object will not plan memory, but still needs the cost
* to place the job into the correct queue.
-   * @param cost
+   * @param cost query cost
*/
 
   void setCost(double cost);
 
   /**
-   * Admit the query into the cluster. Blocks until the query
-   * can run. (Later revisions may use a more thread-friendly
-   * approach.)
-   * @throws QueryQueueException if something goes wrong with the
-   * queue mechanism
-   * @throws QueueTimeoutException if the query timed out waiting to
-   * be admitted.
+   * Admit the query into the cluster. Starts enqueueing process in 
separate thread.
--- End diff --

If we want to do this, please provide an interface with the needed 
services. Also, if we want to do this, then the implementation should change. 
Maybe admit returns a status: ADMITTED, QUEUED, FAILED. (Or, FAILED can be in 
the form of an exception.)

The mechanism should be: pass in a handle to the query. If admit, tell the 
handle to run. If blocked, figure out how to receive the notification from ZK. 
Once the query is admitted, let it run. If a timeout occurs, tell the handle 
that the query failed.

With a handle, we can unit test the mechanism.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268229#comment-16268229
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153394006
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -426,48 +413,25 @@ private void runPhysicalPlan(final PhysicalPlan plan) 
throws ExecutionSetupExcep
 queryManager.setTotalCost(plan.totalCost());
 work.applyPlan(drillbitContext.getPlanReader());
 logWorkUnit(work);
-admit(work);
-queryManager.setQueueName(queryRM.queueName());
-
-final List planFragments = work.getFragments();
-final PlanFragment rootPlanFragment = work.getRootFragment();
-assert queryId == rootPlanFragment.getHandle().getQueryId();
 
-drillbitContext.getWorkBus().addFragmentStatusListener(queryId, 
queryManager.getFragmentStatusListener());
-
drillbitContext.getClusterCoordinator().addDrillbitStatusListener(queryManager.getDrillbitStatusListener());
+fragmentsRunner.setPlanFragments(work.getFragments());
+fragmentsRunner.setRootPlanFragment(work.getRootFragment());
+fragmentsRunner.setRootOperator(work.getRootOperator());
--- End diff --

Actually, why not just accept the `work` object? I has all the items we 
need.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268248#comment-16268248
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153407376
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/ThrottledResourceManager.java
 ---
@@ -314,13 +315,49 @@ public void exit() {
   lease = null;
 }
 
+@Override
+public void cancel() {
+  if (queueAcquirerThread != null) {
+queueAcquirerThread.interrupt();
+  }
+  foreman.moveToState(QueryState.CANCELED, null);
+}
+
 @Override
 public boolean hasQueue() { return true; }
 
 @Override
 public String queueName() {
   return lease == null ? null : lease.queueName();
 }
+
+/**
+ * Is used to start query enqueue process in separate thread.
+ * Changes query state depending on the result.
+ */
+private class QueueAcquirer implements Runnable {
+
+  private final QueryQueue queryQueue;
+  private final Foreman foreman;
+  private final double queryCost;
+
+  QueueAcquirer(QueryQueue queryQueue, Foreman foreman, double 
queryCost) {
+this.queryQueue = queryQueue;
+this.foreman = foreman;
+this.queryCost = queryCost;
+  }
+
+  @Override
+  public void run() {
+try {
+  queryQueue.enqueue(foreman.getQueryId(), queryCost);
+  foreman.moveToState(QueryState.STARTING, null);
+} catch (Exception e) {
--- End diff --

This has FAR to much visibility into the Foreman. See above.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268246#comment-16268246
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153406881
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/DefaultResourceManager.java
 ---
@@ -73,13 +75,20 @@ public void setCost(double cost) {
 @Override
 public void admit() {
   // No queueing by default
+  foreman.moveToState(UserBitShared.QueryResult.QueryState.STARTING, 
null);
--- End diff --

Would REALLY prefer that the resource manager not depend on the Foreman; 
makes unit testing impossible. The Foreman should depend on this class, but not 
the other way around.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268240#comment-16268240
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153405696
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268233#comment-16268233
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153389627
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileUtil.java
 ---
@@ -19,30 +19,35 @@
 
 import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
 
+import java.util.HashMap;
+import java.util.Map;
+
 public class ProfileUtil {
-  // Display names for QueryState enum in UserBitShared.proto
-  private static final String[] queryStateDisplayNames = {
-"Starting", // STARTING = 0
-"Running", // RUNNING = 1
-"Succeeded", // COMPLETED = 2
-"Canceled", // CANCELED = 3
-"Failed", // FAILED = 4
-"CancellationRequested", // CANCELLATION_REQUESTED = 5
-"Enqueued" // ENQUEUED = 6
-  };
 
+  private static final Map queryStateDisplayMap = new 
HashMap<>(QueryState.values().length);
+
+  static {
+queryStateDisplayMap.put(QueryState.PREPARING, "Preparing");
+queryStateDisplayMap.put(QueryState.PLANNING, "Planning");
+queryStateDisplayMap.put(QueryState.ENQUEUED, "Enqueued");
+queryStateDisplayMap.put(QueryState.STARTING, "Starting");
+queryStateDisplayMap.put(QueryState.RUNNING, "Running");
+queryStateDisplayMap.put(QueryState.COMPLETED, "Succeeded");
+queryStateDisplayMap.put(QueryState.FAILED, "Failed");
+queryStateDisplayMap.put(QueryState.CANCELLATION_REQUESTED, 
"Cancellation Requested");
+queryStateDisplayMap.put(QueryState.CANCELED, "Canceled");
+  }
 
   /**
-   * Utility to return display name for query state
-   * @param queryState
+   * Utility method to return display name for query state
+   * @param queryState query state
* @return display string for query state
*/
-  public final static String getQueryStateDisplayName(QueryState 
queryState) {
-int queryStateOrdinal = queryState.getNumber();
-if (queryStateOrdinal >= queryStateDisplayNames.length) {
-  return queryState.name();
-} else {
-  return queryStateDisplayNames[queryStateOrdinal];
+  public static String getQueryStateDisplayName(QueryState queryState) {
+String displayName = queryStateDisplayMap.get(queryState);
+if (displayName == null) {
--- End diff --

It isn't. But, this project added new states. Before this UI was updated, 
we would want to gracefully handle the intermediate condition. Same as the next 
time someone adds a state.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268231#comment-16268231
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153402805
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -426,48 +413,25 @@ private void runPhysicalPlan(final PhysicalPlan plan) 
throws ExecutionSetupExcep
 queryManager.setTotalCost(plan.totalCost());
 work.applyPlan(drillbitContext.getPlanReader());
 logWorkUnit(work);
-admit(work);
-queryManager.setQueueName(queryRM.queueName());
-
-final List planFragments = work.getFragments();
-final PlanFragment rootPlanFragment = work.getRootFragment();
-assert queryId == rootPlanFragment.getHandle().getQueryId();
 
-drillbitContext.getWorkBus().addFragmentStatusListener(queryId, 
queryManager.getFragmentStatusListener());
-
drillbitContext.getClusterCoordinator().addDrillbitStatusListener(queryManager.getDrillbitStatusListener());
+fragmentsRunner.setPlanFragments(work.getFragments());
+fragmentsRunner.setRootPlanFragment(work.getRootFragment());
+fragmentsRunner.setRootOperator(work.getRootOperator());
 
-logger.debug("Submitting fragments to run.");
-
-// set up the root fragment first so we'll have incoming buffers 
available.
-setupRootFragment(rootPlanFragment, work.getRootOperator());
-
-setupNonRootFragments(planFragments);
-
-moveToState(QueryState.RUNNING, null);
-logger.debug("Fragments running.");
+admit();
   }
 
-  private void admit(QueryWorkUnit work) throws ForemanSetupException {
+  private void admit() throws ForemanSetupException {
 queryManager.markPlanningEndTime();
-try {
-  queryRM.admit();
-} catch (QueueTimeoutException e) {
-  throw UserException
-  .resourceError()
-  .message(e.getMessage())
-  .build(logger);
-} catch (QueryQueueException e) {
-  throw new ForemanSetupException(e.getMessage(), e);
-} finally {
-  queryManager.markQueueWaitEndTime();
-}
-moveToState(QueryState.STARTING, null);
+planningQueries.dec();
--- End diff --

Agree. A better solution is to have a separate function for each new state 
to avoid big hairy switch statements from current state to desired state.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268232#comment-16268232
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user paul-rogers commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153403903
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentsRunner.java
 ---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.work.foreman;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ArrayListMultimap;
+import com.google.common.collect.Multimap;
+import com.google.common.collect.Sets;
+import io.netty.buffer.ByteBuf;
+import org.apache.drill.common.concurrent.ExtendedLatch;
+import org.apache.drill.common.exceptions.ExecutionSetupException;
+import org.apache.drill.common.exceptions.UserException;
+import org.apache.drill.exec.ops.FragmentContext;
+import org.apache.drill.exec.physical.base.FragmentRoot;
+import org.apache.drill.exec.proto.BitControl;
+import org.apache.drill.exec.proto.BitControl.PlanFragment;
+import org.apache.drill.exec.proto.CoordinationProtos;
+import org.apache.drill.exec.proto.GeneralRPCProtos;
+import org.apache.drill.exec.proto.UserBitShared.QueryId;
+import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
+import org.apache.drill.exec.rpc.RpcException;
+import org.apache.drill.exec.rpc.UserClientConnection;
+import org.apache.drill.exec.rpc.control.Controller;
+import org.apache.drill.exec.server.DrillbitContext;
+import org.apache.drill.exec.testing.ControlsInjector;
+import org.apache.drill.exec.testing.ControlsInjectorFactory;
+import org.apache.drill.exec.work.EndpointListener;
+import org.apache.drill.exec.work.WorkManager.WorkerBee;
+import org.apache.drill.exec.work.fragment.FragmentExecutor;
+import org.apache.drill.exec.work.fragment.FragmentStatusReporter;
+import org.apache.drill.exec.work.fragment.NonRootFragmentManager;
+import org.apache.drill.exec.work.fragment.RootFragmentManager;
+
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+/**
+ * Is responsible for submitting query fragments for running (locally and 
remotely).
+ */
+public class FragmentsRunner {
+
+  private static final org.slf4j.Logger logger = 
org.slf4j.LoggerFactory.getLogger(FragmentsRunner.class);
+  private static final ControlsInjector injector = 
ControlsInjectorFactory.getInjector(FragmentsRunner.class);
+
+  private static final long RPC_WAIT_IN_MSECS_PER_FRAGMENT = 5000;
+
+  private final WorkerBee bee;
+  private final UserClientConnection initiatingClient;
+  private final DrillbitContext drillbitContext;
+  private final Foreman foreman;
+
+  private List planFragments;
+  private PlanFragment rootPlanFragment;
+  private FragmentRoot rootOperator;
+
+  public FragmentsRunner(WorkerBee bee, UserClientConnection 
initiatingClient, DrillbitContext drillbitContext, Foreman foreman) {
+this.bee = bee;
+this.initiatingClient = initiatingClient;
+this.drillbitContext = drillbitContext;
+this.foreman = foreman;
+  }
+
+  public WorkerBee getBee() {
+return bee;
+  }
+
+  public void setPlanFragments(List planFragments) {
+this.planFragments = planFragments;
+  }
+
+  public void setRootPlanFragment(PlanFragment rootPlanFragment) {
+this.rootPlanFragment = rootPlanFragment;
+  }
+
+  public void setRootOperator(FragmentRoot rootOperator) {
+this.rootOperator = rootOperator;
+  }
+
+  /**
+   * Submits root and non-root fragments fragments for running.
+   * In case of success move 

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268044#comment-16268044
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r153388800
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceConfig.java
 ---
@@ -30,18 +30,24 @@
 public class WorkspaceConfig {
 
   /** Default workspace is a root directory which supports read, but not 
write. */
-  public static final WorkspaceConfig DEFAULT = new WorkspaceConfig("/", 
false, null);
+  public static final WorkspaceConfig DEFAULT = new WorkspaceConfig("/", 
false, null, false);
 
   private final String location;
   private final boolean writable;
   private final String defaultInputFormat;
-
+  private final Boolean allowAccessOutsideWorkspace; // allow access 
outside the workspace by default. This
--- End diff --

Yes it would, I believe. But we want the value to be `true` for backward 
compatibility. (This also addresses your next comment). So we need to know if 
the value is missing. Can only do that with a non primitive AFAIK.


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268007#comment-16268007
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

Github user akumarb2010 commented on the issue:

https://github.com/apache/drill/pull/1027
  
As this PR is corrupted, we have created a new PR with single commit 
https://github.com/apache/drill/pull/1052


> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4779) Kafka storage plugin support

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268005#comment-16268005
 ] 

ASF GitHub Bot commented on DRILL-4779:
---

GitHub user akumarb2010 opened a pull request:

https://github.com/apache/drill/pull/1052

DRILL-4779: Kafka storage plugin new PR(Kamesh Bhallamudi & Anil Kumar 
Batchu)

As https://github.com/apache/drill/pull/1027 PR is corrupted while squash, 
creating a new PR with same codebase, but with single commit.

This is tested against Kafka and verified unit tests.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/akumarb2010os/drill master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1052.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1052


commit d8b4adab3215ff8d07c8183f42a5c6927785a264
Author: Anil Kumar Batchu 
Date:   2017-11-28T02:44:55Z

DRILL-4779: Kafka storage plugin new PR (Kamesh Bhallamudi & Anil Kumar 
Batchu)




> Kafka storage plugin support
> 
>
> Key: DRILL-4779
> URL: https://issues.apache.org/jira/browse/DRILL-4779
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Storage - Other
>Affects Versions: 1.11.0
>Reporter: B Anil Kumar
>Assignee: B Anil Kumar
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.12.0
>
>
> Implement Kafka storage plugin will enable the strong SQL support for Kafka.
> Initially implementation can target for supporting json and avro message types



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267672#comment-16267672
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r153336979
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceConfig.java
 ---
@@ -30,18 +30,24 @@
 public class WorkspaceConfig {
 
   /** Default workspace is a root directory which supports read, but not 
write. */
-  public static final WorkspaceConfig DEFAULT = new WorkspaceConfig("/", 
false, null);
+  public static final WorkspaceConfig DEFAULT = new WorkspaceConfig("/", 
false, null, false);
 
   private final String location;
   private final boolean writable;
   private final String defaultInputFormat;
-
+  private final Boolean allowAccessOutsideWorkspace; // allow access 
outside the workspace by default. This
--- End diff --

As far I remember `@JsonProperty("allowAccessOutsideWorkspace") boolean 
allowAccessOutsideWorkspace` will set false by default if value is not present 
during deserialization. Could you please check?


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5961) For long running queries (> 10 min) Drill may raise FragmentSetupException for completed/cancelled fragments

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-5961:
-
Fix Version/s: 1.13.0

> For long running queries (> 10 min) Drill may raise FragmentSetupException 
> for completed/cancelled fragments
> 
>
> Key: DRILL-5961
> URL: https://issues.apache.org/jira/browse/DRILL-5961
> Project: Apache Drill
>  Issue Type: Bug
>Reporter: Vlad Rozov
>Assignee: Vlad Rozov
> Fix For: 1.13.0
>
>
> {{WorkEventBus}} uses {{recentlyFinishedFragments}} cache to check for 
> completed or cancelled fragments. Such check is not reliable as entries in 
> {{recentlyFinishedFragments}} expire after 10 minutes, so 
> {{FragmentSetupException}} is raised even for completed or cancelled queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267671#comment-16267671
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user arina-ielchiieva commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r153337275
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceConfig.java
 ---
@@ -30,18 +30,25 @@
 public class WorkspaceConfig {
 
   /** Default workspace is a root directory which supports read, but not 
write. */
-  public static final WorkspaceConfig DEFAULT = new WorkspaceConfig("/", 
false, null);
+  public static final WorkspaceConfig DEFAULT = new WorkspaceConfig("/", 
false, null, false);
 
   private final String location;
   private final boolean writable;
   private final String defaultInputFormat;
-
+  private final Boolean allowAccessOutsideWorkspace; // allow access 
outside the workspace by default. This
+ // field is a Boolean 
(not boolean) so that we can
+ // assign a default 
value if it is not defined in a
+ // storage plugin 
config
   public WorkspaceConfig(@JsonProperty("location") String location,
  @JsonProperty("writable") boolean writable,
- @JsonProperty("defaultInputFormat") String 
defaultInputFormat) {
+ @JsonProperty("defaultInputFormat") String 
defaultInputFormat,
+ @JsonProperty("allowAccessOutsideWorkspace") 
Boolean allowAccessOutsideWorkspace
+  ) {
 this.location = location;
 this.writable = writable;
 this.defaultInputFormat = defaultInputFormat;
+//this.allowAccessOutsideWorkspace = allowAccessOutsideWorkspace != 
null ? allowAccessOutsideWorkspace : false ;
+this.allowAccessOutsideWorkspace = true;
--- End diff --

It seems we should not always set true...


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267632#comment-16267632
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153285121
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -256,12 +239,14 @@ public void run() {
 
 // track how long the query takes
 queryManager.markStartTime();
-enqueuedQueries.dec();
 runningQueries.inc();
 
+queryText = queryRequest.getPlan();
+recordNewState(QueryState.PLANNING);
--- End diff --

why this code is moved out of the try block ? I believe getPlan can throw 
an exception which we should handle. Also, should this be handled under 
moveToState ? 


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267638#comment-16267638
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r15422
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/ThrottledResourceManager.java
 ---
@@ -288,8 +287,10 @@ public void setCost(double cost) {
 }
 
 @Override
-public void admit() throws QueueTimeoutException, QueryQueueException {
-  lease = rm.queue().enqueue(foreman.getQueryId(), queryCost);
+public void admit() {
+  QueueAcquirer queueAcquirer = new QueueAcquirer(rm.queue(), foreman, 
queryCost);
+  queueAcquirerThread = new Thread(queueAcquirer);
--- End diff --

General comment about the approach. We create minimum 2 threads for any 
query. With this, when queuing is enabled, we are creating minimum three 
threads. Instead, when a query is cancelled,  is it possible to interrupt the 
foreman thread itself if that is blocked ?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267639#comment-16267639
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153294703
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileUtil.java
 ---
@@ -19,30 +19,35 @@
 
 import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
 
+import java.util.HashMap;
+import java.util.Map;
+
 public class ProfileUtil {
-  // Display names for QueryState enum in UserBitShared.proto
-  private static final String[] queryStateDisplayNames = {
-"Starting", // STARTING = 0
-"Running", // RUNNING = 1
-"Succeeded", // COMPLETED = 2
-"Canceled", // CANCELED = 3
-"Failed", // FAILED = 4
-"CancellationRequested", // CANCELLATION_REQUESTED = 5
-"Enqueued" // ENQUEUED = 6
-  };
 
+  private static final Map queryStateDisplayMap = new 
HashMap<>(QueryState.values().length);
+
+  static {
+queryStateDisplayMap.put(QueryState.PREPARING, "Preparing");
+queryStateDisplayMap.put(QueryState.PLANNING, "Planning");
+queryStateDisplayMap.put(QueryState.ENQUEUED, "Enqueued");
+queryStateDisplayMap.put(QueryState.STARTING, "Starting");
+queryStateDisplayMap.put(QueryState.RUNNING, "Running");
+queryStateDisplayMap.put(QueryState.COMPLETED, "Succeeded");
--- End diff --

To be consistent, should we name this state QueryState.SUCCEEDED ?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267628#comment-16267628
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153268878
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileUtil.java
 ---
@@ -19,30 +19,35 @@
 
 import org.apache.drill.exec.proto.UserBitShared.QueryResult.QueryState;
 
+import java.util.HashMap;
+import java.util.Map;
+
 public class ProfileUtil {
-  // Display names for QueryState enum in UserBitShared.proto
-  private static final String[] queryStateDisplayNames = {
-"Starting", // STARTING = 0
-"Running", // RUNNING = 1
-"Succeeded", // COMPLETED = 2
-"Canceled", // CANCELED = 3
-"Failed", // FAILED = 4
-"CancellationRequested", // CANCELLATION_REQUESTED = 5
-"Enqueued" // ENQUEUED = 6
-  };
 
+  private static final Map queryStateDisplayMap = new 
HashMap<>(QueryState.values().length);
+
+  static {
+queryStateDisplayMap.put(QueryState.PREPARING, "Preparing");
+queryStateDisplayMap.put(QueryState.PLANNING, "Planning");
+queryStateDisplayMap.put(QueryState.ENQUEUED, "Enqueued");
+queryStateDisplayMap.put(QueryState.STARTING, "Starting");
+queryStateDisplayMap.put(QueryState.RUNNING, "Running");
+queryStateDisplayMap.put(QueryState.COMPLETED, "Succeeded");
+queryStateDisplayMap.put(QueryState.FAILED, "Failed");
+queryStateDisplayMap.put(QueryState.CANCELLATION_REQUESTED, 
"Cancellation Requested");
+queryStateDisplayMap.put(QueryState.CANCELED, "Canceled");
+  }
 
   /**
-   * Utility to return display name for query state
-   * @param queryState
+   * Utility method to return display name for query state
+   * @param queryState query state
* @return display string for query state
*/
-  public final static String getQueryStateDisplayName(QueryState 
queryState) {
-int queryStateOrdinal = queryState.getNumber();
-if (queryStateOrdinal >= queryStateDisplayNames.length) {
-  return queryState.name();
-} else {
-  return queryStateDisplayNames[queryStateOrdinal];
+  public static String getQueryStateDisplayName(QueryState queryState) {
+String displayName = queryStateDisplayMap.get(queryState);
+if (displayName == null) {
--- End diff --

when is it possible for a query to have state that is not in the map ?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267633#comment-16267633
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153297806
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -873,105 +791,133 @@ public void close() throws Exception {
 }
   }
 
-  private void moveToState(final QueryState newState, final Exception 
exception) {
-logger.debug(queryIdString + ": State change requested {} --> {}", 
state, newState,
-  exception);
+  public synchronized void moveToState(final QueryState newState, final 
Exception exception) {
+logger.debug(queryIdString + ": State change requested {} --> {}", 
state, newState, exception);
 switch (state) {
-case ENQUEUED:
-  switch (newState) {
-  case FAILED:
-Preconditions.checkNotNull(exception, "exception cannot be null 
when new state is failed");
-recordNewState(newState);
-foremanResult.setFailed(exception);
-foremanResult.close();
-return;
+  case PLANNING:
+switch (newState) {
+  case ENQUEUED:
+recordNewState(newState);
+enqueuedQueries.inc();
+return;
+  case CANCELLATION_REQUESTED:
+assert exception == null;
+recordNewState(newState);
+foremanResult.setCompleted(QueryState.CANCELED);
+foremanResult.close();
+return;
+  case FAILED:
+assert exception != null;
+recordNewState(newState);
+foremanResult.setFailed(exception);
+foremanResult.close();
+return;
--- End diff --

Here and below, should we have a default case and throw exception i.e. 
catch invalid state transitions ?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267635#comment-16267635
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153323467
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/ThrottledResourceManager.java
 ---
@@ -314,13 +315,49 @@ public void exit() {
   lease = null;
 }
 
+@Override
+public void cancel() {
+  if (queueAcquirerThread != null) {
+queueAcquirerThread.interrupt();
+  }
+  foreman.moveToState(QueryState.CANCELED, null);
+}
+
 @Override
 public boolean hasQueue() { return true; }
 
 @Override
 public String queueName() {
   return lease == null ? null : lease.queueName();
 }
+
+/**
+ * Is used to start query enqueue process in separate thread.
+ * Changes query state depending on the result.
+ */
+private class QueueAcquirer implements Runnable {
+
+  private final QueryQueue queryQueue;
+  private final Foreman foreman;
+  private final double queryCost;
+
+  QueueAcquirer(QueryQueue queryQueue, Foreman foreman, double 
queryCost) {
+this.queryQueue = queryQueue;
+this.foreman = foreman;
+this.queryCost = queryCost;
+  }
+
+  @Override
+  public void run() {
+try {
+  queryQueue.enqueue(foreman.getQueryId(), queryCost);
+  foreman.moveToState(QueryState.STARTING, null);
+} catch (Exception e) {
--- End diff --

Should we handle interruptedException (for cancel above) here ? I don't 
think we want to move Query to FAILED state by default.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267631#comment-16267631
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153296504
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -873,105 +791,133 @@ public void close() throws Exception {
 }
   }
 
-  private void moveToState(final QueryState newState, final Exception 
exception) {
-logger.debug(queryIdString + ": State change requested {} --> {}", 
state, newState,
-  exception);
+  public synchronized void moveToState(final QueryState newState, final 
Exception exception) {
+logger.debug(queryIdString + ": State change requested {} --> {}", 
state, newState, exception);
 switch (state) {
-case ENQUEUED:
-  switch (newState) {
-  case FAILED:
-Preconditions.checkNotNull(exception, "exception cannot be null 
when new state is failed");
-recordNewState(newState);
-foremanResult.setFailed(exception);
-foremanResult.close();
-return;
+  case PLANNING:
--- End diff --

can you add the case for PREPARING as well ?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267634#comment-16267634
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153290872
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -426,48 +413,25 @@ private void runPhysicalPlan(final PhysicalPlan plan) 
throws ExecutionSetupExcep
 queryManager.setTotalCost(plan.totalCost());
 work.applyPlan(drillbitContext.getPlanReader());
 logWorkUnit(work);
-admit(work);
-queryManager.setQueueName(queryRM.queueName());
-
-final List planFragments = work.getFragments();
-final PlanFragment rootPlanFragment = work.getRootFragment();
-assert queryId == rootPlanFragment.getHandle().getQueryId();
 
-drillbitContext.getWorkBus().addFragmentStatusListener(queryId, 
queryManager.getFragmentStatusListener());
-
drillbitContext.getClusterCoordinator().addDrillbitStatusListener(queryManager.getDrillbitStatusListener());
+fragmentsRunner.setPlanFragments(work.getFragments());
+fragmentsRunner.setRootPlanFragment(work.getRootFragment());
+fragmentsRunner.setRootOperator(work.getRootOperator());
 
-logger.debug("Submitting fragments to run.");
-
-// set up the root fragment first so we'll have incoming buffers 
available.
-setupRootFragment(rootPlanFragment, work.getRootOperator());
-
-setupNonRootFragments(planFragments);
-
-moveToState(QueryState.RUNNING, null);
-logger.debug("Fragments running.");
+admit();
   }
 
-  private void admit(QueryWorkUnit work) throws ForemanSetupException {
+  private void admit() throws ForemanSetupException {
 queryManager.markPlanningEndTime();
-try {
-  queryRM.admit();
-} catch (QueueTimeoutException e) {
-  throw UserException
-  .resourceError()
-  .message(e.getMessage())
-  .build(logger);
-} catch (QueryQueueException e) {
-  throw new ForemanSetupException(e.getMessage(), e);
-} finally {
-  queryManager.markQueueWaitEndTime();
-}
-moveToState(QueryState.STARTING, null);
+planningQueries.dec();
--- End diff --

Better to handle increment and decrement of counters in moveToState. 


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to 

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267636#comment-16267636
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153297425
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -873,105 +791,133 @@ public void close() throws Exception {
 }
   }
 
-  private void moveToState(final QueryState newState, final Exception 
exception) {
-logger.debug(queryIdString + ": State change requested {} --> {}", 
state, newState,
-  exception);
+  public synchronized void moveToState(final QueryState newState, final 
Exception exception) {
+logger.debug(queryIdString + ": State change requested {} --> {}", 
state, newState, exception);
 switch (state) {
-case ENQUEUED:
-  switch (newState) {
-  case FAILED:
-Preconditions.checkNotNull(exception, "exception cannot be null 
when new state is failed");
-recordNewState(newState);
-foremanResult.setFailed(exception);
-foremanResult.close();
-return;
+  case PLANNING:
+switch (newState) {
+  case ENQUEUED:
+recordNewState(newState);
+enqueuedQueries.inc();
--- End diff --

Decrement plannedQueries ?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267637#comment-16267637
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153293973
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -177,9 +155,10 @@ public Foreman(final WorkerBee bee, final 
DrillbitContext drillbitContext,
 queryManager = new QueryManager(queryId, queryRequest, 
drillbitContext.getStoreProvider(),
 drillbitContext.getClusterCoordinator(), this);
 
-recordNewState(QueryState.ENQUEUED);
-enqueuedQueries.inc();
+recordNewState(QueryState.PREPARING);
--- End diff --

Better to handle all state transitions under moveToState.


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267629#comment-16267629
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153278034
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -221,11 +200,15 @@ public QueryManager getQueryManager() {
   }
 
   /**
-   * Cancel the query. Asynchronous -- it may take some time for all 
remote fragments to be
-   * terminated.
+   * Cancel the query. Asynchronous -- it may take some time for all 
remote fragments to be terminated.
+   * For planning and enqueued states we cancel immediately since these 
states are done locally.
+   *
+   * Note this can be called from outside of run() on another thread, or 
after run() completes
*/
   public void cancel() {
-// Note this can be called from outside of run() on another thread, or 
after run() completes
+if (QueryState.PLANNING == state || QueryState.ENQUEUED == state) {
+  moveToState(QueryState.CANCELLATION_REQUESTED, null);
--- End diff --

Please add a comment explaining why you are moving to 
CANCELLATION_REQUESTED state only when query is in  PLANNING and ENQUEUED 
states. For ex., why not PREPARING State ?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267630#comment-16267630
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

Github user ppadma commented on a diff in the pull request:

https://github.com/apache/drill/pull/1051#discussion_r153289675
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java ---
@@ -426,48 +413,25 @@ private void runPhysicalPlan(final PhysicalPlan plan) 
throws ExecutionSetupExcep
 queryManager.setTotalCost(plan.totalCost());
 work.applyPlan(drillbitContext.getPlanReader());
 logWorkUnit(work);
-admit(work);
-queryManager.setQueueName(queryRM.queueName());
-
-final List planFragments = work.getFragments();
-final PlanFragment rootPlanFragment = work.getRootFragment();
-assert queryId == rootPlanFragment.getHandle().getQueryId();
 
-drillbitContext.getWorkBus().addFragmentStatusListener(queryId, 
queryManager.getFragmentStatusListener());
-
drillbitContext.getClusterCoordinator().addDrillbitStatusListener(queryManager.getDrillbitStatusListener());
+fragmentsRunner.setPlanFragments(work.getFragments());
+fragmentsRunner.setRootPlanFragment(work.getRootFragment());
+fragmentsRunner.setRootOperator(work.getRootOperator());
--- End diff --

can we have a single fragmentsRunner function that does all these ?


> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4286) Have an ability to put server in quiescent mode of operation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267588#comment-16267588
 ] 

ASF GitHub Bot commented on DRILL-4286:
---

Github user paul-rogers commented on the issue:

https://github.com/apache/drill/pull/921
  
Interesting issue! Let's think about this a bit.

In a production system, we do want the grace period; it is an essential 
part of the graceful shutdown procedure.

However, if we are doing a non-graceful shutdown, the grace is unneeded.

Also, if the cluster contains only one node (as in most unit tests), there 
is nothing to wait for, so the grace period is not needed. The same is true in 
an embedded Drillbit for Sqlline.

So, can we provide a solution that handles these cases rather than simply 
turning off the grace period always?

If using the local cluster coordinator, say, then no grace is needed. If 
using ZK, but there is only one Drillbit, no grace is needed. (There is a race 
condition, but may be OK.)

Or, if we detect we are embedded, no grace period.

Then, also, if we are doing a graceful shutdown, we need the grace. But, if 
we are doing a "classic" shutdown, no grace is needed.

The result should be that the grace period is used only in production 
servers, only when doing a graceful shutdown.

There are probably some details to simplify, but I hope the above 
communicates the idea. Can we make this work?


> Have an ability to put server in quiescent mode of operation
> 
>
> Key: DRILL-4286
> URL: https://issues.apache.org/jira/browse/DRILL-4286
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - Flow
>Affects Versions: 1.11.0
>Reporter: Victoria Markman
>Assignee: Venkata Jyothsna Donapati
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.12.0
>
> Attachments: consoleText.txt, consoleText_after.txt, 
> consoleText_before.txt
>
>
> I think drill will benefit from mode of operation that is called "quiescent" 
> in some databases. 
> From IBM Informix server documentation:
> {code}
> Change gracefully from online to quiescent mode
> Take the database server gracefully from online mode to quiescent mode to 
> restrict access to the database server without interrupting current 
> processing. After you perform this task, the database server sets a flag that 
> prevents new sessions from gaining access to the database server. The current 
> sessions are allowed to finish processing. After you initiate the mode 
> change, it cannot be canceled. During the mode change from online to 
> quiescent, the database server is considered to be in Shutdown mode.
> {code}
> This is different from shutdown, when processes are terminated. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267504#comment-16267504
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r152862722
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java 
---
@@ -359,15 +363,30 @@ private static Path handleWildCard(final String root) 
{
 }
   }
 
-  private static String removeLeadingSlash(String path) {
-if (path.charAt(0) == '/') {
+  public static String removeLeadingSlash(String path) {
+if (!path.isEmpty() && path.charAt(0) == '/') {
   String newPath = path.substring(1);
   return removeLeadingSlash(newPath);
 } else {
   return path;
 }
   }
 
+  // Check if the path is a valid sub path under the parent after removing 
backpaths. Throw an exception if
+  // it is not
+  // We pass subpath in as a parameter only for the error message
+  public static boolean checkBackPaths(String parent, String combinedPath, 
String subpath) {
+Preconditions.checkArgument(!parent.isEmpty());
+Preconditions.checkArgument(!combinedPath.isEmpty());
--- End diff --

Done


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267508#comment-16267508
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r153318818
  
--- Diff: 
exec/java-exec/src/test/java/org/apache/drill/exec/store/dfs/TestFileSelection.java
 ---
@@ -59,4 +61,53 @@ public void testEmptyFolderThrowsTableNotFound() throws 
Exception {
   throw ex;
 }
   }
+
+  @Test
+  public void testBackPath() throws Exception {
--- End diff --

Done


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267505#comment-16267505
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r152862665
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java 
---
@@ -359,15 +363,30 @@ private static Path handleWildCard(final String root) 
{
 }
   }
 
-  private static String removeLeadingSlash(String path) {
-if (path.charAt(0) == '/') {
+  public static String removeLeadingSlash(String path) {
+if (!path.isEmpty() && path.charAt(0) == '/') {
   String newPath = path.substring(1);
   return removeLeadingSlash(newPath);
 } else {
   return path;
 }
   }
 
+  // Check if the path is a valid sub path under the parent after removing 
backpaths. Throw an exception if
--- End diff --

Done


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267507#comment-16267507
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r152862954
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceConfig.java
 ---
@@ -30,18 +30,24 @@
 public class WorkspaceConfig {
 
   /** Default workspace is a root directory which supports read, but not 
write. */
-  public static final WorkspaceConfig DEFAULT = new WorkspaceConfig("/", 
false, null);
+  public static final WorkspaceConfig DEFAULT = new WorkspaceConfig("/", 
false, null, false);
 
   private final String location;
   private final boolean writable;
   private final String defaultInputFormat;
-
+  private final Boolean allowAccessOutsideWorkspace; // allow access 
outside the workspace by default. This
--- End diff --

I need to check if the value is not present (i.e. null). That will be the 
case with all storage plugin configurations that have already been created.


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267503#comment-16267503
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r152862636
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java 
---
@@ -252,11 +252,15 @@ private static String buildPath(final String[] path, 
final int folderIndex) {
 return builder.toString();
   }
 
-  public static FileSelection create(final DrillFileSystem fs, final 
String parent, final String path) throws IOException {
+  public static FileSelection create(final DrillFileSystem fs, final 
String parent, final String path,
+  final boolean allowAccessOutsideWorkspace) throws IOException {
 Stopwatch timer = Stopwatch.createStarted();
 boolean hasWildcard = path.contains(WILD_CARD);
 
 final Path combined = new Path(parent, removeLeadingSlash(path));
+if (!allowAccessOutsideWorkspace) {
+  checkBackPaths(parent, combined.toString(), path);
--- End diff --

Done


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5964) Do not allow queries to access paths outside the current workspace root

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267506#comment-16267506
 ] 

ASF GitHub Bot commented on DRILL-5964:
---

Github user parthchandra commented on a diff in the pull request:

https://github.com/apache/drill/pull/1050#discussion_r152862693
  
--- Diff: 
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSelection.java 
---
@@ -359,15 +363,30 @@ private static Path handleWildCard(final String root) 
{
 }
   }
 
-  private static String removeLeadingSlash(String path) {
-if (path.charAt(0) == '/') {
+  public static String removeLeadingSlash(String path) {
+if (!path.isEmpty() && path.charAt(0) == '/') {
   String newPath = path.substring(1);
   return removeLeadingSlash(newPath);
 } else {
   return path;
 }
   }
 
+  // Check if the path is a valid sub path under the parent after removing 
backpaths. Throw an exception if
+  // it is not
+  // We pass subpath in as a parameter only for the error message
+  public static boolean checkBackPaths(String parent, String combinedPath, 
String subpath) {
--- End diff --

Done


> Do not allow queries to access paths outside the current workspace root
> ---
>
> Key: DRILL-5964
> URL: https://issues.apache.org/jira/browse/DRILL-5964
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
>  Labels: doc-impacting
>
> Workspace definitions in the dfs plugin are intended to provide a convenient 
> shortcut to long directory paths. However, some users may wish to disallow 
> access to paths outside the root of the workspace, possibly to prevent 
> accidental access. Note that this is a convenience option and not a 
> substitute for permissions on the file system.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5963:

Reviewer: Paul Rogers

> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. two new states were added: PREPARING (when foreman is initialized) and 
> PLANNING (includes logical and / or physical planning).
> b. process of query enqueuing was made non-blocking. Once query was enqueued, 
> fragments runner is called to submit fragments locally and remotely.
> c. ability to cancel query during planning and enqueued states was added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5993) Allow Copier to Copy a Record to an Arbitrary Index

[Paul Rogers (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267374#comment-16267374
 ] 

Paul Rogers commented on DRILL-5993:


Trivial for fixed-width columns. Impossible for variable-width columns because 
of the way offset vectors work.

So, let's take a step back. Why do we think we need this?

> Allow Copier to Copy a Record to an Arbitrary Index
> ---
>
> Key: DRILL-5993
> URL: https://issues.apache.org/jira/browse/DRILL-5993
> Project: Apache Drill
>  Issue Type: New Feature
>Reporter: Timothy Farkas
>Assignee: Timothy Farkas
>
> Currently the copier can only copy record from an incoming batch to the 
> beginning of an outgoing batch. We need to be able to copy a record to any 
> index in the outgoing batch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4286) Have an ability to put server in quiescent mode of operation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267326#comment-16267326
 ] 

ASF GitHub Bot commented on DRILL-4286:
---

Github user dvjyothsna commented on the issue:

https://github.com/apache/drill/pull/921
  
This was due to grace_period during shutdown of drillbits. Fixed it by 
changing grace_period = 0 in .conf


> Have an ability to put server in quiescent mode of operation
> 
>
> Key: DRILL-4286
> URL: https://issues.apache.org/jira/browse/DRILL-4286
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - Flow
>Affects Versions: 1.11.0
>Reporter: Victoria Markman
>Assignee: Venkata Jyothsna Donapati
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.12.0
>
> Attachments: consoleText.txt, consoleText_after.txt, 
> consoleText_before.txt
>
>
> I think drill will benefit from mode of operation that is called "quiescent" 
> in some databases. 
> From IBM Informix server documentation:
> {code}
> Change gracefully from online to quiescent mode
> Take the database server gracefully from online mode to quiescent mode to 
> restrict access to the database server without interrupting current 
> processing. After you perform this task, the database server sets a flag that 
> prevents new sessions from gaining access to the database server. The current 
> sessions are allowed to finish processing. After you initiate the mode 
> change, it cannot be canceled. During the mode change from online to 
> quiescent, the database server is considered to be in Shutdown mode.
> {code}
> This is different from shutdown, when processes are terminated. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5993) Allow Copier to Copy a Record to an Arbitrary Index

[Timothy Farkas (JIRA)]

Timothy Farkas created DRILL-5993:
-

 Summary: Allow Copier to Copy a Record to an Arbitrary Index
 Key: DRILL-5993
 URL: https://issues.apache.org/jira/browse/DRILL-5993
 Project: Apache Drill
  Issue Type: New Feature
Reporter: Timothy Farkas
Assignee: Timothy Farkas


Currently the copier can only copy record from an incoming batch to the 
beginning of an outgoing batch. We need to be able to copy a record to any 
index in the outgoing batch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5970) DrillParquetReader always builds the schema with "OPTIONAL" dataMode columns instead of "REQUIRED" ones

[Pritesh Maker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pritesh Maker updated DRILL-5970:
-
Reviewer: salim achouche

> DrillParquetReader always builds the schema with "OPTIONAL" dataMode columns 
> instead of "REQUIRED" ones
> ---
>
> Key: DRILL-5970
> URL: https://issues.apache.org/jira/browse/DRILL-5970
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen, Execution - Data Types, Storage - 
> Parquet
>Affects Versions: 1.11.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>
> The root cause of the issue is that adding REQUIRED (not-nullable) data types 
> to the container in the all MapWriters is not implemented.
> It can lead to get invalid schema. 
> {code}
> 0: jdbc:drill:zk=local> CREATE TABLE dfs.tmp.bof_repro_1 as select * from 
> (select CONVERT_FROM('["hello","hai"]','JSON') AS MYCOL, 'Bucket1' AS Bucket 
> FROM (VALUES(1)));
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
> details.
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (2.376 seconds)
> {code}
> Run from Drill unit test framework (to see "data mode"):
> {code}
> @Test
>   public void test() throws Exception {
> setColumnWidths(new int[] {25, 25});
> List queryDataBatches = testSqlWithResults("select * from 
> dfs.tmp.bof_repro_1");
> printResult(queryDataBatches);
>   }
> 1 row(s):
> ---
> | MYCOL | Bucket|
> ---
> | ["hello","hai"]  | Bucket1  |
> ---
> Total record count: 1
> {code}
> {code}
> vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
> parquet-tools-1.6.0rc3-SNAPSHOT.jar schema /tmp/bof_repro_1/0_0_0.parquet 
> message root {
>   repeated binary MYCOL (UTF8);
>   required binary Bucket (UTF8);
> }
> {code}
> To simulate of obtaining the wrong result you can try the query with 
> aggregation by using a new parquet reader (used by default for complex data 
> types) and old parquet reader. False "Hash aggregate does not support schema 
> changes" error will happen. 
> 1) Create two parquet files.
> {code}
> 0: jdbc:drill:schema=dfs> CREATE TABLE dfs.tmp.bof_repro_1 as select * from 
> (select CONVERT_FROM('["hello","hai"]','JSON') AS MYCOL, 'Bucket1' AS Bucket 
> FROM (VALUES(1)));
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (1.122 seconds)
> 0: jdbc:drill:schema=dfs> CREATE TABLE dfs.tmp.bof_repro_2 as select * from 
> (select CONVERT_FROM('[]','JSON') AS MYCOL, 'Bucket1' AS Bucket FROM 
> (VALUES(1)));
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (0.552 seconds)
> 0: jdbc:drill:schema=dfs> select * from dfs.tmp.bof_repro_2;
> {code}
> 2) Copy the parquet files from bof_repro_1 to bof_repro_2.
> {code}
> [root@naravm1 ~]# hadoop fs -ls /tmp/bof_repro_1
> Found 1 items
> -rw-r--r--   3 mapr mapr415 2017-07-25 11:46 
> /tmp/bof_repro_1/0_0_0.parquet
> [root@naravm1 ~]# hadoop fs -ls /tmp/bof_repro_2
> Found 1 items
> -rw-r--r--   3 mapr mapr368 2017-07-25 11:46 
> /tmp/bof_repro_2/0_0_0.parquet
> [root@naravm1 ~]# hadoop fs -cp /tmp/bof_repro_1/0_0_0.parquet 
> /tmp/bof_repro_2/0_0_1.parquet
> [root@naravm1 ~]#
> {code}
> 3) Query the table.
> {code}
> 0: jdbc:drill:schema=dfs> ALTER SESSION SET  `planner.enable_streamagg`=false;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | planner.enable_streamagg updated.  |
> +---++
> 1 row selected (0.124 seconds)
> 0: jdbc:drill:schema=dfs> select * from dfs.tmp.bof_repro_2;
> +--+--+
> |  MYCOL   |  Bucket  |
> +--+--+
> | ["hello","hai"]  | Bucket1  |
> | null | Bucket1  |
> +--+--+
> 2 rows selected (0.247 seconds)
> 

[jira] [Commented] (DRILL-5970) DrillParquetReader always builds the schema with "OPTIONAL" dataMode columns instead of "REQUIRED" ones

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267262#comment-16267262
 ] 

ASF GitHub Bot commented on DRILL-5970:
---

Github user priteshm commented on the issue:

https://github.com/apache/drill/pull/1047
  
@sachouche can you please review this?


> DrillParquetReader always builds the schema with "OPTIONAL" dataMode columns 
> instead of "REQUIRED" ones
> ---
>
> Key: DRILL-5970
> URL: https://issues.apache.org/jira/browse/DRILL-5970
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Codegen, Execution - Data Types, Storage - 
> Parquet
>Affects Versions: 1.11.0
>Reporter: Vitalii Diravka
>Assignee: Vitalii Diravka
>
> The root cause of the issue is that adding REQUIRED (not-nullable) data types 
> to the container in the all MapWriters is not implemented.
> It can lead to get invalid schema. 
> {code}
> 0: jdbc:drill:zk=local> CREATE TABLE dfs.tmp.bof_repro_1 as select * from 
> (select CONVERT_FROM('["hello","hai"]','JSON') AS MYCOL, 'Bucket1' AS Bucket 
> FROM (VALUES(1)));
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
> details.
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (2.376 seconds)
> {code}
> Run from Drill unit test framework (to see "data mode"):
> {code}
> @Test
>   public void test() throws Exception {
> setColumnWidths(new int[] {25, 25});
> List queryDataBatches = testSqlWithResults("select * from 
> dfs.tmp.bof_repro_1");
> printResult(queryDataBatches);
>   }
> 1 row(s):
> ---
> | MYCOL | Bucket|
> ---
> | ["hello","hai"]  | Bucket1  |
> ---
> Total record count: 1
> {code}
> {code}
> vitalii@vitalii-pc:~/parquet-tools/parquet-mr/parquet-tools/target$ java -jar 
> parquet-tools-1.6.0rc3-SNAPSHOT.jar schema /tmp/bof_repro_1/0_0_0.parquet 
> message root {
>   repeated binary MYCOL (UTF8);
>   required binary Bucket (UTF8);
> }
> {code}
> To simulate of obtaining the wrong result you can try the query with 
> aggregation by using a new parquet reader (used by default for complex data 
> types) and old parquet reader. False "Hash aggregate does not support schema 
> changes" error will happen. 
> 1) Create two parquet files.
> {code}
> 0: jdbc:drill:schema=dfs> CREATE TABLE dfs.tmp.bof_repro_1 as select * from 
> (select CONVERT_FROM('["hello","hai"]','JSON') AS MYCOL, 'Bucket1' AS Bucket 
> FROM (VALUES(1)));
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (1.122 seconds)
> 0: jdbc:drill:schema=dfs> CREATE TABLE dfs.tmp.bof_repro_2 as select * from 
> (select CONVERT_FROM('[]','JSON') AS MYCOL, 'Bucket1' AS Bucket FROM 
> (VALUES(1)));
> +---++
> | Fragment  | Number of records written  |
> +---++
> | 0_0   | 1  |
> +---++
> 1 row selected (0.552 seconds)
> 0: jdbc:drill:schema=dfs> select * from dfs.tmp.bof_repro_2;
> {code}
> 2) Copy the parquet files from bof_repro_1 to bof_repro_2.
> {code}
> [root@naravm1 ~]# hadoop fs -ls /tmp/bof_repro_1
> Found 1 items
> -rw-r--r--   3 mapr mapr415 2017-07-25 11:46 
> /tmp/bof_repro_1/0_0_0.parquet
> [root@naravm1 ~]# hadoop fs -ls /tmp/bof_repro_2
> Found 1 items
> -rw-r--r--   3 mapr mapr368 2017-07-25 11:46 
> /tmp/bof_repro_2/0_0_0.parquet
> [root@naravm1 ~]# hadoop fs -cp /tmp/bof_repro_1/0_0_0.parquet 
> /tmp/bof_repro_2/0_0_1.parquet
> [root@naravm1 ~]#
> {code}
> 3) Query the table.
> {code}
> 0: jdbc:drill:schema=dfs> ALTER SESSION SET  `planner.enable_streamagg`=false;
> +---++
> |  ok   |  summary   |
> +---++
> | true  | planner.enable_streamagg updated.  |
> +---++
> 1 row selected (0.124 seconds)
> 0: jdbc:drill:schema=dfs> select * from dfs.tmp.bof_repro_2;
> +--+--+
> |  MYCOL   |  Bucket  |
> 

[jira] [Commented] (DRILL-5769) IndexOutOfBoundsException when querying JSON files

[David Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267136#comment-16267136
 ] 

David Lee commented on DRILL-5769:
--

I believe my other issue is a completely different problem. it looks like Drill 
is sampling the first xx number of json records to learn the schema and if it 
encounters a schema change down the road it throws a Index Out of Bounds. Can 
anyone confirm this? In my second case the error happens on record 10,200+ 
after the first 349 megs of the JSON file.

> IndexOutOfBoundsException when querying JSON files
> --
>
> Key: DRILL-5769
> URL: https://issues.apache.org/jira/browse/DRILL-5769
> Project: Apache Drill
>  Issue Type: Bug
>  Components:  Server, Storage - JSON
>Affects Versions: 1.10.0
> Environment: *jdk_8u45_x64*
> *single drillbit running on zookeeper*
> *Following options set to TRUE:*
> drill.exec.functions.cast_empty_string_to_null
> store.json.all_text_mode
> store.parquet.enable_dictionary_encoding
> store.parquet.use_new_reader
>Reporter: David Lee
>Assignee: Jinfeng Ni
> Fix For: Future
>
> Attachments: 001.json, 100.json, 111.json
>
>
> *Running the following SQL on these three JSON files fail: *
> 001.json 100.json 111.json
> select t.id
> from dfs.`/tmp/???.json` t
> where t.assetData.debt.couponPaymentFeature.interestBasis = '5'
> *Error:*
> org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: 
> IndexOutOfBoundsException: index: 1024, length: 1 (expected: range(0, 1024)) 
> Fragment 0:0 [Error Id: ....
> *However running the same SQL on two out of three files works:*
> select t.id
> from dfs.`/tmp/1??.json` t
> where t.assetData.debt.couponPaymentFeature.interestBasis = '5'
> select t.id
> from dfs.`/tmp/?1?.json` t
> where t.assetData.debt.couponPaymentFeature.interestBasis = '5'
> select t.id
> from dfs.`/tmp/??1.json` t
> where t.assetData.debt.couponPaymentFeature.interestBasis = '5'
> *Changing the selected column from t.id to t.* also works: *
> select *
> from dfs.`/tmp/???.json` t
> where t.assetData.debt.couponPaymentFeature.interestBasis = '5'



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-3640) Drill JDBC driver support Statement.setQueryTimeout(int)

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-3640:

Fix Version/s: (was: 1.12.0)
   1.13.0

> Drill JDBC driver support Statement.setQueryTimeout(int)
> 
>
> Key: DRILL-3640
> URL: https://issues.apache.org/jira/browse/DRILL-3640
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Client - JDBC
>Affects Versions: 1.2.0
>Reporter: Chun Chang
>Assignee: Kunal Khatua
> Fix For: 1.13.0
>
>
> It would be nice if we have this implemented. Run away queries can be 
> automatically canceled by setting the timeout. 
> java.sql.SQLFeatureNotSupportedException: Setting network timeout is not 
> supported.
>   at 
> org.apache.drill.jdbc.impl.DrillStatementImpl.setQueryTimeout(DrillStatementImpl.java:152)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5973) Support injections of a time-bound pause after which the server resumes

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5973:

Fix Version/s: (was: 1.12.0)

> Support injections of a time-bound pause after which the server resumes
> ---
>
> Key: DRILL-5973
> URL: https://issues.apache.org/jira/browse/DRILL-5973
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Tools, Build & Test
>Affects Versions: 1.11.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
> Fix For: 1.13.0
>
>
> While working on DRILL-3640 , when creating a unit test for a server-induced 
> timeout, the injecting a pause leaves the JUnit framework's DrillClient 
> without a handle to the query on the server. This is because we injected the 
> pause to occur before the server could send back a query ID, so the 
> DrillClient has no way to unpause the server.
> The workaround to support this unit test is to allow for injecting pauses 
> with a defined time-bound, after which the server would resume.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5972) Slow performance for query on INFORMATION_SCHEMA.TABLE

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5972:

Fix Version/s: (was: 1.12.0)
   1.13.0

> Slow performance for query on INFORMATION_SCHEMA.TABLE
> --
>
> Key: DRILL-5972
> URL: https://issues.apache.org/jira/browse/DRILL-5972
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Storage - Information Schema
>Affects Versions: 1.11.0
>Reporter: Padma Penumarthy
>Assignee: Padma Penumarthy
> Fix For: 1.13.0
>
>
> A query like the following on INFORMATION_SCHEMA takes a long time to 
> execute. 
> select TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE from 
> INFORMATION_SCHEMA.`TABLES` WHERE TABLE_NAME LIKE '%' AND ( TABLE_SCHEMA = 
> 'hive.default' ) ORDER BY TABLE_TYPE, TABLE_CATALOG, TABLE_SCHEMA, 
> TABLE_NAME; 
> Reason being we fetch table information for all schemas instead of just 
> 'hive.default' schema.
> If we  change the predicate like this, it executes very fast.
> select TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, TABLE_TYPE from 
> INFORMATION_SCHEMA.`TABLES` WHERE  ( TABLE_SCHEMA = 'hive.default' ) AND 
> TABLE_NAME LIKE '%'  ORDER BY TABLE_TYPE, TABLE_CATALOG, TABLE_SCHEMA, 
> TABLE_NAME; 
> The difference is in the order in which we evaluate the expressions in the 
> predicate.
> In the first case,  we first evaluate TABLE_NAME LIKE '%' and decide that it 
> is inconclusive (since we do not know the schema). So, we go get all tables 
> for all the schemas.
> In the second case, we first evaluate  TABLE_SCHEMA = 'hive.default' and 
> decide that we need to fetch only tables for that schema.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5971) Fix INT64, INT32 logical types in complex parquet reader

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5971:

Fix Version/s: (was: 1.12.0)
   1.13.0

> Fix INT64, INT32 logical types in complex parquet reader
> 
>
> Key: DRILL-5971
> URL: https://issues.apache.org/jira/browse/DRILL-5971
> Project: Apache Drill
>  Issue Type: Bug
>Affects Versions: 1.11.0
>Reporter: Parth Chandra
>Assignee: Parth Chandra
> Fix For: 1.13.0
>
>
> The 'complex' Parquet reader does not recognize the Parquet logical types 
> INT64, and INT32. 
> Should be a simple change to add these logical types.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5270) Improve loading of profiles listing in the WebUI

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5270:

Fix Version/s: (was: 1.12.0)
   1.13.0

> Improve loading of profiles listing in the WebUI
> 
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Web Server
>Affects Versions: 1.9.0
>Reporter: Kunal Khatua
>Assignee: Kunal Khatua
> Fix For: 1.13.0
>
>
> Currently, as the number of profiles increase, we reload the same list of 
> profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and 
> only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 
> core server. With the caching, we can get it down to as much as a few 
> milliseconds.
> To render the cache as invalid, we inspect the last modified time of the 
> directory to confirm whether a reload is needed. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5797) Use more often the new parquet reader

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5797:

Fix Version/s: (was: 1.12.0)
   1.13.0

> Use more often the new parquet reader
> -
>
> Key: DRILL-5797
> URL: https://issues.apache.org/jira/browse/DRILL-5797
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - Parquet
>Reporter: Damien Profeta
>Assignee: Damien Profeta
> Fix For: 1.13.0
>
>
> The choice of using the regular parquet reader of the optimized one is based 
> of what type of columns is in the file. But the columns that are read by the 
> query doesn't matter. We can increase a little bit the cases where the 
> optimized reader is used by checking is the projected column are simple or 
> not.
> This is an optimization waiting for the fast parquet reader to handle complex 
> structure.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-5963) Canceling a query hung in planning state, leaves the query in ENQUEUED state for ever.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267025#comment-16267025
 ] 

ASF GitHub Bot commented on DRILL-5963:
---

GitHub user arina-ielchiieva opened a pull request:

https://github.com/apache/drill/pull/1051

DRILL-5963: Query state process improvements

1. Added two new query states: PREPARING (when foreman is initialized) and 
PLANNING (includes logical and / or physical planning).
2. Process of query enqueuing was made non-blocking. Once query was 
enqueued, fragments runner is called to submit fragments locally and remotely.
3. Ability to cancel query during planning and enqueued states was added.
4. Logic for submitting fragments was moved from Foreman to new class 
FragmentsRunner.
5. Major type in DrillFuncHolderExpr was cached for better performance.

Other details in 
[DRILL-5963](https://issues.apache.org/jira/browse/DRILL-5963)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/arina-ielchiieva/drill DRILL-5963

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/1051.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1051


commit 1df913951ae9cadf3813836c4a03e66645d68965
Author: Arina Ielchiieva 
Date:   2017-11-24T15:59:42Z

DRILL-5963: Query state process improvements

1. Added two new query states: PREPARING (when foreman is initialized) and 
PLANNING (includes logical and / or physical planning).
2. Process of query enqueuing was made non-blocking. Once query was 
enqueued, fragments runner is called to submit fragments locally and remotely.
3. Ability to cancel query during planning and enqueued states was added.
4. Logic for submitting fragments was moved from Foreman to new class 
FragmentsRunner.
5. Major type in DrillFuncHolderExpr was cached for better performance.




> Canceling a query hung in planning state, leaves the query in ENQUEUED state 
> for ever.
> --
>
> Key: DRILL-5963
> URL: https://issues.apache.org/jira/browse/DRILL-5963
> Project: Apache Drill
>  Issue Type: Bug
>  Components: Execution - Flow
>Affects Versions: 1.12.0
> Environment: Drill 1.12.0-SNAPSHOT, commit: 
> 4a718a0bd728ae02b502ac93620d132f0f6e1b6c
>Reporter: Khurram Faraaz
>Assignee: Arina Ielchiieva
>Priority: Critical
> Fix For: 1.13.0
>
> Attachments: enqueued-2.png
>
>
> Canceling the below query that is hung in planning state, leaves the query in 
> ENQUEUED state for ever.
> Here is the query that is hung in planning state
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> select 1 || ',' || 2 || ',' || 3 || ',' || 4 || 
> ',' || 5 || ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' 
> AS CSV_DATA from (values(1));
> +--+
> |  |
> +--+
> +--+
> No rows selected (304.291 seconds)
> {noformat}
> Explain plan for that query also just hangs.
> {noformat}
> explain plan for select 1 || ',' || 2 || ',' || 3 || ',' || 4 || ',' || 5 || 
> ',' || 6 || ',' || 7 || ',' || 8 || ',' || 9 || ',' || 0 || ',' AS CSV_DATA 
> from (values(1));
> ...
> {noformat}
> The above issues show the following problems:
> *1. Simple query with reasonable number of concat functions hangs.*
> In reality query does not hang it just take lots of time to execute. The root 
> cause is that during planning time DrillFuncHolderExpr return type is 
> extensively used to determine matching function, matching type etc. Though 
> this type is retrieved via 
> [getter|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/DrillFuncHolderExpr.java#L41]
>  in reality complex logic is executed beaneath it. For example for [concat 
> function|https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/output/ConcatReturnTypeInference.java#L47].
>  Since function return type can not be changes during DrillFuncHolderExpr 
> life time, it is safe to cache it.
> *2. No mechanism to cancel query during ENQUEUED state.*
> Currently Drill does not have mechanism to cancel query before STARTING / 
> RUNNING states. Plus ENQUEUED state includes two PLANNING and ENQUEUED.
> Also submitting mechanism for submitting query to the queue is blocking, 
> making foreman wait till enqueueing is done Making it non-blocking will 
> prevent consuming threads that just sit idle in a busy system and also is 
> important when we move to a real admission control solution.
> The following changes were made to address above issues:
> a. 

[jira] [Commented] (DRILL-5992) Move html code generation for query profile from Java into freemarker

[Arina Ielchiieva (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-5992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16266679#comment-16266679
 ] 

Arina Ielchiieva commented on DRILL-5992:
-

FYI [~kkhatua] [~Paul.Rogers] [~prasadns14]

> Move html code generation for query profile from Java into freemarker
> -
>
> Key: DRILL-5992
> URL: https://issues.apache.org/jira/browse/DRILL-5992
> Project: Apache Drill
>  Issue Type: Improvement
>Affects Versions: 1.11.0
>Reporter: Arina Ielchiieva
> Fix For: Future
>
>
> Drill has lots of html generation that we have in Java for query profile 
> (https://github.com/apache/drill/tree/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile).
>  To adhere with MVC pattern  we should consider moving all that code into 
> freemarker + js if needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (DRILL-5992) Move html code generation for query profile from Java into freemarker

[Arina Ielchiieva (JIRA)]

Arina Ielchiieva created DRILL-5992:
---

 Summary: Move html code generation for query profile from Java 
into freemarker
 Key: DRILL-5992
 URL: https://issues.apache.org/jira/browse/DRILL-5992
 Project: Apache Drill
  Issue Type: Improvement
Affects Versions: 1.11.0
Reporter: Arina Ielchiieva
 Fix For: Future


Drill has lots of html generation that we have in Java for query profile 
(https://github.com/apache/drill/tree/master/exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile).
 To adhere with MVC pattern  we should consider moving all that code into 
freemarker + js if needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-5919) Add non-numeric support for JSON processing

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-5919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-5919:

Reviewer: Paul Rogers  (was: Arina Ielchiieva)

> Add non-numeric support for JSON processing
> ---
>
> Key: DRILL-5919
> URL: https://issues.apache.org/jira/browse/DRILL-5919
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Storage - JSON
>Affects Versions: 1.11.0
>Reporter: Volodymyr Tkach
>Assignee: Volodymyr Tkach
>  Labels: doc-impacting
> Fix For: Future
>
>
> Add session options to allow drill working with non standard json strings 
> number literals like: NaN, Infinity, -Infinity. By default these options will 
> be switched off, the user will be able to toggle them during working session.
> *For documentation*
> 1. Added two session options {{store.json.reader.non_numeric_numbers}} and 
> {{store.json.reader.non_numeric_numbers}} that allow to read/write NaN and 
> Infinity as numbers. By default these options are set to false.
> 2. Extended signature of {{convert_toJSON}} and {{convert_fromJSON}} 
> functions by adding second optional parameter that enables read/write NaN and 
> Infinity.
> For example:
> {noformat}
> select convert_fromJSON('{"key": NaN}') from (values(1)); will result with 
> JsonParseException, but
> select convert_fromJSON('{"key": NaN}', true) from (values(1)); will parse 
> NaN as a number.
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (DRILL-4286) Have an ability to put server in quiescent mode of operation

[Arina Ielchiieva (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DRILL-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arina Ielchiieva updated DRILL-4286:

Attachment: consoleText_after.txt
consoleText_before.txt

> Have an ability to put server in quiescent mode of operation
> 
>
> Key: DRILL-4286
> URL: https://issues.apache.org/jira/browse/DRILL-4286
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - Flow
>Affects Versions: 1.11.0
>Reporter: Victoria Markman
>Assignee: Venkata Jyothsna Donapati
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.12.0
>
> Attachments: consoleText.txt, consoleText_after.txt, 
> consoleText_before.txt
>
>
> I think drill will benefit from mode of operation that is called "quiescent" 
> in some databases. 
> From IBM Informix server documentation:
> {code}
> Change gracefully from online to quiescent mode
> Take the database server gracefully from online mode to quiescent mode to 
> restrict access to the database server without interrupting current 
> processing. After you perform this task, the database server sets a flag that 
> prevents new sessions from gaining access to the database server. The current 
> sessions are allowed to finish processing. After you initiate the mode 
> change, it cannot be canceled. During the mode change from online to 
> quiescent, the database server is considered to be in Shutdown mode.
> {code}
> This is different from shutdown, when processes are terminated. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (DRILL-4286) Have an ability to put server in quiescent mode of operation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1623#comment-1623
 ] 

ASF GitHub Bot commented on DRILL-4286:
---

Github user arina-ielchiieva commented on the issue:

https://github.com/apache/drill/pull/921
  
@dvjyothsna all tests has passed. Though there is significant performance 
degradation.
Before your changes unit test run takes ~ 33 minutes, after ~ 1 hour 18 
minutes. 
For example, exec package:
before `[INFO] exec/Java Execution Engine . SUCCESS 
[17:14 min]`,
after `[INFO] exec/Java Execution Engine . SUCCESS 
[56:45 min]`.
Since you did not add that many unit tests, could you please explain the 
reason of such performance degradation and ideally fix it.
Attached two unit tests outputs in Jira.



> Have an ability to put server in quiescent mode of operation
> 
>
> Key: DRILL-4286
> URL: https://issues.apache.org/jira/browse/DRILL-4286
> Project: Apache Drill
>  Issue Type: New Feature
>  Components: Execution - Flow
>Affects Versions: 1.11.0
>Reporter: Victoria Markman
>Assignee: Venkata Jyothsna Donapati
>  Labels: doc-impacting, ready-to-commit
> Fix For: 1.12.0
>
> Attachments: consoleText.txt
>
>
> I think drill will benefit from mode of operation that is called "quiescent" 
> in some databases. 
> From IBM Informix server documentation:
> {code}
> Change gracefully from online to quiescent mode
> Take the database server gracefully from online mode to quiescent mode to 
> restrict access to the database server without interrupting current 
> processing. After you perform this task, the database server sets a flag that 
> prevents new sessions from gaining access to the database server. The current 
> sessions are allowed to finish processing. After you initiate the mode 
> change, it cannot be canceled. During the mode change from online to 
> quiescent, the database server is considered to be in Shutdown mode.
> {code}
> This is different from shutdown, when processes are terminated. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)