[
https://issues.apache.org/jira/browse/ORC-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hgs updated ORC-1029:
---------------------
Description:
1.line 140 code in DataMask
public static DataMask build(DataMaskDescription mask,
TypeDescription schema,
MaskOverrides overrides) {
Iterator<Provider> iterator = LOADER.iterator();
LOG.info("hgs--"+Thread.currentThread().getName()+"---start........"
+mask.getName());
LOG.info("hgs--"+Thread.currentThread().getName()+"—start........");
{color:#ff0000}140{color} while(iterator.hasNext())
{ LOG.info("hgs--"+Thread.currentThread().getName()+"--"+
iterator.next().getClass().getName()); }
LOG.info("hgs--"+Thread.currentThread().getName()+"---end........");
for(Provider provider: LOADER) {
DataMask result = provider.build(mask, schema, overrides);
if (result != null)
{ return result; }
}
throw new IllegalArgumentException("Can't find data mask - " + mask);
}
2. the stacktrace
21/10/19 17:30:35 ERROR executor.Executor: Exception in task 12.0 in stage
0.0 (TID 12)
java.util.NoSuchElementException
at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:59)
at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
at org.apache.orc.DataMask$Factory.build(DataMask.java:140)
at
org.apache.orc.impl.WriterImpl$StreamFactory.getUnencryptedMask(WriterImpl.java:443)
at
org.apache.orc.impl.writer.EncryptionTreeWriter.<init>(EncryptionTreeWriter.java:61)
at org.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:130)
at org.apache.orc.impl.writer.StructTreeWriter.<init>(StructTreeWriter.java:41)
at
org.apache.orc.impl.writer.TreeWriter$Factory.createSubtree(TreeWriter.java:181)
at org.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:133)
at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:216)
at org.apache.orc.OrcFile.createWriter(OrcFile.java:1008)
at
org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:48)
at
org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:123)
at
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:156)
at
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:142)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:276)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:210)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:209)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
21/10/19 17:30:35 ERROR executor.Executor: Exception in task 14.0 in stage 0.0
(TID 14)
3. with one core is OK
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---start........nullify
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---start........
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7--org.apache.orc.impl.mask.SparkOrcMaskProvider
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7--org.apache.orc.impl.mask.MaskProvider
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---end........
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---start........nullify
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---start........
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7--org.apache.orc.impl.mask.SparkOrcMaskProvider
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7--org.apache.orc.impl.mask.MaskProvider
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---end........
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---start........nullify
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---start........
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7--org.apache.orc.impl.mask.SparkOrcMaskProvider
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7--org.apache.orc.impl.mask.MaskProvider
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---end........
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---start........nullify
21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
for task 7---start........
was:
1.line 140 code in DataMask
public static DataMask build(DataMaskDescription mask,
TypeDescription schema,
MaskOverrides overrides) {
Iterator<Provider> iterator = LOADER.iterator();
LOG.info("hgs--"+Thread.currentThread().getName()+"---start........"
+mask.getName());
LOG.info("hgs--"+Thread.currentThread().getName()+"—start........");
{color:#ff0000}140{color} while(iterator.hasNext())
{ LOG.info("hgs--"+Thread.currentThread().getName()+"--"+
iterator.next().getClass().getName()); }
LOG.info("hgs--"+Thread.currentThread().getName()+"---end........");
for(Provider provider: LOADER) {
DataMask result = provider.build(mask, schema, overrides);
if (result != null)
{ return result; }
}
throw new IllegalArgumentException("Can't find data mask - " + mask);
}
2. the stacktrace
21/10/19 17:30:35 ERROR executor.Executor: Exception in task 12.0 in stage
0.0 (TID 12)
java.util.NoSuchElementException
at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:59)
at java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
at org.apache.orc.DataMask$Factory.build(DataMask.java:140)
at
org.apache.orc.impl.WriterImpl$StreamFactory.getUnencryptedMask(WriterImpl.java:443)
at
org.apache.orc.impl.writer.EncryptionTreeWriter.<init>(EncryptionTreeWriter.java:61)
at org.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:130)
at org.apache.orc.impl.writer.StructTreeWriter.<init>(StructTreeWriter.java:41)
at
org.apache.orc.impl.writer.TreeWriter$Factory.createSubtree(TreeWriter.java:181)
at org.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:133)
at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:216)
at org.apache.orc.OrcFile.createWriter(OrcFile.java:1008)
at
org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:48)
at
org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:123)
at
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:156)
at
org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:142)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:276)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:210)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:209)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
21/10/19 17:30:35 ERROR executor.Executor: Exception in task 14.0 in stage 0.0
(TID 14)
> Could not load 'org.apache.orc.DataMask.Provider' when using orc encryption
> and spark executor with multi cores!
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ORC-1029
> URL: https://issues.apache.org/jira/browse/ORC-1029
> Project: ORC
> Issue Type: Bug
> Components: Java
> Affects Versions: 1.6.11
> Environment: jdk1.8
> spark 2.4.7
> orc 1.6.11
> Reporter: hgs
> Priority: Blocker
>
> 1.line 140 code in DataMask
> public static DataMask build(DataMaskDescription mask,
> TypeDescription schema,
> MaskOverrides overrides) {
> Iterator<Provider> iterator = LOADER.iterator();
> LOG.info("hgs--"+Thread.currentThread().getName()+"---start........"
> +mask.getName());
> LOG.info("hgs--"+Thread.currentThread().getName()+"—start........");
> {color:#ff0000}140{color} while(iterator.hasNext())
> { LOG.info("hgs--"+Thread.currentThread().getName()+"--"+
> iterator.next().getClass().getName()); }
> LOG.info("hgs--"+Thread.currentThread().getName()+"---end........");
> for(Provider provider: LOADER) {
> DataMask result = provider.build(mask, schema, overrides);
> if (result != null)
> { return result; }
> }
> throw new IllegalArgumentException("Can't find data mask - " + mask);
> }
>
>
> 2. the stacktrace
> 21/10/19 17:30:35 ERROR executor.Executor: Exception in task 12.0 in stage
> 0.0 (TID 12)
> java.util.NoSuchElementException
> at sun.misc.CompoundEnumeration.nextElement(CompoundEnumeration.java:59)
> at
> java.util.ServiceLoader$LazyIterator.hasNextService(ServiceLoader.java:357)
> at java.util.ServiceLoader$LazyIterator.hasNext(ServiceLoader.java:393)
> at java.util.ServiceLoader$1.hasNext(ServiceLoader.java:474)
> at org.apache.orc.DataMask$Factory.build(DataMask.java:140)
> at
> org.apache.orc.impl.WriterImpl$StreamFactory.getUnencryptedMask(WriterImpl.java:443)
> at
> org.apache.orc.impl.writer.EncryptionTreeWriter.<init>(EncryptionTreeWriter.java:61)
> at org.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:130)
> at
> org.apache.orc.impl.writer.StructTreeWriter.<init>(StructTreeWriter.java:41)
> at
> org.apache.orc.impl.writer.TreeWriter$Factory.createSubtree(TreeWriter.java:181)
> at org.apache.orc.impl.writer.TreeWriter$Factory.create(TreeWriter.java:133)
> at org.apache.orc.impl.WriterImpl.<init>(WriterImpl.java:216)
> at org.apache.orc.OrcFile.createWriter(OrcFile.java:1008)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcOutputWriter.<init>(OrcOutputWriter.scala:48)
> at
> org.apache.spark.sql.execution.datasources.orc.OrcFileFormat$$anon$1.newInstance(OrcFileFormat.scala:123)
> at
> org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:156)
> at
> org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:142)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:276)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:210)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:209)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
> at org.apache.spark.scheduler.Task.run(Task.scala:123)
> at
> org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 21/10/19 17:30:35 ERROR executor.Executor: Exception in task 14.0 in stage
> 0.0 (TID 14)
>
>
> 3. with one core is OK
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---start........nullify
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---start........
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7--org.apache.orc.impl.mask.SparkOrcMaskProvider
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7--org.apache.orc.impl.mask.MaskProvider
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---end........
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---start........nullify
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---start........
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7--org.apache.orc.impl.mask.SparkOrcMaskProvider
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7--org.apache.orc.impl.mask.MaskProvider
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---end........
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---start........nullify
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---start........
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7--org.apache.orc.impl.mask.SparkOrcMaskProvider
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7--org.apache.orc.impl.mask.MaskProvider
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---end........
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---start........nullify
> 21/10/19 18:08:17 INFO orc.DataMask$Factory: hgs--Executor task launch worker
> for task 7---start........
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
